Grouping, Shuffling and Randomization in the Pollfish Questionnaire (2022 Update)

As many researchers learn early on, it can be hard to limit bias in surveys which has negative consequences on the quality of data. 

Some best practices for minimizing bias include randomization of questions and the distribution of the questionnaires themselves to reach a diverse audience. Shuffling unordered answers or reversing the order of a scale has also been shown to reduce bias by displaying clear, but differentiated answer options to respondents. 

Although Pollfish has supported shuffling and grouping within individual questions, we have recently released the ability to fully randomize your questionnaire. You can shuffle groups of questions and even shuffle the questions within the group for even more randomized survey opportunities. 

Here’s how to apply it in your next survey.

Grouping Questions in Pollfish

When you visit the questionnaire section of your Pollfish dashboard, you’ll see we offer new randomization capabilities. Here, you’ll be able to assign questions into groups on the questionnaire. 

This can be helpful for managing your survey in specific sections, for example, if you wanted to first investigate behavior before diving into specific interests. 

There is no need for a group to contain any certain number of questions. A “group” can contain a single question if needed, however if Grouping is applied to the questionnaire, every question will need to be assigned to a group.

You can manually re-order the groups or the questions within the groups by dragging and dropping, adding or deleting questions as you would in a regular Pollfish survey.

Keep in mind that when you delete a group, you may also delete the questions within the group by selecting “Delete All.” If you’d like to remove the grouping feature but keep your questions, make choose the option that says “Delete Group Only.”

Shuffling and Randomizing Questions in Groups

The groups themselves and the questions within the group can be shuffled randomly or reversed in order to reduce bias without disrupting the logical flow of the questionnaire.

However, there is no ”Grouping and Shuffling” as an aggregate option anymore. Instead the option has been divided into ”Randomize Questionnaire” (located on the left side) and Group of Questions (located at the bottom of the Question type drop-down menu).

A group can also be “fixed” while other groups are shuffled. For example, perhaps you want to provide a description before a group of questions that will provide context. In this case, you can choose to put a description question-type in a group and shuffle the remaining groups in your questionnaire.

Don’t forget that you can (and should) include shuffling and reverse order options on your individual questions as well, adding to even more randomization and cleaner data.

Originally, we had 3 main ways to shuffle: 

• Shuffle the answers in an individual question
• Shuffle groups of questions in your questionnaire
• Shuffle the questions within a group

But now, there are more options for randomizing questions.

You can enable Randomize Questionnaire and select Shuffle All, to shuffle a group of questions with the rest of the questions. If you select the group, you have an additional option; you can randomize the questions within the group. When you turn off the Randomize Questionnaire toggle and shuffle the sub-questions, the platform only shuffles the questions within a particular group, the rest of the questions won’t be shuffled.

Essentially, you can shuffle anything you choose while leaving certain questions fixed.

Here’s how else you can shuffle on Pollfish:

• Shuffle but keep first one fixed
• Shuffle but keep first two fixed
• Shuffle but keep last one fixed
• And many other ways!

Pollfish Question Randomization Capabilities

Now, it’s time to see exactly what our question randomization options look like on the Pollfish platform. To get a visual representation of the functions mentioned in the previous section, look no further.

The following images show how we’re offering randomization capabilities on our platform:

Randomize Questions

Randomize Question Options

Randomize Sub-Questions

Groups of Questions

Adding logic to groups: The Van Wesendorp Pricing Meter and A/B testing (2022 update)

Although grouped questions didn’t originally include skip logic, the product team has been hard at work to include this feature. Thus, advanced skip logic will support grouped and shuffled questions, as well as allow for A/B testing, a critical and useful feature in advertising effectiveness surveys and more. 

You can now use advanced skip logic when a respondent exits a group or an A/B test. You can also use it with the Van Westendorp Price Sensitivity Meter. This is a pricing model that provides data in regard to consumer price preferences. It is used to determine customers’ willingness to pay a range of prices.

This meter takes the form of a graph, which visualizes ratings on price and value from respondents. This data is presented as responses to survey questions that focus on the prices of different products and services.

Thus, when a user exits a group via the A/B testing or Van Westendorp features, you can route them to a specific question.