How to use Pollfish screening questions like a pro

The best way to learn about Pollfish is straight from the pros of the Pollfish customer experience team—a team of real people available 24/7 to help clients get the most out of the platform. 

Pollfish expert Jim Theodoropoulos answers some common questions about how to best use Pollfish screening questions for surveys. Check out his expertise in the interview below, and get tips for high-quality audience targeting.

Tell us a little about yourself.

I’ve been with Pollfish for around 2.5 years. The platform is very alive, every day is different. I’ve learned a lot since my first year—the way I communicate with the clients, the things I’ve learned have changed a lot. Now I know what kinds of problems they may have and I can anticipate those to avoid future problems. 

What are some things to keep in mind when creating a screening question in general?

You should formulate screening questions for surveys by thinking about the words you will choose because, in a way, you’re playing with people’s minds. Consider how people will read and receive your content within the screening question. For example, people are biased towards a positive response, so “yes or no” answer options are not a good structure. It’s best to frame the question in a way where it is not clear what the “correct” response should be.

It’s really important to spend time setting it up correctly because, you know, this is your data. Even experienced researchers struggle with screening question design. They don’t realize that the design and the structure of the screening questions are like 80% of their survey’s performance. 

What’s the first thing people should know about building a screening question on Pollfish? Is that different from other platforms?

The first step in Pollfish is to set up demographic targeting. The role of the screening question is to target even further at a more granular level, so it works in conjunction with Pollfish targeting, not instead of it. It’s very important to design it so that it works together to narrow in on the audience. 

For example, if you wanted to target mothers who are also runners, you need to first select the female gender and the “number of children” filter to ensure that you’re targeting mothers. This builds a qualified audience to ask a behavioral question about running. Use the screening question to ask about exercise habits and ensure that the mothers are runners. 

Many researchers are used to the traditional providers, so I don’t think they realize that they need to design their demographic targeting first to first get the best broad audience possible, then from there they can screen further.

What are some misuses or mistakes you’ve seen with screening questions? What happened?

A common mistake is when people try to do all of the targeting through the screening question, without using the filters. This sends the surveys to everyone, so there will be low incidence rates and a lot of data that isn’t usable.

Another common mistake is when people use a multiple-selection screening question and do not realize that if they let respondents select other choices that they will be screened in as well. They sometimes think that the multiple-selection is multiple users selecting an answer and not a user selecting multiple answers.

Wouldn’t a good screening question end up saving researchers money in the long run?

Yes. Some researchers might think that they are saving money by not using them because the screening questions cost a little more, but they will actually cost more if they don’t set them up properly. There will be no value in their collected data if they aren’t reaching the right audience.

For example, if you’re distributing a survey asking people about cars and you’re distributing it to people who don’t drive, there’s no value in that data and it’s not a good use of money because you’re paying for responses that you can’t use. 

What do you think confuses people the most about screening question creation?

Letting the wrong people enter the survey, but also what I mentioned earlier—the yes or no screening question—many people think that the platform can detect when a respondent has lied on an answer. Our fraud prevention can remove patterns and many other behaviors, but it is best to write the answers in an unbiased way to make extra sure. This is why screening questions should contain answers that are ambiguous about what we would like the respondent to choose. 

Do screening questions encourage more people to take surveys? 

Not really. Screening questions focus on increasing the quality of the audience entering the survey, not the number of surveys they complete. It narrows targeting on who receives the questionnaire at the beginning.

What should people do with the “incidence rate” part of a screening question?

If a client knows the incidence rate of the screening question, then they should input it during this step. If they don’t know it, that’s not an issue. Many people do not know their screening question IR. It’s dependent on their experience researching the topic. For example, if you know that a medical condition is related to a specific percentage of people, like 2% of the US general population, then you can add the IR there. This feature helps us with a more accurate calculation of how long the survey will take, but is not required to use and won’t harm the data if the client doesn’t know it.

What question types are available for screening questions? Is one better than another?

Single and multiple-choice selection are the only two kinds available on Pollfish. One is not better than the other, they’re just better for different scenarios. For example, if you want to allow people familiar with only your brand to enter a survey, single-selection is best. If you want to include respondents who are familiar with your brand and several competitors, multiple-selection can allow that.

Is there a way to do any branching with the screening questions?

Not yet, but in general, every survey should be sent to a relevant audience. So if you have two different audiences, then you need to divide your survey instead of using branching and directing them to different questions.

Sometimes people try to do too many things at once with one survey and it creates confusion or issues segmenting at the end. It’s better to create more surveys that are clear than to force everything into one.

How do you help people who have multiple audiences that they need to screen?

Sometimes the screening question applies to two or more audiences, so what I do—and I don’t know about other people—but what I do is suggest dividing the survey, and then I also calculate the total cost to allocate it to the different surveys so they know how much each will cost. I’ll also help them figure out the total responses needed for each survey to maintain a statistical sample size for both surveys. That way they can stay within their budget, but also ensure the data has value to them.

How can you be sure you’ll have enough respondents if you use a screening question?

Because we are not a traditional sample provider, we cannot tell exactly the number of people who will be available and also a fit for a screening question. If someone is available to take a survey today, they may not be tomorrow. The network of real people means there are potential respondents everywhere, taking randomized surveys but at different times. We calculate the feasibility based on past users from the network.

What do you do if you don’t have enough? Like, in the event of a very targeted screening question?

We have screening questions for surveys come through about medical conditions that can be very specialized and the incidence rate is extremely low. If the IR is too low, the survey will be stopped automatically by the system. In such cases, we first try to see if we can expand something about the targeting to add more respondents. If nothing can be expanded, then we reimburse the credits back to the account and suggest that they restructure and launch another survey.

And you guys will work with them to help them restructure and launch that second survey to make it better?

We will advise and help them understand the best ways to restructure their survey, however, it is a DIY tool so there are some limits to how much we can do for them. But we now have a professional research service so they can hire someone to join their team and build the survey for them if it’s very complicated. The researcher can discuss their specific purpose for research, then structure the screening question to be very precise for that audience. They can always bring on a research expert if they need extra help.

What is the most creative/ funny/ weird screening question you have seen on Pollfish?

I have seen a lot of stuff during these years. We have to review every question for content, and sometimes they can be pretty funny. We’re careful with the content because sometimes the audience targeting is set for minors, and we want to make sure it’s okay for them. I think the most interesting questions are the political ones.

What is one piece of advice or thing you would like people to know about Pollfish screening questions? 

Make sure that “yes or no” questions are avoided and use screening questions with targeting, not as a separate part. They work together to narrow the results so the targeting should be set first. 

Get more information on using screening questions to reach your target audience before launching your survey, or reach out to Jim and the rest of the Pollfish customer experience team any time to help with screening question FAQs.