How the Pollfish methodology works


As market research has been around for a long time, there are many methods for collecting survey responses. All have pros and cons, and before you can fully understand the Pollfish methodology, you should know who we are up against.

Survey Methodology Examples

Random Digit Dialing (RDD)

This is an outdated method of polling where researchers build a sample and call potential respondents on the phone. This method has many positives, including polling respondents in a neutral location, getting respondents who are not planning ahead to take a survey and other factors that increase data quality. But as people move toward mobile devices instead of home phones, this survey methodology has serious challenges.

Online Panels

Online panels collect responses from many different sources, incentivizing people to sign up as professional survey takers in exchange for rewards like cash, airline miles and more. This has a number of downsides, most of all people rushing through surveys in order to get their reward and get to the next lucrative survey.

Assisted Crowdsourcing

Assisted crowdsourcing uses recruitment ads on social networks. High coverage but low response rates slow the speed of this methodology, ads can be pricey and, most importantly, there is no way to prevent the Facebook algorithm from showing your ad to people more interested in politics than the average person. This creates sample bias and reduces data quality.

How does the Pollfish methodology work? 

Pollfish utilizes a new survey methodology called Random Device Engagement. RDE is the natural successor to Random Digit Dialing. By delivering your survey inside popular mobile apps, RDE utilizes the same neutral environment as RDD, and an audience who are not taking premeditated surveys, by reaching them inside mobile apps they were using anyway.

Pollfish uses non-monetary incentives like an extra life in a game or access to premium content. With additional layers of survey fraud prevention including AI and machine learning, Pollfish removes potentially biased responses, improving data quality even further.

How is this done? How can we reach and audience of millions without monetary incentives?

Let’s start at the source—our audience

There are roughly 7.5 billion people living on planet earth. About 5 billion of those people have mobile devices, and the number is growing. 

Every one of those people is a “mobile consumer”, and a potential respondent to your survey questions. Using the apps that they have installed on their devices that they are in every day, we can quickly gather a massive volume of responses and target them accurately.

Why hasn’t this been done before?

Mobile surveys have been tried by researchers before—and it hasn’t gone great. River Sampling randomly sent surveys through mobile, disrupting the mobile respondent experience and offering no way for researchers to know who was taking the survey or if they fit into the target audience. Response quality was so bad that researchers rushed back to their imperfect-but-better-than-nothing online panels, and haven’t looked back for 20 years. 

Here we found the perfect opportunity to shake things up. With a group of Android designers and engineers as our founding team, it was a no-brainer that our research software was going to be rooted in mobile development. But since nobody wants to download an app to take surveys (gross) there had to be another way to reach an audience at scale that provided the response quality that researchers needed.

So we designed an SDK that integrates natively with mobile apps.

Integration for better information

An SDK (software development kit) is a single line of code that can non-intrusively be included in the code of an app. This provides an API (Application programming interface) that more or less lets our platform and the app “talk to each other.” 

By integrating directly with apps, we’re able to collect unique user information to manage and track respondents across devices using their mobile ad ID. While we don’t collect any PII, we are still able to complete a respondent profile during their initial opt-in phase, including everything from their basic demographic information to their age, marital status, race, and geographic location. This allows us to distribute surveys to a randomized group of people who fit the targeting criteria of a given survey in a process called Random Device Engagement.

This is what River Sampling was lacking. They had the right idea to survey real users on mobile devices, but with no way of gathering information back from the respondent, responses couldn’t be verified and fraud couldn’t be prevented.

To see the full evolution of market research, check out our webinar below.


The app partnership model.

“In-app survey… That sounds sketchy.”

Because of our delivery model, this is actually less sketchy than you’d expect. Unlike some of our competitors, we don’t blackmail our respondents with a paywall until they take a survey. It’s completely optional, every-single time. 

With in-app surveys occurring only with engaged users, they have the option to opt-in or out in the moment and their responses are naturally occurring in real-time. If they opt-in, they could get a small, appropriate perk that is specific to the app they are using. If they opt-out, there’s always the option to take it the next time they’re invited to participate.

“So then, why do they even take the survey, if they can just skip it and go back to what they were doing?”

Here’s why: Incentives. But not incentives that are cumulative or monetarily based. 

As much as survey tools love to promote data integrity, allowing respondents to take multiple surveys and collect points or cash leads to bias. When panelists are incentivized to take as many surveys as possible to accumulate rewards, the volume of responses goes up but data quality plummets. 

This is yet another advantage of our app partnership model: the developers have control over respondent incentives specific to their individual apps. For example, if we have partnered with an app for world news, a survey incentive might be a premium content article. For a fitness app, perhaps a free yoga class. In a game, an extra life— and so on. The publishers know what will motivate their users and prompt them to choose a survey when it is presented.

Because we don’t bribe our audience with monetary incentives, we enjoy significant cost savings that we are able to pass on to our customers—along with the highest quality data available from any mobile survey tool.

This model benefits the app publishers too. Mobile surveys are one of the most highly-paid monetization options available to app developers. Instead of running advertisements, they can work with us to offer their users mobile surveys. Users are more engaged with surveys than ads, and the app publishers make a higher return on them than they do from their typical mobile advertising options.

? Learn more about our Audience

AI and Fraud Prevention

“But what about people who opt-in and just don’t care?”

There’s always going to be a certain number of people who opt-in to take a survey, and then don’t exhibit above-board behavior. Cheaters, speeders, and bots have long plagued the effectiveness of consumer surveys. 

Fortunately, we developed a proprietary machine-learning algorithm that can detect fraudulent responses early and remove respondents immediately who aren’t up to our standards. 

This can mean anything from straight-lining responses to nonsense written in open-ended answers. Even suspicious behavior, like spending too much time in a questionnaire, is enough for us to remove a respondent from the survey. We have the audience participation to fill virtually any quota so there’s no need for us to settle for subpar response quality, and it’s why we throw out up to 30% of our responses

We take our fraud prevention extremely seriously to keep your response quality and overall data safe.

? Read more about our data quality

Mobile delivery and user experience.

Pollfish keeps two audiences in mind during development: The recipient of your survey, taking it on their mobile device, and you, the user, building a survey on our platform. The design of both experiences is critically important to the data quality, completion rates, and keeping frustration low for all parties involved. 

Respondent Experience 

The surveys we send to respondents are mobile-optimized to be an organic part of the experience from beginning to end. With UX thinking leading our development, we prioritize how consumers experience mobile, how surveys fit into that behavior, and uphold best practices to be mindful of the way people naturally react in these settings.

For example, surveys that can be completed in-app without taking a user to a third-party location are less disruptive to the mobile experience. When surveys are quick and convenient for respondents, they’re happy to answer a few questions and get an appropriate reward. This means better completion rates for you.

We take into consideration the layout, design, and length of surveys to ensure a positive user experience for the respondent, driving higher quality responses overall and a quick turnaround from the instant collection method. 

? 50 ideas to improve your survey response rate 
? Learn to write a good survey question

The respondent sees a brief, mobile-optimized questionnaire that never removes them from the app they were in.

User Experience

Don’t worry— we didn’t forget about you.

Much like the surveys respondents take, the Pollfish platform offers a friendly, intuitive interface that tells our users what they need to know upfront. Things like screening questions and demographic quotas help you narrow in on the exact audience you want to reach. Features like feasibility, audience preview, and transparent pricing make it easy to understand how long or expensive it will be to run the survey in question.

The platform even updates in real-time, so once the survey is launched, you can watch your results as they come in, knowing each one is a real user, engaged in your survey, on their device.

We help people from all sorts of backgrounds with all sorts of survey-related problems—including a lot of people who are new to the world of research or survey-tools in general. This is why we also offer our resource center and personalized help from our 24/7 customer experience staff—even the option to hire a research expert to help you with your survey. 

-> Interact with sample results
-> Get comfortable with the platform

To summarize, Pollfish is a platform for you that grants access to a massive network of respondents who opt-in when a survey is presented to them. Using randomized mobile delivery, advanced AI, and targeting capabilities, you can reach a narrow audience segment with higher speed and accuracy than alternative survey options. 

We take a lot of the work out by offering our platform and audience all-in-one, but you can also use our third-party offering to connect your existing survey to our audience network. Check out the third-party page to learn more about compatible tools.