Survey Panels Vs Organic Sampling: Which is Better for Market Research?

Survey Panels Vs Organic Sampling: Which is Better for Market Research?

Survey panels and organic sampling are two of the major methods used to collect survey data. Since getting survey responses without the assistance of software is an almost impossible feat, these two methods have reigned supreme.

Both of these survey response mechanisms are unlikely to wane anytime soon, due to the prevalence of online surveys. As a matter of fact, the online survey software space has risen by 8% between 2015 and 2020.

With online surveys granting market researchers and marketers a large sweep of survey types to conduct, the point of contention becomes: which type of survey response collection data is better, survey panels or organic sampling?

This article will explore both forms of data collection so that you can objectively decide which to opt for in your survey research campaigns. 

Defining Online Survey Panels

Survey panels, also called online panels or research panels, all denote a data collection method in which responses are collected by way of pre-recruited and pre-screened respondents who agreed to take part in a survey.

This method helps businesses in that it assures them that a group of people will take their survey; usually, members of their target market are called upon to take the survey. 

There are a number of ways survey panels can be assembled. One such method involves mail-in recruitment, which has significantly declined in the digital age. Another relies on phone calls via Random-Digit Dial (RDD), a method in which respondents are chosen to take a survey from random telephone numbers. 

When it comes to recruiting survey panels digitally, they are garnered through an opt-in format such as a signup page or through an email invite that routes users to the survey page.  All of these recruitment manners have low participation, as few people opt into panels.

That is why some brands and market researchers resort to compensating their pre-screened panelists.

The Pros and Cons of Survey Panels

Now that you understand the basic methodology behind survey panels, you ought to consider their pros and cons before fully forming your opinion on whether to use them or not. 

The pros of online panels:

  1. They provide a network of respondents for continuous survey participation. This is especially useful if you need to conduct longitudinal studies. 
  2. They are inexpensive and create fast studies. 
  3. Returning to the same respondents, allows you to detect changing opinions over time, allowing you to see how your target market changes its opinions.
  4. They allow you to create informed custom polling questions based on previous research.

The cons:

  1. Repeated survey participation causes panel fatigue, a term denoting the decline in the quality of survey data, due to the boredom or exhaustion of a panelist. This creates them to inaccurately provide responses, either due to skipping questions, ticking off the “don’t know” option or rushing through a survey. 
  2. They occur in non-organic (unnatural) environments — inauthentic environments create inauthentic responses. This is because a survey’s environment can affect its’ respondents’ mindsets.
  3. Web panels gather respondents either on desktop or mobile, creating scenarios in which participants are dependent on device types. As such, the survey experience is not very adaptive. 
  4. Panel conditioning: Repeated survey participation can change respondents’ true attitudes, behaviors and knowledge. This makes it difficult to differentiate between actual changes and changes in reporting behaviors.

Random Device Engagement (RDE): A Precursor to Organic Sampling 

Before you analyze the organic sampling method, you should understand what makes it tick. That is because organic sampling is reliant on the delivery structure known as Random Device Sampling (RDE).

This framework implements intent-based behavioral targeting, typically used by advertisers, to narrow down random respondents in a digital setting, such as on websites, mobile sites and mobile apps. 

Thus, it provides a solution for randomization and capturing the correct audience. 

Random Device Sampling works by tracing the unique IDs of respondents, which are used to track them across devices. RDE, therefore, institutes a mechanism that is both random and organic. 

Organic Sampling Defined

Also called random organic sampling, this method refers to an RDE-based response collection method in which a survey is deployed randomly to users who are already in apps and other digital spaces.

Since surveys are randomly transferred, this method allows respondents to take the surveys while they are in their organic environments. These are the spaces that users spend time in organically, meaning they chose to take part in those environments, rather than being taken there via a promotion, incentive or signing up at a web panel.

Organic sampling works by giving optional invitations (or call-outs) to users in organic settings, so that they would partake in quick surveys. These invitations (along with the surveys themselves) are natively integrated within the digital environments (ex: apps) that the users are in.

This makes several benefits possible. 

The Pros and Cons of Organic Sampling

Powered by Random Device Engagement, organic sampling has many upper hands. However, like the panel survey method, organic sampling also presents certain disadvantages that you should carefully consider. You ought to weigh them against one another before deciding whether or not to use them. 

The Pros of Organic Sampling

  1. An organic environment allows you to avoid panel conditioning, allowing you to extract genuine responses from participants.
  2. It targets respondents’ unique IDs so that they can be traced even while shuffling between devices, preventing the same participants from taking the same survey twice. 
  3. Using respondents’ IDs allows you to create respondent profiles, which build an overview of the respondents’ behaviors demographics, which are critical data to possess.
  4. In turn, the respondent profile helps prevent fraud, as multiple accounts or bots won’t able to submit their surveys. 
  5. It yields high response rates (higher than RDD and survey panels).
  6. It creates a seamless UX across all device types, rather than being tied to just one with little wiggle room. 
  7. It provides vast coverage, as RDE integrates natively with a bevy of digital and mobile platforms. That entails greater accuracy. 

The Cons:

  1. It relies on non-probability sampling, wherein some of the members of a population have been excluded and this amount cannot be calculated, which limits how much you can determine about the population from the sample.
  2. It will include biases based on the populations you have preset to include in your sample, even if you assign quotas.
  3. Organic sampling and RDE are at the mercy of the websites and apps that they can be integrated with. It is possible that your targeted sample pool visits apps and other digital places that your RDE survey isn't integrated with.

Which Sample Pool Method Reigns Supreme?

While random organic sampling has made a mark in the survey realm, due to Random Device Engagement, it still faces its rivalrous counterpart: survey panels. 

Other forms of survey sampling methods have been on the decline, yet survey panels are still in use. While they may appear to have fewer advantages and more disadvantages, survey panels still provide value to researchers. 

As such, it is entirely up to you to decide which survey sampling method is best for your market research needs. Random organic sampling is our survey sampling method of choice, as it continues to rise above survey challenges, provides solid results and does so in a short span of time.

Frequently asked questions

What is an online survey panel?

Also called online panels or research panels, an online survey panel is a way of repeatedly collecting data from pre-recruited respondents via digital surveys.

What are some of the benefits of an online survey panel?

Online survey panels allow companies to quickly and inexpensively collect data from a group of people since the recruitment and screening process only needs to be performed once. Since responses are collected from the same group over time, they are able to show if and how opinions or behaviors change over.

What is panel fatigue?

Panel fatigue is a concern associated with survey panels. It occurs when panel participants become bored or tired of replying to surveys. This can result in the overall inaccuracy of the data collected from that participant.

What is organic sampling?

Organic sampling, or random organic sampling, refers to the process of distributing surveys to a random group of respondents. The respondents are typically sourced from websites or apps where they are given the option to participate in a survey.

What are the benefits of organic sampling?

Organic sampling can help prevent some of the pitfalls of survey panels including panel fatigue and panel conditioning. Organic sampling also tends to yield higher response rates and can help prevent fraud or data errors.


How to Get Insight from Consumer Panel Surveys: 10 Tips

How to Get Insight from Consumer Panel Surveys: 10 Tips

Alongside focus groups and ad-hoc questionnaires, consumer panel surveys are a popular tool for businesses looking to gain insight into what their audiences – and consumers at large – are thinking.

Using pre-recruited and pre-screened groups (or panels) of respondents, consumer panel surveys have specific advantages over alternative consumer research methods. You can control your audience, for example, and return to the panel over time to collect data repeatedly from the same set of people. This lets you dig deep into highly targeted audiences – and, crucially, compare changes in attitude over time.

Yet, as with all research methods, to get real value from consumer panel surveys, you need to refine your technique. That’s why, in this article, we’re sharing 10 tips and best practices to get the most from your consumer panel surveys. We’re talking how to shrewdly recruit and screen respondents, how to ask questions that elicit authentic opinion, and how to use the right analytical methods to yield deep insight.

Note: Don’t forget, Pollfish has created a new consumer research methodology that works a little differently. One that fulfills the promise of consumer panel surveys, while keeping bad data to a minimum – and making access to deep insights easier and more efficient. 

Where to Start with Consumer Panel Surveys?

Consumer research can deliver highly valuable insight for your business. Whether you want to test how an ad campaign will land, or get feedback on a new product line, opinion from authentic consumers – and potential customers – can be priceless.

Yet, you can’t just go and fire questions around at random. This won’t tell you anything of value. Rather, thinking through who you are asking and why is crucial before you start. 

Here are three tips on how to get that done.

#1: Gain Clarity on Your Research Problem and Goal

We’re starting with the basics, yes. But a true understanding of what you want from your research is an essential element that too often goes overlooked, particularly by businesses conducting research themselves. If you want insights that have value for you, you need absolute clarity on your larger goals.

Do you want to boost sales, refine a specific product, or determine brand positioning? A robust research objective will help you sculpt questions that deliver sharper insights – and save you time asking questions that don’t deliver value.

#2: Refine Your Research’s Scope and Scale

Once armed with a research objective, you now need to assess the desired scale of your research. This can get a little complex – but it ultimately shapes the reliability of your research.

Of course, the ideal survey would consult 100% of your target population – whether that’s marketing graduates in Scotland or cyclists among your company staff. While this scale would yield data on which you can surely rely, this is usually not possible in practice. Even if your target population is very small.

However, the further from 100% of the population, the greater the likelihood of inaccuracy. What you need to decide as a business is what margin of error you can tolerate. If you are making high-stakes decisions, you want data you can trust – and the larger the sample size the better.

#3 Source and Speak to the Right People

For authentic insights that deliver value for you, speak to the right audience. You can’t take this for granted. Before you conduct any survey, you need to recruit respondents that suit your target demographic.

This is what screening questions are for. These filter out respondents who aren’t right for you. However, as our resident expert, Jim Theodoropoulos, notes, using screening questions can be difficult to get right. These need to be combined with demographic targeting to narrow in on your audience.

Using Pollfish, for example, if you are seeking mothers who are runners, you can use the “gender” and “number of children” filters, before asking a screening question about preferred exercise. Like this, you can access deeper insights into specific audiences – and keep the professional panelists to a minimum.

Asking Questions that Deliver Real Value

The way you ask questions – and the form in which you receive answers – affects the quality of your data. And avoiding bias and panel fatigue, for example, can make conducting consumer panel surveys a bit of a minefield.

This is why many businesses use options like Pollfish to help design the questions. But if you’re going it alone, below are some tips on how to ask good survey questions and get answers you can trust. 

Note: For more on this, check out our article on how to ask good survey questions.

#4 Ensure Neutrality

Bias is a bad look in consumer research. For surveys to yield data that is actually valuable, they need to tap into the authentic opinion of respondents. Not reflect the prejudices of the researchers. 

That means avoiding leading questions, those questions that direct respondents to specific answers. For example, “How would you rate our award-winning customer service?” has the potential to skew responses by including the qualifier “award-winning”. 

However, even “what problems do you see in our customer service?” distorts the insights you will receive – as it assumes problems, whereas none may have been seen by the respondent.

This also means providing multiple-choice responses that reflect the range of opinion. If “how would you rate our customer service?” is your neutral question, the possible responses need to achieve similar neutrality. “Extremely helpful / Helpful / Neither helpful or unhelpful / Unhelpful / Extremely unhelpful”, for example, keeps the balance between positive and negative options while keeping framing consistent.

#5 Use Strategies to Mitigate Panel Fatigue

During surveys, and particularly during panel surveys, it’s not unlikely that respondents will become bored or disengaged – even though they sign up to participate.

When respondents are asked too many questions, we call this panel fatigue. It’s one of the biggest downsides to consumer panel surveys. Respondents might lean on “don’t know” or they might start “straightlining” – where they choose the same response for every question.

To avoid this, change up your question formats and shuffle questions around. It boosts engagement. It ensures more trustworthy responses. And it ups your completion rate – making data more reliable overall.

#6 Be Specific – and Remember Your Research Goal

Every question in your survey should provide an insight into your audience’s opinions, attitudes, and behaviors. Those that are irrelevant or too vague are not going to deliver the insights you need to make effective business decisions.

In this way, if you are a gaming app, an open question like “what do you like to do in your downtime?” is not focused enough – and it doesn’t direct respondents’ thoughts to gaming. As such, it won’t provide data that will be of use to your research goal.

Understanding Insights from Consumer Panel Research

You’ve asked the questions. Now, you need to understand the responses. 

This will first mean collating recurring answers, identifying patterns, and highlighting anomalies – a process that can be hugely time-consuming if you are doing it alone. That’s why Pollfish, by the way, delivers automated analytics in real-time. More on that later.

Once you’ve arranged the data, you need to analyze it. Here’s how to get started.

#7 Compare Your Data to Larger Trends

Numbers, let’s remember, are all relative. And getting valuable insights from them is difficult when they are viewed in isolation.

Your survey tells you 80% of respondents said that they were happy with your customer service. But for this to be valuable to you, this needs to be put in context. How does this compare, for example, to other businesses of your size in your industry? Or to your results from last year?

Larger trends are your friends – and they help make sense of your customer panel survey. If this is your first survey of its kind, let these results be the benchmark from which you understand future developments. If you have numbers from previous surveys, use these to assess your progress.

#8 Cross-Tabulate – and Dig Deep

Similarly, when you are reading your results, the big picture isn’t always the most useful view.

Say that 65% of respondents in a general population survey said they found your new ad campaign funny or very funny. That might feel like a win. However, these numbers could be disguising insights of more specific value.

That’s what cross-tabulation is about. Using crosstabs can let you see how different data fields overlap – and this is an indispensable tool for analyzing your data. For example, maybe the ad landed much less effectively with women. Or the joke was completely lost on 21-30-year olds. 

This matters. There may be an insight you need to address. You just need to dig deeper to find it.

#9 Keep on Researching

In a world in which business competition is sharp, a single survey won’t do. As a result, there’s one tip that needs to be stressed: keep returning to your audience to do more research.

The unique value of consumer panel research is that going back and asking questions is easy. Your panelists are there, pre-recruited, ready, and willing to respond. Make use of them. There is really no such thing as too much data.

#10 Use Pollfish to Make Deeper Insights Easier

At Pollfish, we’ve moved beyond the traditional consumer panel survey. We’ve developed a research methodology that makes gathering insights into consumer opinion easier. It makes recruiting panelists and understanding insights faster, more cost-effective, and less labor-intensive, too. 

For example, our consumer research platform sources insights from over half a billion engaged consumers, connected via our network of 120,000 app providers. We screen every participant based on pre-collected demographic, biographic, and behavioral data – as well as your screening questions – to take the stress out of finding the right audience.

Pollfish makes setting research objectives and framing questions easier too. We give you some of the most common research goals and pre-crafted question templates to choose from. To keep bias to a minimum and ensure participant engagement, all you have to do is select.

Finally, analyzing data is easier with Pollfish, too. Our advanced data tools let you filter your findings in real-time. All admin is taken care of, so that you can dedicate yourself to visualizing your insights – and ultimately putting them to work.

Frequently asked questions

What is a consumer panel survey?

A consumer panel survey is a type of survey that is conducted at set intervals over a period of time with the same group of respondents. The time period can range from days to years.

Why are consumer panel surveys used?

With interviews conducted over a period of time, consumer panel surveys are ideal for understanding how the thoughts, feelings, or behaviors of people evolve over time.

What are the benefits of conducting a consumer panel survey?

One of the primary benefits of consumer panel surveys is that they can be used to gain deep insights about a highly targeted audience. In addition, individual surveys are easy to deploy because the respondents have already been identified.

How are screening questions used?

Screening questions are used to identify the appropriate audience or panel for a survey. Combined with demographic targeting, these questions can help identify the best audience for a survey.

What is panel fatigue?

Panel fatigue occurs when survey respondents become bored or disengaged from the survey process, resulting in mediocre or inaccurate data.