Survey Panels Vs Organic Sampling: Which is Better for Market Research?

Survey Panels Vs Organic Sampling: Which is Better for Market Research?

Survey panels and organic sampling are two of the major methods used to collect survey data. Since getting survey responses without the assistance of software is an almost impossible feat, these two methods have reigned supreme.

Both of these survey response mechanisms are unlikely to wane anytime soon, due to the prevalence of online surveys. As a matter of fact, the online survey software space has risen by 8% between 2015 and 2020.

With online surveys granting market researchers and marketers a large sweep of survey types to conduct, the point of contention becomes: which type of survey response collection data is better, survey panels or organic sampling?

This article will explore both forms of data collection so that you can objectively decide which to opt for in your survey research campaigns. 

Defining Online Survey Panels

Survey panels, also called online panels or research panels, all denote a data collection method in which responses are collected by way of pre-recruited and pre-screened respondents who agreed to take part in a survey.

This method helps businesses in that it assures them that a group of people will take their survey; usually, members of their target market are called upon to take the survey. 

There are a number of ways survey panels can be assembled. One such method involves mail-in recruitment, which has significantly declined in the digital age. Another relies on phone calls via Random-Digit Dial (RDD), a method in which respondents are chosen to take a survey from random telephone numbers. 

When it comes to recruiting survey panels digitally, they are garnered through an opt-in format such as a signup page or through an email invite that routes users to the survey page.  All of these recruitment manners have low participation, as few people opt into panels.

That is why some brands and market researchers resort to compensating their pre-screened panelists.

The Pros and Cons of Survey Panels

Now that you understand the basic methodology behind survey panels, you ought to consider their pros and cons before fully forming your opinion on whether to use them or not. 

The pros of online panels:

  1. They provide a network of respondents for continuous survey participation. This is especially useful if you need to conduct longitudinal studies. 
  2. They are inexpensive and create fast studies. 
  3. Returning to the same respondents, allows you to detect changing opinions over time, allowing you to see how your target market changes its opinions.
  4. They allow you to create informed custom polling questions based on previous research.

The cons:

  1. Repeated survey participation causes panel fatigue, a term denoting the decline in the quality of survey data, due to the boredom or exhaustion of a panelist. This creates them to inaccurately provide responses, either due to skipping questions, ticking off the “don’t know” option or rushing through a survey. 
  2. They occur in non-organic (unnatural) environments — inauthentic environments create inauthentic responses. This is because a survey’s environment can affect its’ respondents’ mindsets.
  3. Web panels gather respondents either on desktop or mobile, creating scenarios in which participants are dependent on device types. As such, the survey experience is not very adaptive. 
  4. Panel conditioning: Repeated survey participation can change respondents’ true attitudes, behaviors and knowledge. This makes it difficult to differentiate between actual changes and changes in reporting behaviors.

Random Device Engagement (RDE): A Precursor to Organic Sampling 

Before you analyze the organic sampling method, you should understand what makes it tick. That is because organic sampling is reliant on the delivery structure known as Random Device Sampling (RDE).

This framework implements intent-based behavioral targeting, typically used by advertisers, to narrow down random respondents in a digital setting, such as on websites, mobile sites and mobile apps. 

Thus, it provides a solution for randomization and capturing the correct audience. 

Random Device Sampling works by tracing the unique IDs of respondents, which are used to track them across devices. RDE, therefore, institutes a mechanism that is both random and organic. 

Organic Sampling Defined

Also called random organic sampling, this method refers to an RDE-based response collection method in which a survey is deployed randomly to users who are already in apps and other digital spaces.

Since surveys are randomly transferred, this method allows respondents to take the surveys while they are in their organic environments. These are the spaces that users spend time in organically, meaning they chose to take part in those environments, rather than being taken there via a promotion, incentive or signing up at a web panel.

Organic sampling works by giving optional invitations (or call-outs) to users in organic settings, so that they would partake in quick surveys. These invitations (along with the surveys themselves) are natively integrated within the digital environments (ex: apps) that the users are in.

This makes several benefits possible. 

The Pros and Cons of Organic Sampling

Powered by Random Device Engagement, organic sampling has many upper hands. However, like the panel survey method, organic sampling also presents certain disadvantages that you should carefully consider. You ought to weigh them against one another before deciding whether or not to use them. 

The Pros of Organic Sampling

  1. An organic environment allows you to avoid panel conditioning, allowing you to extract genuine responses from participants.
  2. It targets respondents’ unique IDs so that they can be traced even while shuffling between devices, preventing the same participants from taking the same survey twice. 
  3. Using respondents’ IDs allows you to create respondent profiles, which build an overview of the respondents’ behaviors demographics, which are critical data to possess.
  4. In turn, the respondent profile helps prevent fraud, as multiple accounts or bots won’t able to submit their surveys. 
  5. It yields high response rates (higher than RDD and survey panels).
  6. It creates a seamless UX across all device types, rather than being tied to just one with little wiggle room. 
  7. It provides vast coverage, as RDE integrates natively with a bevy of digital and mobile platforms. That entails greater accuracy. 

The Cons:

  1. It relies on non-probability sampling, wherein some of the members of a population have been excluded and this amount cannot be calculated, which limits how much you can determine about the population from the sample.
  2. It will include biases based on the populations you have preset to include in your sample, even if you assign quotas.
  3. Organic sampling and RDE are at the mercy of the websites and apps that they can be integrated with. It is possible that your targeted sample pool visits apps and other digital places that your RDE survey isn't integrated with.

Which Sample Pool Method Reigns Supreme?

While random organic sampling has made a mark in the survey realm, due to Random Device Engagement, it still faces its rivalrous counterpart: survey panels. 

Other forms of survey sampling methods have been on the decline, yet survey panels are still in use. While they may appear to have fewer advantages and more disadvantages, survey panels still provide value to researchers. 

As such, it is entirely up to you to decide which survey sampling method is best for your market research needs. Random organic sampling is our survey sampling method of choice, as it continues to rise above survey challenges, provides solid results and does so in a short span of time.

Frequently asked questions

What is an online survey panel?

Also called online panels or research panels, an online survey panel is a way of repeatedly collecting data from pre-recruited respondents via digital surveys.

What are some of the benefits of an online survey panel?

Online survey panels allow companies to quickly and inexpensively collect data from a group of people since the recruitment and screening process only needs to be performed once. Since responses are collected from the same group over time, they are able to show if and how opinions or behaviors change over.

What is panel fatigue?

Panel fatigue is a concern associated with survey panels. It occurs when panel participants become bored or tired of replying to surveys. This can result in the overall inaccuracy of the data collected from that participant.

What is organic sampling?

Organic sampling, or random organic sampling, refers to the process of distributing surveys to a random group of respondents. The respondents are typically sourced from websites or apps where they are given the option to participate in a survey.

What are the benefits of organic sampling?

Organic sampling can help prevent some of the pitfalls of survey panels including panel fatigue and panel conditioning. Organic sampling also tends to yield higher response rates and can help prevent fraud or data errors.


All About Panel Surveys: Using this Method for Market Research

All About Panel Surveys: Using this Method for Market Research

Panel surveys (not to be confused with survey panels) are used in survey research as part of longitudinal studies. The purpose of this kind of study is to reap continual observations on different variables on the same sample pool over a period of time. 

The variables include consumer feelings, attitudes and opinions regarding a number of matters. As such, the surveys are conducted across the same sample pool, i.e., the survey panel.

This way, the opinions of the same panelists can be monitored, as opposed to bringing in new respondents (even if they are part of the same target market). Panel surveys are conducted in waves (sequences of surveys) to measure changes in consumer thoughts and behaviors. 

There are many key aspects of panel surveys; this article will cover all of them to help you decide on whether your business needs to use the panel survey method.

How Panel Surveys are Conducted

First off, panel surveys are but one type of longitudinal studies, which also include retrospective studies and record linkages.

As their name indicates, panel surveys use surveying as the data collection method. This data is mainly collected through the aforementioned survey panels, a method in which a pre-recruited pool of respondents agreed to take part in your survey.

However, panel surveys can also be collected through other online survey methods, mainly DIY survey platforms. These allow you to target a specific set of respondents, but unlike survey panels, they are not conducted among the same exact group of individuals. (More on this below).

Since panel surveys are longitudinal, this sampling pool stays the same and is used for repeated studies and observations.

Panel surveys are part of panel research and can take the form of either qualitative or quantitative studies for measuring consumer behavior. 

The Pros of Panel Surveys

It’s worth considering several of the advantages that panel surveys carry when considering this surveying method. You ought to then weigh them against the disadvantages to determine if this is the correct survey route for your market research needs.

  1. Easy Collection of the Sampling Pool: Whether you intend to use an online survey platform or a survey panel, panel surveys make it easy to use a qualified sampling pool. That is because there is no need to look for respondents on an individual basis; instead, you either opt for recruited participants, or the online survey platform you use prescreens respondents for qualification.

  2. Longitudinal Benefits: As it follows the longitudinal survey method, panel surveys allow you to truly access the psyche of your target market. This is because consumers and the general public can change their minds — some more regularly than others. By conducting this survey research, you can stay up to date with any and all changes in attitude, thoughts and behaviors your target market undergoes.
  3. Speed to Insights: Online survey tools allow you to quickly garner all the respondents that match your screening requirements and demographics selections. A survey panel entails that respondents have already opted into the survey. In any case, getting results will be quick.
  4. Optimized Design: Whether you use an online survey panel or organic random sampling via a survey platform, each allows you to create customized questionnaires. The degree of this will change from one survey platform to the next, but the convenience of optimizing your survey to your favor is still present.
  5. Affordability: Panel surveys are relatively affordable. With online survey panels, you should be wise about how you incentivize your panel — this can be done by paying the panelists small amounts per each survey they take, or a larger sum for a set amount of surveys. A decent online survey tool should include several payment plans, with at least one perfect for your budget. 

The Cons of Panel Surveys

While an invaluable form of primary research, panel surveys are not without setbacks. Here are a few of the disadvantages these surveys harbor. Some may appear minor, while others are more considerable.

    1. Panel Fatigue: Predominantly found in survey panels, this occurrence refers to the reduction of interest among the panelist(s) when they take part in too many survey waves. This breeds full-fledged boredom and exhaustion, leading to a decline in the quality of data. In this case, panelists may take part in flat-lining or other inaccurate answer tactics. There are 5 types of survey respondents like these to look out for. 

    2. Limited Internet Traffic: A problem principally found in online survey tools, as they are under the heel of the publisher sites and apps that deploy the surveys. There may not be enough qualifying respondents in one site or app, depending on who visits it during survey distribution.
    3. Survey Attrition: Piggybacking off of panel fatigue as the most severe case of it, survey attrition alludes to the dropping out of a panel. Panelists may experience a negative UX with the panel, thereby attriting at any wave of the study. These respondents are especially difficult to replace in survey panels, as they’ve already provided some crucial data, so there will be a void when it comes to conducting further surveys based on their responses. Some panelists do not drop out permanently, as they may return to the panel at some later time. With online surveys, this is relatively non-existent, as new responders are screened in each survey wave.
    4. Inclination towards Bias: Particular to survey panels, this occurs when panel respondents have taken too many surveys, thus becoming programmed to the way your surveys are set up. As such, they become less like genuine research subjects and more like trained survey-taking professionals. They may not put forth too much effort or thought into further survey waves because of this; rather they will clamber to get out of a survey as soon as possible.
    5. Respondent Identity Fraud: Especially common in online survey tools where you cannot validate the identity of your sampling pool, this can lead to respondent fraud. Respondents may lie about their age, employment and any other type of demographic identifier. Your data will suffer as a result. Survey panels may be immune to this if you choose your panel via face-to-face interviews, or if you know any member of the panel.

Panel Surveys: Uses and Applications

Panel surveys have various applications that you can put to use for your brand. Whether you’re looking to innovate an existing product, develop a new one, understand how your target market responds to current affairs, how they respond to certain communications or virtually anything else you need data for, panel surveys are useful ind=struments to leverage. 

Here are several specific uses and applications for these kinds of surveys.

  1. Detect common customer behaviors in relation to purchasing, clicking on an ad, or interacting with your brand in any other way (digital or physical).
  2. Analyze the costs of a product, service or subscription.
  3. Predict sales for particular campaigns and seasons.

  4. Understand how to use current affairs in brand messaging (including the knowledge of which subjects and rhetoric may be too sensitive for your target market).
  5. Monitor trends in how customers buy from brands.
  6. Find how recent and distant events have affected the attitude and opinions of your target market.
  7. Segmenting your target market. 

Marrying Survey Panels and Organic Sampling 

Closing off, it is crucial to reiterate that one of the key differentiators of panel surveys is their ability to incorporate two different types of survey respondent collection methods: survey panels and organic sampling (in online survey tools).

These fall into diametric opposition with one another, as the former involves recruiting willing survey respondents (the panel), while the latter uses organic sampling, in which respondents opt into a survey in real-time when they discover it in a website or app.

Although survey panels are the chief method to conduct panel surveys, due to the repeated nature of observation, both of these survey methods have the ability to sustain the longitudinal study method of panel surveys. 

Survey panels are the primary collection method of responses, as panel surveys and other longitudinal studies involve studying the same group of respondents

Survey panels do just that. However, survey tools that use organic sampling, can also study the same group based on their demographics and conditioned screening questions. The only difference is that the responders in organic sampling won’t be the exact same people, but rather those who fit the categories.

Either way these two methods can both be applied or even used hand in hand. We suggest using a robust online software tool that provides a wide range of features to optimize your survey research. 

Frequently asked questions

What is a panel survey?

A panel survey is a type of longitudinal study that follows the behavior of a predetermined group of people over a period of time. In a panel survey, data is gathered through a series of surveys conducted at set intervals.

What is a longitudinal study?

A longitudinal study is a form of research that makes observations and gathers data over a period of time, rather than at one single point in time.

What variables are examined in a panel survey?

Panel surveys tend to track feelings, attitudes, and opinions to understand how they change over time or in response to certain stimuli.

What are some of the benefits of using an online survey panel?

Online survey panels are cost-effective, can be distributed quickly, allow for easy data compilation to discover insights and offer an optimized design to improve response rate.

What is panel fatigue?

Panel fatigue is a phenomenon that occurs when a survey panelist participates in too many surveys and then becomes tired or bored with the survey process. This can lead to inaccuracies in survey data.


4 Panel Survey Examples for Market Research

4 Panel Survey Examples for Market Research

panel surveys

A panel survey is a method of research that uses a consistent panel of participants, with the researchers returning to the same people to run surveys or host interviews repeatedly over time. This is also known as a type of longitudinal study. In this post, we’ll look at four-panel survey examples – and analyze the advantages and disadvantages of using panel surveys for your research. 

Note: If you’re looking to do market research without the drawbacks of conducting panel surveys, try Pollfish. Our unique methodology guarantees authentic high-quality data, and it’s a fast and reliable way to survey targeted (yet randomized) people while they’re already engaged on their devices.

Panel Survey Examples: What Do Researchers Use Panels For? 

The best way to explain the uses of a panel survey is by setting out a panel survey example. In this section, we’ll see four examples of when panel surveys can be used to good effect. 

Example #1: Tracking Changes Over Time

One of the strongest examples of a panel survey is when researchers ask the same question(s) to a group of the same people over a long period of time. 

For example, if you were conducting a sociological experiment to understand if (or how) age correlates with wealth, you might set up a survey to ask about income, expenses, and assets. And this survey might be repeated every five to ten years. 

Alternatively, if you run a business, you’ll need to acquire ongoing insights into the market – and into audience behavior. For example, if your gym wear company wants to track the fitness habits of people over the course of the year, a panel can be questioned every three months. You can run surveys with the same questions about how long people exercise for, where they exercise, and how they exercise.

Example #2: User Experience Research

Another common example of using a panel survey is in UX and UI design research. When a company is building an app or a digital product, they want to gauge the customer experience as they go. While usability testing (i.e. tracking how a person interacts with a specific feature) is often done using different methods, qualitative research about design and experience can be done using panels or focus groups.  

The benefit of a panel survey here is that the participants remain constant – which removes a key variable. And you can select panel participants based on their background or expertise, which will allow your pool to answer questions in a way that adds value to your product development process. 

Example #3: Customer Satisfaction Surveys

If you want to gauge customer satisfaction and positive/negative sentiment over the lifetime of their interaction with your product, you can take a select group of long-term customers and ask them questions periodically about your value proposition. These surveys can be framed to reveal hidden sentiments, or can be explicitly focused on asking how satisfied they are with what you offer. 

In this panel survey example, the business would be able to track patterns of customer happiness over time – for example, to see whether features continue to add value months or years after the customer signs up. And you can incentivize participation with gifts, reduced subscription fees, or extra services.

Example #4: Employee Engagement

One final panel survey example is for businesses to track employee engagement and team morale. In essence, here, your staff are the members of the panel – and you might survey them every week, month, quarter, or year. In this case, you’re surveying the same people about issues such as:

  • Do they enjoy their work?
  • Are they meeting their goals?
  • Do they understand the company mission?
  • Are they happy with their compensation?
  • Where can the company improve?

If this is the type of panel survey you’re looking for, there are plenty of tools out there that are specialized for human resources and employee engagement purposes. 

Pros & Cons: The Advantages and Drawbacks of Panel Surveys

One of the primary advantages of a panel survey is that the participants learn to trust the researchers, and therefore may be open to deeper and more truthful answers. And this ongoing relationship can allow researchers to dig deeper with follow-up questions. Another advantage is that with screening having occurred at the start of the research, all following surveys are quicker than starting from scratch.

However, the disadvantages of panel surveys include:

  • Panel conditioning: When you’re surveying the same people repeatedly, previous surveys might influence their responses and/or behavior. For example, if they know a certain question triggers a longer process, they may choose an answer that offers the path of least resistance. Or if you’re asking about a particular activity, like eating donuts, they may eat less (or more) thereafter.

    Note: Furthermore, panel surveys don’t tend to occur within a “natural” setting, and therefore the artificial environment might affect thought processes and responses to questions.
  • Panel fatigue: If a person feels like the pay-off for participating in a panel survey is no longer worth the effort, the quality of their response may drop. This leads to incomplete or poor-quality survey responses, and survey “straight-lining” – i.e. answering the same to every question.
  • Declining participation: Over time, inevitably, participants will drop out of the process – either due to panel fatigue, or other reasons. This will damage the quality and depth of your data. 

And with survey panels, you run the risk of accidentally signing up “professional” panelists – especially when there are cash-based incentives on offer. Though screening should be designed to pick up these people before they enter the process, they’re adept at slipping through the net. 

Finding a Better Alternative to Panel Surveys

With increased pressure on people’s time, survey response rates have been decreasing. It’s harder than ever to guarantee consistent high-quality data, and as conventional survey companies scramble to include as many willing participants as possible, the quality reduces yet further. 

At Pollfish, we don’t use conventional panels. Our unique methodology, called Organic Probability Sampling, is built on something called Random Device Engagement (RDE). RDE polling relies on advertising networks and other device portals to engage people wherever they are. This might be through a smartphone app or a mobile game, where respondents participate in return for an incentive. 

Each screened participant has a unique ID, which prevents them from taking the same survey more than once, weeding out bias and fraud.

When working with Pollfish, you don’t need to use conventional panels – which are slow and subject to conditioning, fatigue, and declining participation. Instead, get authentic insights from our pool of over half a billion people (and growing) – who are already engaged on their device and ready to participate. 

Frequently asked questions

What is a panel survey?

A panel survey is a type of survey method that involves repeated interviews of a group of people over a specific period of time.

What is a longitudinal study?

A longitudinal study is another term used to describe panel survey. It refers to any study that examines the same group of people over a period of time.

What are the benefits of panel surveys?

Panel participants often learn to trust the researcher and may reveal deeper, most honest insights. This type of survey is also a good way to track progress or evolution over time.

What is panel fatigue?

Panel fatigue occurs when the participant no longer feels that the reward for participating in the panel is worth it. If panel fatigue occurs, the participant may not give deep answers, but instead try to answer the questions as quickly as possible.

How can panel surveys influence participant behavior?

When interviewing the same people repeatedly, the answers to previous surveys may influence their behavior by causing them to reflect on activities that they previously gave little thought to. This influence is known as panel conditioning.