3 Survey Data Quality Issues Sabotaging Your Research (& How We Fix Them)

When conducting market research, data quality is the most critical factor. While that might sound obvious, it’s often not given enough consideration.

Even with a meticulously designed survey — solid logic, good flow, and bias controls — if respondents are fraudulent (e.g., bots) or rushing to finish, the data is flawed. This can lead to costly mistakes and damage credibility with leadership.

In this article, we’ll cover:

• 3 critical survey quality issues
  • Issue #1: Bots posing as real respondents or survey farms (fraud caused by a network of real humans), contaminating data with fraudulent responses
  • Issue #2: Respondent profiles and demographic information may become inaccurate over time, even if initially true
  • Issue #3: Respondents providing low-quality, rushed, or contradictory answers
• How Pollfish, through our parent company Prodege — the world’s largest single panel owner — combats these issues via first-party panel ownership and the robust, multi-layered data quality safeguards we have in place.

Issue #1: Bots posing as real respondents or survey farms, contaminating data with fraudulent responses

The Problem

Survey fraud can be divided into 2 broad categories:

• Bots – Sophisticated automated scripts, or “bots,” masquerade as genuine respondents. These bots may originate from click farms or individuals in various countries attempting to conceal their true location and pose as legitimate participants from target markets like the US.

• Survey Farms – Fraud caused by networks of real humans who register for multiple panelist accounts and attempt to complete as many surveys as possible while avoiding detection. Members of these networks pay a portion of any survey earnings to the owners of the websites where they go to learn the latest tricks to distinguish their location (e.g., using a legitimate IP address to mask their own) and how to create multiple accounts and qualify for surveys.

Our Solution

Prodege has an in-house, 40-person Panel Support, Security and Compliance Department which continually innovates and stays one step ahead of survey fraud. Some of the quality controls and processes we implement include:

❇️ Verifying Respondent Identities Through Diverse Activities Beyond Surveys

Many panels simply have people log in, answer surveys, and get paid without tracking any other behaviors. In contrast, through Prodege’s flagship communities like Swagbucks, members can earn rewards not only from completing surveys but also by playing games, shopping online, uploading supermarket receipts, and more.

These additional activities provide valuable data points to verify respondent identities and build out fuller profiles. For example, if someone uploads a unique shopping receipt image, it confirms they are a real human as a bot couldn’t replicate this behavior.

❇️ 14-Day Evaluation Period

Before a new panelist becomes eligible to participate in their 1st client survey, they must complete a number of internal quality measurement surveys over their first 14 days on the panel.

Panel members failing to meet our quality thresholds are removed from the panel. After the initial evaluation period, we continue to administer these surveys at random points in a member’s panel lifespan.

❇️ Trust Scores

We’ve developed a proprietary solution leveraging AI plus machine learning, where each member is assigned a “Trust Score” shortly after registering for one of our panels. This score is updated based on ongoing inputs.

Based on the Trust Score, we may deactivate a member’s account, block entry to surveys and/or require additional identity verification (e.g., scan driver’s license).

❇️ Fraud Detection Technologies

This includes multi-level registration verification (e.g., CAPTCHA, Device ID Reputation, 100% Double Opt-In, etc.), tech consistency (carrier/targeting alignment check), single user IDs (no duplicate survey takers), sophisticated anti-VPN measures, etc.

Issue #2: Respondent profiles & demographic information may become inaccurate over time, even if initially true

The Problem

While some survey participants might intentionally misrepresent their demographics from the start, a more common issue is that respondent profiles naturally become outdated as individuals’ circumstances and interests change over time.

This gradual drift can lead to inaccuracies in the data, even when participants provide honest answers during the initial profile setup.

Consider this scenario: A participant who initially reported being single may get married and have children but fail to update their marital status and parental information in their profile. Consequently, they may be incorrectly included in or excluded from surveys based on these factors.

The challenge for survey platforms lies in ensuring that participant profiles remain accurate and up-to-date, not just at the point of creation but also throughout the respondent’s lifecycle. Failing to do so can lead to skewed survey results, misallocated targeting efforts, and ultimately, flawed insights that misinform critical business decisions.

Our Solution

❇️ We go beyond relying on self-reported information by leveraging behavioral data. By analyzing purchase receipts and websites visited, we can identify potential mismatches between a participant’s claimed profile and their actual behavior.

For instance, a participant may initially report being single and not having any children. However, if we observe a sudden and consistent increase in purchases related to baby products such as diapers, formula, or child safety equipment (through purchase receipt data), our system may flag this profile for potential updates to parental information.

Our ability to leverage behavioral data for demographic validation is made possible by Prodege’s unique position as a company that owns and maintains its consumer communities directly. This stands in stark contrast to most survey platforms, who rely on third-party audience panels (a critical distinction we’ll explore further in a later section).

Issue #3: Respondents providing rushed, low quality, or contradictory answers

The Problem

Even when a respondent is verified as a real person, the quality of their survey responses may still be lacking. This can manifest in several ways:

• Speeding and straight-lining: If a survey is designed to take 5 minutes on average, but a respondent completes it in 30 seconds, it raises serious doubts about their level of attention and engagement. Also, some respondents may simply select the same answer choice repeatedly (e.g., always choosing “B” on multiple-choice questions) without actually reading the questions.

• Low-quality open-ended responses: When presented with open-ended questions that require written input, some respondents may provide brief, irrelevant, or nonsensical answers rather than investing the time and effort to share meaningful insights.

• Contradictory answers: Respondents who are not fully engaged or attentive may provide inconsistent or contradictory responses throughout the survey. For instance, they might indicate they’ve never used a particular product, but later claim to have purchased it within the past month.

• Disengaged respondents: These are real humans with good intentions who on occasion may be multitasking (e.g., watching a movie or cooking while taking a survey), and thus submit lower quality responses.

Our Solution

❇️ Speeding detection: Our system monitors the time respondents spend on each question and flags those who complete the survey significantly faster than the average, indicating a lack of thoughtful engagement.

❇️ Straight-lining detection: We identify respondents who repeatedly select the same answer choice for multiple questions in a row.

❇️ Manual review of open ended responses: We strongly recommend including at least one open-ended question in every survey as an additional quality check. Our dedicated Project Management Team meticulously reviews these responses and removes and replaces any respondents who provide low-quality answers, such as:

• Random characters or irrelevant text.
• Overly brief answers where a detailed response is warranted.
• Answers that do not make sense given the context of the question.
• Any other indicators of poor engagement or lack of effort.

❇️ Red herring questions: We strategically insert our own “red herring” questions throughout the survey, designed specifically to check for contradictory answers and ensure respondents are paying attention.

❇️ And so much more.

Unique Advantage of Prodege: Staff of Research Experts Who Were End-Users of Data in Their Previous Roles

An additional unique advantage of Prodege versus traditional sample suppliers and audience monetizers is that we have a staff of research experts who were end-users of data in their roles at companies such as Hall & Partners, Ipsos, Kantar, Lieberman Research, Nielsen and NPD.

The team’s experience preparing reports and dashboards to deliver insights to enterprise clients is layered into Prodege’s research best practices, bringing a perspective to data quality missing at other sample firms.

The Bottom Line: Pollfish Delivers Reliable Data

Pollfish’s unique approach, which leverages first-party data from Prodege, enables us to implement robust quality control measures and recruit a representative and engaged member base.

As a result, when you use Pollfish, you can trust that you’re receiving the highest caliber of survey responses. As several of reviewers have noted on Trustpilot:

“The most important point to make is that I trust the responses of my survey. My target audience was very specific and I could tell that the responses were legitimately coming [from] subjects in my target audience.”
– Read the full review here

“Each time I use this service I am blown away by the speed with which the responses come in and with the detail which most of them include. You can tell that the respondents take this very seriously. They must really enjoy doing it.”
– Read the full review here

“I had to get some data on a product in a tricky sector. It was proving challenging to attract participants to my study. I was able to quickly get results and some insights to be able to move forward with my work using Pollfish.”
– Read the full review here

“I was very satisfied with the results I got from Pollfish. My company was conducting market research and received very insightful help from consumers who were all verified homeowners.”
– Read the full review here

Note: Click here to explore additional Pollfish reviews on Trustpilot.

Interested in Learning More About Pollfish?

Beyond exceptional data quality, Pollfish offers several other key benefits:

• Pay-per-response pricing model, starting at $0.95 per response – no subscriptions, no long term commitments, just pay for what you actually use.

• Transparent pricing: As you build your survey, you’ll see the total price update in real-time (it’s clearly visible in the upper-right-hand corner, and we don’t hide it). What you see is exactly what you’ll pay, and there are no surprise charges.

• Quick turnaround time, with general population surveys completed in just a few hours and more specialized audience surveys within a matter of days.

• Comprehensive suite of features, including sophisticated questionnaire design, audience targeting, and data analysis capabilities, to support even the most complex survey research needs.

For more details on the advantages of Pollfish and guidance on choosing the best DIY market research survey tool for your needs, refer to this article.

In addition, for details regarding Prodege’s done-for-you market research service offering, reach out here.

Create a Free Pollfish Account Account or Book a Demo

We invite you to:

• Create a free Pollfish account, no credit card required.

• Book a demo call, where we’ll provide a personalized walkthrough of our platform and answer any questions you may have.