How to Avoid Survey Bias in Your Market Research

Survey bias is one of the biggest roadblocks in market research. Although online surveys are powerful agents in gathering primary research, they are susceptible to unreliable and sometimes inaccurate results. This is due to survey bias.

Unfortunately, survey bias is inevitable, as several aspects can contribute to inaccurate results, many of which are out of the researchers’ control. However, understanding the main types of biases that can arise is critical for avoiding them.

While no one can fully weed out bias from surveys, there are certain measures researchers can put into practice to fend them off. This article illustrates the makeup of survey biases and how to avoid them.

For more information on survey bias, read the Pollfish Ultimate Guide to Remove Sampling Bias and More. This article will cover certain biases not mentioned in the guide. 

Defining Survey Bias 

Survey bias is an encompassing term that does not simply refer to results that are unrepresentative of a studied population. Survey bias is a general term for a variety of biases that influence respondents to provide dishonest or inaccurate answers. These invalidate the findings of a survey.

There are two main types of biases that occur in survey research: survey errors and response bias.

Survey Error

This type of error is rendered when researchers use faulty means in creating a survey, i.e., its screeners, quotas, questions, etc. Referring to the survey-production process, this type of error can also occur in administering a survey and even in post-survey analysis. 

The latter is known as researcher bias. (More on this in a few sections below). 

The survey error is a type of bias that lies entirely on the researchers’ end, as they are responsible for producing the surveys and their processes. As such, it can be avoided via actions taken on the part of the in-house or external research side of the survey campaign. 

Response Bias

Unlike the survey error, which comes from the researcher side, response bias occurs from the survey participants, i.e., the respondents. Response bias occurs when respondents are influenced into providing answers on a survey. A general term, this type of bias can result in a variety of inaccurate responses. This bias can appear either deliberately or subconsciously from the respondent. 

Respondents can be influenced to answer questions in a particular way or they may exhibit reluctance to provide accurate answers themselves. In this regard, there are many types of survey respondents to look out for. 

As a market researcher or survey maker, you ought to ensure you’re implementing efforts to reduce the many varieties of these two main survey biases.

Survey Errors

Construed from the research end of the survey research process, survey errors can occur in the questionnaire, the screener, the distribution/collection stage, or from the interpretation of the researchers themselves. Here are the key survey errors prone to occur on the research side and how to avoid them:

1. Sampling Bias: 

This relates to a bias that occurs in the respondent selection process, specifically when a survey sample (group of respondents) is not chosen at random, or not completely at random. This leads to under or overrepresentation of a certain segment of your targeted population, as only certain types of respondents are taking part.

How to reduce sampling bias: Researchers need to employ several methods of distributing their surveys, so that respondents are as randomized as possible. This includes making mention of it on emails, websites, social media and even physical locations. Alternatively, researchers can use an online survey platform, one which features a large network of publishers. This will ensure that the survey is exposed to thousands, if not millions of people online. 

2. Survey Scope Error:

This error occurs when researchers leave out critical questions needed to fully complete the research around a topic. This results in incomplete findings that require follow-up surveying, which is only possible if researchers use a survey panel. Otherwise, despite deploying surveys to the same market segment, you won’t receive answers from the same respondents due to the nature of random device sampling found in many online surveys. Survey error renders survey research to lack critical aspects of a topic or inquiry.

How to reduce the survey scope error: Create a list of questions and answers before setting up your survey. Consider the answer options you give, will any of them require further probing? If so, create question paths to send respondents to the appropriate question based on their previous answer. This allows you to probe deeper into a topic or subtopic. Follow-up questions can be open-ended. If you require too many questions, making for a long survey, consider breaking up the survey based on subtopics as online users are unwilling to answer lengthy surveys, no matter how well-built the survey platform you use. 

3. Order Bias:

This refers to the bias that arises due to an ill-conceived order of questions and answer choices. This is to say that this bias occurs when questions asked early in the survey affect how respondents answer questions later on in the survey. Order bias can exist in two varieties. One is the assimilation effect, where the response to a concluding question is based on the former questions asked. 

Ex: When conducting a customer satisfaction survey, researchers ask several questions on the CX of a brand, with the final question on the overall experience. The questions leading up to the final question all deal with similar aspects of the experience, therefore influencing the response to the final question. In this case, the response to the final question would be similar to the former questions. Asked on its own, however, it would likely receive a different response.

The second type of order bias is the contrast effect, wherein the response to a concluding question is not similar to prior questions, but more severe in comparison to them than it would be if it was asked earlier on or by itself.  

Order bias can also occur due to the types of answers the researcher provides in multiple-choice questions, as respondents prefer to answer with the first few responses. 

How to reduce order bias: Randomize question and answer options. It also is key to reduce the number of scale questions per survey. When too many such question types are used, you are bound to create the assimilation or contrast effect. Ask questions that better engage respondents; include multiple and single-selection answer questions, questions with visual elements and open-ended questions. 

4. Purpose Creep Error:

This refers to the error that emerges when researchers include unnecessary items in a survey, typically for the sake of obtaining an exhaustive data collection. This action is called a purpose creep. Essentially, this kind of action creates superfluous data, the kind researchers won’t need to use when analyzing or presenting their findings. 

Furthermore, in an attempt to avoid discarding data, researchers may study unneeded aspects of the survey, such as demographics that aren’t needed or tidbits about the target market that are not entirely relevant to examine. Asking unnecessary questions can also negatively affect the respondents’ experience, leading them to answer untruthfully or leave surveys incomplete. 

How to reduce the purpose creep error: Consider the key demographics, screening questions and topics you need to base your survey on. When you prepare questions, look them over before launching the survey to see if there is anything redundant. While some aspects may seem interesting, they may not be entirely pertinent to the survey and overall study.

Survey Biases

Survey biases occur within the respondents themselves, commonly when they are influenced to answer or behave in a particular way. It can also exist from within, meaning that the respondents are inclined to misunderstand questions or easily get bored. Here are the key survey biases common on the respondents’ side of survey research and how to avoid them:

1. Acquiescence Bias: 

Also called agreement bias, this bias occurs when respondents gravitate towards positive or agreeable answers. In this bias, respondents will exhibit the propensity of frequently choosing answers with positive associations. They do this as these answers feel like the correct choices. This kind of bias is more prevalent in Asian cultures, as a study found.

This bias also rears its ugly head when respondents feel tired and thus answer questions without applying any thought to them. 

How to reduce the purpose creep error: Create questions with answers that don’t allow respondents to make positive or negative associations. As such, researchers should avoid using questions that ask if the respondent agrees with an idea. For example, instead of asking:

Do you agree with the following statement? “I found the new feature easy to use.”

• Agree
• Disagree

Ask, 

What do you think about the new features in terms of ease of use?

• I think the feature is easy to use. 
• I think the feature has to be studied.
  

2. Prestige bias:

Prestige bias is tied to respondents’ social desirability, as it deals with responses made specifically to be seen positively. This can manifest in instances where respondents are asked about their income or their associations with notable actions. In these cases, respondents exaggerate to make their circumstances appear more socially desirable. Many times, this bias occurs indeliberately, as respondents may recall memories in a way that’s more favorable to them as a subconscious way to protect their reputation. 

How to reduce prestige bias: Screen your survey participants for their knowledge of certain things relating to the study. Also, be sure to preset the audience with demographics that fit a certain level of education. Ask questions in a way that avoids cheating positive or negative associations. Also, it will help to add a disclaimer at the beginning of your survey, that claims it is a judgment-free study and solicits respondents to answer honestly. 

3. Demand Characteristic bias: 

This bias develops when a demand characteristic is present. This phenomenon denotes cues in the survey that unintentionally influence how responders answer questions. A demand characteristic can take place if the researcher gives away the purpose of the survey or study. As such, the main purpose of a survey should not be overt in the setup of a survey. When a respondent knows the purpose of a study, they may purposely provide answers to influence the study, especially if this will benefit them. Ex: if a survey is conducted to determine a municipal law, the respondent may provide answers that would make the outcome of the survey favorable to them.

How to reduce demand characteristic bias: Researchers should keep the main purpose of a survey private. As such, they can forgo a welcome page that respondents would otherwise see before taking the questionnaire or answering the screening questions. The wording of the questions should not make the purpose of the study too obvious; therefore, researchers can use analogies and keep certain topics vague. 

4. Random Response bias: 

This bias comes into being when a respondent does not know how to answer a question, but does so anyway, creating inaccurate results that are random at worst. In this scenario, respondents guess an answer rather than providing authentic information. This bias is also exhibited when respondents don’t read the questions, deliberately randomly answering them to quickly finish the survey. This is usually seen when there is an incentive that requires completing the survey. 

How to reduce random response bias: When forming the questions, consider your target market with them. Will they understand your questions? Will they know how to respond accurately? Avoid asking questions that deal with how respondents perceive others to think or behave. To catch this bias in action, add trigger items, such as an item that is opposite to another, or reverse scoring to catch potential bias.

Staying Clear of Survey Bias with the Right Online Survey Platform

While providing a potent channel for primary research, surveys are bound to incur bias, whether it manifests in survey errors or survey bias. Often, survey errors — those which are made on the part of the researchers, generate and influence survey bias — which occurs on the part of the survey subjects, i.e., the respondents. 

Therefore, researchers should always scrutinize their survey efforts, both in the screening and questionnaire portions of the survey they’re launching. An online survey platform can help minimize errors as a potent one provides randomization of respondents and many question-and-answer option setups. 

While certain biases are unavoidable, researchers ought to carefully assess the online survey tool they intend to use for their survey research.