How survey response bias can happen (and how you can prevent it)

There are many ports of entry for survey response bias and other types of bias, from the population selection method, sampling method, survey design, medium, question and answer wording, the interviewer, and, in particular, the respondents.

Unless you were to survey everyone in your population, it would be nearly impossible to eliminate survey bias – but we can reduce it. Each of the survey sampling methods has its benefits and unique detractors. Here’s a brief explanation.

First, assuming that you’ve chosen the right population, you need to decide the survey sample size, confidence level, and how you will reach the audience and collect the results. But wait – how do you define the population?

Defining the Population

You could use personas to be precise, or you may just want to reach a population of “teens in high school who have tried drinking”. Sounds reasonable. However– that population definition would rule out anyone who may have dropped out, or who is home-schooled—which would affect your sample. And what about the teens who have tried drinking, but don’t want to respond—they would be underrepresented in your sample.

And, if you are trying to survey the population of “teens who have tried drinking” to see if they are talking to their parents about contraception (vs the consequences of drinking and driving), you may have missed the mark entirely—as this is a small subset of the total population you may want to survey.

There are other forms of Sample Bias:

Sample Collection

Once you’ve established your population, and defined your survey goals, you want to make sure you choose the best method possible for collecting data from your sample. You could choose a from a variety of collection methods, which are either probability sampling or non-probability sampling—and all have pros and cons.

There are different methods to collect data including traditional methods like telephone, in-person, and mail-in, surveying. More modern methods include online surveys via email, websites, social media, and mobile.

Telephone surveys were very popular for quite some time, but over the years have developed several downsides—particularly for respondents as this article explains—and have shrunken in popularity and effectiveness.

In-person interviews, including focus groups, have the benefit of obtaining feedback directly and immediately—however interviewer bias and other social factors such as moderator bias, biased answers, and biased reporting can set in.

Mail-in forms rely on accurately targeting the respondent and validating that they are the ones in the household that completed the survey; it is also subject to non-response bias.

Online surveys have many advantages, but since respondents complete them in private, they are subject to interpretation, and therefore question length, wording, style—even format and coloring can affect responses, and survey completion rates.

Social media surveys have inherent issues with population size and characteristics. They don’t allow for sophisticated survey questions and can be perceived as “fun” or “entertaining” which may not fulfill your survey goals. Surveys on personal networks have the issue of respondents within the network (such as on Facebook) being somehow affiliated with the researcher.

Many mobile surveys can have issues with the survey design itself, and its ability to keep the respondent engaged on the mobile screen. Many surveys are mobile-unfriendly, and even mobile-friendly designs may fail to capture full audience participation.

Sample Size

What’s the right sample size? A lot of that depends on the population size, the ability to reach them, and your budget. However, you need to make sure the sample size is representative of the population or segment (strata) from which you are trying to gain an understanding.

Under coverage happens when not enough members of the population are adequately represented. Non-response bias is when potential survey respondents are unwilling or unable to participate.