Diving into the Cognitive Testing Survey to Reduce Research Bias

The cognitive testing survey incorporates a critical tactic used in market research used to reduce bias. Particularly in reference to reducing bias originating in survey questions, this kind of survey allows researchers to develop questions that avoid cognitive bias and other biases.

No one is immune to biases, including researchers who form the questionnaire — the heart of a survey and its broader campaign. Given that both researchers and their respondents are prone to survey bias, researchers must find ways to eliminate the bias, and this begins with the questions themselves.   

Cognitive testing allows researchers to build better survey questions to fend off cognitive biases and thereby create more reliable and accurate campaigns.

This article explains the cognitive testing survey, its broader concept, how it improves survey research and market research at large, along with how to create one.

Defining Cognitive Testing

Cognitive testing is a method in market research used to test how comprehensible survey questions are to respondents in order to improve them. It is therefore an optimization method used so that researchers can foster effective surveys for market research campaigns. 

It should not be confused with cognitive interviewing, a practice used by the police when dealing with witnesses.

During a cognitive test, researchers probe participants on their cognitive processes during the question-answering portion to find problems with those questions. This way, researchers can form better questions for their survey studies

This kind of test is designed to improve the quality of answers by limiting cognitive bias, which is a kind of preconceived notion stemming from existing, perceived to be existing or lack of information that skews perception and leads to misconception.

However, it does not solely aim to limit cognitive bias; rather, cognitive testing helps build inclusive research design, a kind of design methodology in which researchers consider the complete scope of respondent diversity in order for respondents to complete a survey accurately and with ease

The kinds of diverse factors researchers must consider include ability, language, culture, gender, age and other differences. The goal of cognitive testing and its own survey is to make a market research survey more inclusive and minimize research bias.

Understanding the Cognitive Testing Survey

As its name suggests, the cognitive testing survey is a kind of survey designed as a tool to perform cognitive testing. A part of inclusive research design, this kind of survey seeks to weed out several issues within questions that would otherwise incur miscommunication and biases on the part of both the researcher and respondent. 

Particularly designed to improve the experience and accessibility of the respondents, this kind of survey works to avoid biases and inaccurate information from the respondents. This survey is specifically designed to avoid cognitive biases in market research, such as in other surveys themselves, but it can be applied to any form of research in which responders are questioned. 

The cognitive testing survey tests respondents questions in order to avoid the following:

  • Confusion in survey questions or answers
  • Misinterpretations of questions or answers
  • Difficulty in properly answering questions
  • Feelings of irrelevance when in fact questions touch upon relevant matters to respondents

Poised to make surveys more comprehensible, this survey type functions as more of a test on researchers’ upcoming or prospective survey studies rather than a study on consumers. It shows researchers the flaws within their own questioning and formatting. 

As such, when researchers create a cognitive testing survey, they can test questions to various campaigns and survey types. In this regard, this is one of the most thematically diverse survey types.

The Pros and Cons of the Cognitive Testing Survey

The cognitive testing survey serves as a useful test run of a survey before it is launched. However, it does not cover all bases when it comes to inclusive research design, or understanding respondent perceptions.

As such, market researchers ought to understand its advantages and limitations before running such a survey. The following lays out the various benefits and drawbacks of this kind of survey:

The Pros 

  1. It provides a direct method to understand how respondents interpret the questions and instructions you aim to use in potential surveys. 
  2. It allows you to understand the various meanings associated with your questions.
  3. It exists as a pilot test or test run, so that you avoid running several surveys.
  4. It allows you to avoid guessing which respondents answered accurately and which didn’t due to a misunderstanding. 
  5. It shows you how different respondent conditions or abilities affect respondents’ faculty in answering the questions.
  6. It is a far more proactive approach to inclusive design and avoiding biases than merely adjusting demographic questions.
  7. It helps ward off would-be biases from the respondents should the cognitive testing survey not have been run.
  8. It prepares you to create questions that are better suited to be understood by your target market. 
  9. It works to prevent respondents from answering with “don’t know” or “not sure” in future surveys.
  10. It helps form standardized questions and reliable answers.

The Cons

  1. It doesn’t tell you all of the thought processes that occur among respondents when they answer the questions.
  2. You won’t be able to see if respondents understood particular terms in the question or the question itself. 
  3. You won’t know if respondents accurately recall the kind of information needed to answer a question.
  4. It can’t relay whether respondents can fit their desired answer into the provided format or answer categories.

How to Create a Cognitive Testing Survey

Although the cognitive testing survey has its share of limitations, it can still explore the items listed under the above “cons” section. This will rely on the way that researchers frame their survey questions.

This is to say that some questions will be one-offs, while others will require follow-up questions to understand the respondents’ interpretations and opinions in greater depth

When executing a cognitive testing survey campaign, researchers can compare data produced over time by the same and different segments of a target market. 

First off, the survey questions in this type of survey are delivered via the technique known as “mode mimicking,” a method that asks questions in a mode as close to the main survey (the one you’re testing for) as possible.

This means the format of the questions and the questions themselves must be virtually identical with the survey you’re testing. If, for example, you intend your actual survey to use a Likert scale question, you should test it in the cognitive testing survey as well. 

In this survey type, it is more crucial to use advanced skip logic, that is — the automattic function wherein respondents are routed to appropriate follow-up questions based on their answers to a previous question.

This will allow researchers to understand their respondents’ comprehension and interpretations to a granular extent, so that they can perfect their surveys and avoid any cognitive biases.  

Furthermore, the cognitive testing survey naturally fosters qualitative market research, as it must also include open-ended questions so that respondents are at liberty to explain themselves. After all, the point of this survey type is to optimize the survey experience and avoid any confusion. 

There are three main techniques used in questioning the respondents in a cognition testing survey:

  1. Observation Questions: This involves questions that seek out hesitation and confusion 
    1. Particular to the cognition testing survey are requests for clarification.
    2. As such, researchers must include this as an option, unlike with most survey types that simply include a “don’t know” option.
    3. Example: An option with an open-ended answer that allows responders to explain what they don’t understand.
  2. Think Aloud Questions: This involves respondents expressing their thought processes in writing.
    1. As such, these questions must include an option for an open-ended response. They can also ask multiple-selection questions that deal with thoughts.
    2. Example: “Please explain what immediately comes to mind when you see this kind of question.”
      1. “Please explain how you think about answering this question.”
      2. “Does this question create an instant answer or does it take some time for you to answer it?”
  3. Probing Questions: These questions inquire into the way respondents perceive a particular word or phrase, the scope of time they think about when recalling information and whether they believe a question is missing any options. 
    1. These questions can be asked one after the other by way of skip logic or after the entire test questionnaire. 
    2. Example: “What do you think of when you see this word?”
      1. “What time period in your life do you think about if you were to answer the following question?”
      2. “Which phrase would make you think of a physical home?” (Multiple-choice)

Creating Inclusive and Bias-Free Market Research

In summary, the cognitive testing survey takes the concept behind cognitive testing and reimagines it as the basis of the survey. Running this kind of survey across multiple segments of a target market saves money, time, resources and the manpower required to carry out the study, analyze it and test it further.

Even the sharpest of researchers will understand that no survey will be interpreted exactly the way it is intended to or in the same way. As such, researchers should preface their survey campaigns by running the cognitive testing survey. 

This will stave off various biases, as it teaches researchers which question formats and wording to avoid, and which will be most easily and accurately interpreted by respondents. 

Given that target markets may include those with disabilities, those with particular interpretation patterns and simply those who think differently, it is key to design surveys to their favor and understanding. Doing so will ensure you foster inclusive market research along with stamping out biases. 

The key to carrying out this kind of survey is by choosing the proper online survey platform. Such a tool should make it easy and practical to run numerous surveys, test various segments of a target market, cull the information with relentless quality and retrieve results quickly.