Chat with us
  1. aytm Help Center
  2. Knowledge Base
  3. Building Surveys

Articles in this section

See more

Let’s start by defining “data quality control” questions. We prefer to call them quality controls (QCs) as
those questions are not “red herrings” and not “trap questions”. We are not trying to mislead or “trap”
the survey respondents. The goal of the QCs is to confirm the survey participants are carefully reading
the questions and the response options verbiage. QCs in surveys should measure attention, not other
constructs related to memory or specific knowledge. They can improve the data quality of the survey,
however, if they are too convoluted, there is a risk of “over-cleaning” the data and losing valuable
responses. We should also be considerate of the engaged respondents who are the vast majority in every
survey and be respectful of the person sitting behind the other side of the screen by applying
an empathetic approach to QCs design.


Quality Control Questions “Don’ts”

• Don’t add QCs to the screening portion of the survey. The screening portion is the very beginning
of the survey where respondents usually do pay full attention. Also, respondents who are
terminated may affect the overall IR of the survey.
• Don’t include multiple QCs, unless the survey is longer than 20 minutes. Instead, use the QCs in
conjunction with other data quality metrics such as speeding, straightlining, open ends,
conflicting answers, etc.
• Don’t include QCs that have something to do with the actual survey, as respondents will try to
interpret the question in the context of the questionnaire.
• Don’t include math questions (such as, “How much is 2+2?”) – the QCs are not quizzes or math
tests. It could also be demeaning to some respondents.
• Don’t include “red herrings” such as fake brands – respondents might be confused and think of a
similar brand that does actually exist.

 

Quality Control Questions “Dos”

• Do keep QCs simple – the accurate response option should be only one (no multi-select, grid
questions, etc.), so survey respondents can move on to the actual survey questions as easily as
possible.
• Do include an indication in the question verbiage the question is for “quality control” purposes
only. Otherwise, some respondents may think there is a mistake in the questionnaire and will try
to answer “correctly” based on the survey topic, which may not be the correct answer.
• Do make sure all response options are culturally neutral and applicable for the occasion of multicountry surveys. All response options items should have clear translations in all languages and
should be native to each country.

 

Suggested data quality checks templates

For data quality control purposes, please select 'Strawberry' from the list below. [SINGLE CHOICE]

  1. Apple
  2. Banana
  3. Peach
  4. Orange
  5. Strawberry

The following question is for data quality control purposes only. Which of the following is NOT an animal?
[RANDOMIZE, SINGLE CHOICE]

  1. Cat
  2. Dog
  3. Car
  4. Monkey

Please select “very well” from the following list – this question is for data quality purposes only.
[SINGLE CHOICE]

  1. Extremely well
  2. Very well
  3. Somewhat well
  4. Not at all well

For data quality control purposes, please select the color 'green' from the list below. [SINGLE CHOICE]

  1. Blue
  2. Green
  3. Orange
  4. Red
  5. White

We very much appreciate your thoughtful responses to this survey so far. The following question has
been added for data quality control purposes only. Please select “B” below: [SINGLE CHOICE]

  1. A
  2. B
  3. C
  4. D

The following question is for data quality control purposes only. Please select the option 'disagree'.
[SINGLE CHOICE]

  1. Strongly Disagree
  2. Disagree
  3. Neither agree or disagree
  4. Agree
  5. Strongly Agree

Please select the only answer option that is a color. (This question is for data quality purposes only).
[RANDOMIZE, SINGLE CHOICE]

  1. Picture
  2. Shirt
  3. Purple
  4. Ocean
  5. River

For quality control purposes only, please select “dark chocolate” below:
[RANDOMIZE, SINGLE CHOICE]

  1. White Chocolate
  2. Milk chocolate
  3. Dark chocolate
  4. Truffles
  5. None of the above [ANCHOR]

The following question is for data quality control purposes only. Please select the item that is not an ice
cream flavor below: [RANDOMIZE, SINGLE CHOICE]

  1. Strawberry
  2. Cookie Dough
  3. Mint Chocolate Chips
  4. Basketball
  5. French Vanilla

We very much appreciate your thoughtful responses to this survey so far. The following question has
been added for data quality control purposes only. Please select 'soccer' from the list of sports below.
[SINGLE CHOICE] only.

  1. Basketball
  2. Football
  3. Hockey
  4. Soccer
  5. Tennis

Please select “neutral” from the following list – this question is for data quality purposes only.
[SINGLE CHOICE]

  1. Very unimportant
  2. Unimportant
  3. Neutral
  4. Important
  5. Very important

The following question is for data quality control purposes only. Which of the following items is not a
sport? [RANDOMIZE, SINGLE CHOICE]

  1. Tennis
  2. Peanut Butter
  3. Soccer
  4. Basketball

Please select the last response option below – this question is for data quality purposes only:
[SINGLE CHOICE]

  1. Strongly Disagree
  2. Disagree
  3. Neither agree or disagree
  4. Agree
  5. Strongly Agree

We very much appreciate your thoughtful responses throughout the survey. The following question has
been inserted as a quick and simple quality check before moving forward. Of the five response options
below, please select "Very Likely" at the bottom to continue: [SINGLE CHOICE]

  1. Very unlikely
  2. Unlikely
  3. Neither Likely nor Unlikely
  4. Likely
  5. Very likely