GuidedTrack How to check for participant comprehension and attention (1) GuidedTrack Docs

How to check for participant comprehension and attention🔗

To improve the quality of the information you derive from your survey or experiment, it is important to check that participants are understanding the survey/experiment in the way that you intended them to understand it. Here are some tips for your comprehension check questions:

  • If you use a multiple-choice answer format:
    • Make sure the incorrect options sound reasonable — someone should only be able to distinguish between the answers if they have correctly understood your survey/experiment up to that point, not just because they know how to spot "reasonable-sounding" answers
    • Make sure the correct answer doesn't stand out visually (e.g., it could stand out if it is longer/shorter than the other options, if it has a full stop when the other options don't have a full stop, and so on)
    • Don't put the correct answer as the first option or the middle option — the best option might be to *shuffle the answer options (so that each participant has a randomized answer option order)
  • If you ask participants to fill in a free text response to a question:
    • Ensure that there is only one way to answer the question
    • Use forgiving (non-case-sensitive) text comparisons

Good examples of comprehension checks🔗

Here are some good examples of how to check for participant comprehension without making the answer too obvious. In this first example, notice how we keep answer options structurally/syntactically similar:

>> comprehendedEnrollment = 0

*question: Which of the following options describes the study that we are offering you enrollment in?
	*shuffle
	This study will take you 2 hours in total and will last 2 weeks. You will need to fill out one survey a week.
	This study will take you 3 hours in total and will last 4 weeks. You will need to fill out two surveys a week.
	This study will take you 6 hours in total and will last 6 weeks. You will need to fill out two surveys a week.
	This study will take you 4 hours in total and will last 7 weeks. You will need to fill out one survey a week.
	This study will take you 1 hour in total and will last 8 weeks. You will need to fill out one survey a week.
		>> comprehendedEnrollment = 1

*if: comprehendedEnrollment = 0
	Sorry, that's not right. Please read the study description again.
	*goto: studyRules

In the above example, participants would have to have read the description of what study participation entails in order to know the correct answer. All options sound reasonable, are of similar length and structure, and have the same punctuation; so the correct answer does not stand out visually.

Here's a good example of how to make all or most of the answer options sound reasonable. In this example, suppose that you are administering a survey that seeks to replicate the findings by Desvousges et al. (1992) by asking people how much they would be willing to pay the company that supplies their gas if it meant that the oil company could then cover their oil pits with nets to prevent birds from landing and drowning in them. If you gave participants a scenario outlining the way in which the company would cover pits with nets and how that was expected to stop birds from drowning, you would need to then check the degree to which participants have understood these key points before you can interpret their answers to the rest of the study questions. You might do this as follows:

>> comprehensionCheck = 0
>> numberIncorrectResponses = 0

*question: In the scenario you just read, if you chose to pay the oil company more, what effect would this have?
	*shuffle
	*confirm
	It would allow the oil company to use my money to cover more pits with nets, which will save more birds from death.
		>> comprehensionCheck = 1
		>> interpretedAsIndividuallyImpactful = 1
	It wouldn't have any effect.
		>> numberIncorrectResponses = numberIncorrectResponses + 1
	The company would have more money, but that doesn't necessarily mean that they will cover more pits just because I paid them more.
		>> comprehensionCheck = 1
		>> interpretedAsIndividuallyImpactful = 0
	Please take me back to the explanation
		*goto: explanation part 2
	It wouldn't change the number of birds who survive, because covering pits with nets doesn't actually work.
		>> numberIncorrectResponses = numberIncorrectResponses + 1
	It would allow the oil company to invest in developing alternative, more sustainable energy sources.
		>> numberIncorrectResponses = numberIncorrectResponses + 1
	It would allow the oil company to invest in campaigns to stop people trying to shoot water birds.

*if: comprehensionCheck = 0
	Sorry, that's not right. Please read the scenario again.
	*goto: scenarioExplainer

In the above example, there are two correct answers (so if the participant chose either of those, their comprehensionCheck would be set to 1 and they would be allowed to proceed). This is because there are two valid ways to interpret the scenario in this example. Also note that the incorrect options nevertheless still sound reasonable (e.g., "It would allow the oil company to invest in developing alternative, more sustainable energy sources" sounds like something a company might do as part of corporate social responsibility campaigning, for instance).

In this example, the number of times that the participants answer incorrectly are recorded, so that participants can be excluded from analyses if they meet some predefined criterion indicating poor comprehension (e.g., two incorrect answers in a row).

Bad example of comprehension checks🔗

Now, here's a bad example of a "comprehension check" that doesn't actually require comprehension because the answer is obvious:

*question: What will participation in this experiment involve?
	Nothing
	Answering five 10-minute surveys, one week apart.
	Lots of driving

In the above example, the correct answer stands out because (1) it is arguably the only one that sounds like a reasonable answer, (2) it has a full stop, while the other options don't have one, and (3) it is longer in length than the others. If someone answered this question correctly, it would be impossible to know whether they've actually read the experiment description or whether they were able to infer which answer was correct based on other clues. Therefore, this kind of question would not be appropriate as a comprehension check, though it might be suitable as a simple attention check (see below).

Examples of attention checks🔗

In addition to the comprehension checks explained above, it is recommended that you always include at least two attention-checking questions in a given survey or experiment task and that you warn people at the start of the experiment that there will be attention checks. Attention checks should be able to be answered if and only if someone is human and is paying attention. Here are some examples of attention check questions you could use:

Next: