How should you choose when to shuffle and when to use a specific order? 🔗
If your answer options cannot be ordered into a logical or
intuitive order and as long as the list of answer options is not
so long that participants would struggle to process it, we
recommend using the
keyword to ensure that the order of presentation does not unduly
affect the responses that people are more likely to choose on
average. (Since people read from the top, they may stop at the
first answer they think is acceptable, even if there is a better
answer they haven't seen yet. This "satisficing" tendency may
bias people towards answers at the top of the list.) On the
other hand, if your list of answer options is particularly long
(e.g., a long list of locations, diseases, or drugs), then it
may be advisable to place the options in alphabetical order
(despite the risk of satisficing), so that participants can more
quickly check for the answer(s) that most closely reflects their
answer(s) to the question.
If answer options follow a natural progression or ranking (e.g., from one end of a scale to another), we recommend keeping the ordering of answer options in alignment with people's natural assumptions and intuitions. For questions involving quantities, we recommend placing higher numbers toward the top of the scale, and lower numbers toward the bottom. For questions including answers that range on a scale from "good" or "agree" to "bad" or "disagree," we recommend placing the "good" or "agree" options at the top of the scale.
Good answer option shuffling examples 🔗
Here's a good example of shuffling a question's answer options:
*question: Have you ever had any of the following experiences without them being due to the use of psychedelics or other drugs? *tip: Please tick all that apply, and leave all unchecked if you have had NONE of these experiences. *type: checkbox *shuffle *save: psychosisWithoutDrugs Losing touch with reality so you see, hear or believe things that aren't physically real A persistent combination of several of the following factors: difficulty thinking, feeling the world is unreal, seeing unreal things, fear of external organisations, and unconventional beliefs A manic episode A depressive episode followed by a somewhat manic episode Less intense rapid emotional cycling between negative and positive emotions A traumatic childbirth experience A spiritually powerful experience while giving birth to a child
In the above example, since there is no natural ordering to the
question categories, it is advisable to use the
keyword so that, on average, satisficing tendencies do not
affect the same question across the whole sample (since the
presentation order is randomized across people).
Here are some good examples of ordered quantity and frequency-related questions:
*question: How often do you visit the dentist? Less than once per year About once per year About once every six months More than once every six months *question: How many times have you gone camping in your life? Never Once or twice Three to five times More than five times
In the above questions, there is a natural presentation order, and this will help people to quickly locate the answer that most closely approximates the answer they are thinking about after reading the question.
Bad answer option shuffling examples 🔗
Here's a bad example of an inappropriately non-shuffled question:
*question: Have you ever had any of the following experiences without them being due to the use of psychedelics or other drugs? *type: checkbox A traumatic experience A period of rapid emotional cycling between negative and positive emotions A depressive episode followed by a hypomanic episode A depressive episode without a hypomanic episode A manic episode
The above categories should be shuffled to reduce the effects of satisficing systematically affecting the study results.
Here's a bad example of a frequency question that doesn't employ natural ordering:
*question: How often do you visit the dentist? *shuffle Less than once per year About once per year About once every six months More than once every six months
In the above example, categories that naturally progress from one end of a scale to another (that would be easier to read and sort between if they were left in order) have been unnecessarily shuffled.
Next: How to use and create coding variables for continuum scales