Leading Questions: Examples and How to Avoid Them in AI Interviews
A leading question suggests the answer it wants. With an AI moderator, bias can come from questions, probe notes and the way the moderator restates an answer.
Overview
A leading question puts an answer into the mind of the participant. Most participants want to be helpful, so they agree with what the question suggests. The result is data that confirms your hypothesis and tells you nothing new.
An AI moderator asks a leading question in exactly the same way in every session. One biased question or note can affect the whole study. Bias can also come from the follow-ups, because the moderator writes them from your probe notes.
In User Evaluation, Eva drafts open, non-leading questions. You can also ask Eva to fix a question, for example "Make question 3 less leading". After a pilot, Eva lists suggested fixes to the guide.
Rules
- 01
Remove adjectives that name the answer.
Words such as "easy", "frustrating" or "helpful" tell the participant how to rate the experience before they describe it.
Do notHow easy was the new checkout?
DoTell me how the checkout went the last time you bought something.
- 02
Do not put the cause in the question.
If the question names a cause, the participant tends to confirm it. You lose the cause they would have named on their own.
Do notDid the price make you cancel?
DoWhat made you decide to cancel?
- 03
Ask about past behavior, not future intent.
People are poor predictors of what they will do. "Would you use…?" invites a polite yes. A past event shows what they did.
Do notWould you use a feature that sends you a weekly summary?
DoTell me about the last time you needed to know what your team did that week.
- 04
Keep loaded words out of the probe notes.
The moderator writes follow-ups from your notes and often copies their words. A loaded note becomes a loaded follow-up.
Do notNote: ask how frustrating the setup was.
DoNote: ask how the setup went. When they describe a feeling, use their word for it.
- 05
Tell the moderator not to restate an answer as a conclusion.
A summary such as "So the export was the main problem?" invites a yes. The participant may agree to be polite, even if the export was a small issue.
Do notSo the export was the main problem for you?
DoWhat else, if anything, made the task harder?
- 06
Do not reveal what you built or what you hope to hear in the introduction.
If participants know the sponsor's goal, they answer toward it. Tell them the topic, not the hypothesis.
Do notWe built a new AI planner and want to hear how it helps you.
DoWe want to learn how you plan your work for the week.
- 07
Give balanced options in choice questions, and change their order.
Unbalanced labels push the choice. The first concept that a person sees also colors the rest. In User Evaluation you can shuffle the order of the items you show.
Do notDo you prefer the simple plan or the complicated one?
DoHere are two plans, A and B. Which one fits how you work, and why?
- 08
Search the transcripts for leading follow-ups after the first sessions.
Your questions can be neutral and the follow-ups still lead. Search for words such as "easy", "better", "so you" and "don't you think" in the moderator turns.
Checklist before launch
- No question contains an adjective that rates the experience.
- No question names a cause, a feature or a competitor before the participant does.
- Questions ask about past events, not future intent.
- Probe notes use neutral words.
- The introduction states the topic, not the hypothesis.
- Choice questions have balanced labels and a shuffled order where possible.
- You read the moderator's follow-ups in the pilot for leading words.
Questions
A leading question suggests an answer. For example, "How much did you like the new design?" assumes that the person liked it.
Yes. The moderator writes follow-ups from your probe notes and from the answer. If a note uses loaded words, or the moderator restates an answer as a conclusion, the follow-up can lead.
No. "Did you use the export last week?" is closed but neutral. A closed question leads when it suggests which answer is correct.
Ask real people your question
The AI moderator interviews screened people by voice or text. Each claim in the study report links to its clip. The first 3 interviews are free.