How to Check Research Findings That an AI Wrote
An AI can write a study report in minutes. You are still responsible for each claim in it. These rules show how to check claims against the clips before you share them.
Overview
AI-written findings fail in a few typical ways. A claim says more than its clip. A theme rests on one talkative participant. A quote came from a leading follow-up. The minority view disappears. Each of these is easy to find if you check in a fixed order.
In a User Evaluation study report, every claim cites at least one clip. A claim cannot be saved without one. Each theme shows a confidence level of High, Medium or Low, with the number of cited items. Thin evidence is labelled Low instead of being dropped.
This does not make the report correct by itself. A citation shows where to look. It does not show that the clip supports the claim. That check is your job.
Rules
- 01
Play the clip for every claim you will repeat.
A claim can be close to the clip and still be wrong. Check that the clip says what the claim says, with the same strength.
Do notClaim: "Participants found the setup too hard." Clip: "Setup took a while, but it was fine."
DoClaim: "Several participants said setup took longer than they expected." Clip: "Setup took a while, but it was fine."
- 02
Count participants, not clips.
One participant can supply several clips to the same theme. Check how many different people the clips come from before you accept words such as "most" or "many".
- 03
Read the moderator's question before each key quote.
A strong quote can be the answer to a leading follow-up. If the moderator suggested the words, the quote is weak evidence.
Do notModerator: "So the price felt unfair?" Participant: "Yes, unfair."
DoModerator: "How did you feel about the price?" Participant: "Honestly, it felt unfair for what we used."
- 04
Ask for the disagreement.
Summaries favor the majority view. Ask a question such as "Who disagreed with the majority, and why?". In User Evaluation, the Ask the evidence card answers only from the clips of the study.
- 05
Separate what people did from what they said they would do.
"Participants will pay for X" is a prediction. "Three participants paid for a similar tool last year" is behavior. Rewrite claims that mix the two.
- 06
Treat Low confidence themes as leads, not findings.
A Low theme may be true, but the evidence is thin. Put it under a heading such as "To check next" instead of in the main findings.
- 07
Read some full transcripts, not only the clips.
Clips are a selection. In User Evaluation, if a study has no clips, Eva first cuts them from the longest answers. Short answers can then be under-represented. Also read the Other moments section for clips that fit no theme.
- 08
Check every count the AI gives.
A count such as "five participants mentioned price" can include passing mentions. Open the clips and check that each one is about the same thing.
- 09
Remove bad sessions before you regenerate.
A fake or careless session can still feed a theme. In User Evaluation, a session flagged for replacement is not used the next time you generate or regenerate the report.
Checklist before launch
- You played the clip for every claim in the summary.
- Each "most" or "many" is backed by a count of different participants.
- You read the question before each key quote.
- You asked who disagreed, and the report says so.
- Claims about behavior and claims about intent are separate.
- Low confidence themes are marked as leads.
- You read at least a few full transcripts and the Other moments section.
- Flagged sessions are out, and the report was regenerated.
Questions
Trust it as a first draft. Check each claim you will share against its clips before you present it.
A claim that is stronger or wider than its evidence, for example "users" when only two participants said it.
It depends on the number of claims. Plan time to play the clip behind each claim you will share, not only to read the report.
Ask real people your question
The AI moderator interviews screened people by voice or text. Each claim in the study report links to its clip. The first 3 interviews are free.