How to Pilot an AI Interview Before Launch
A pilot shows how the moderator follows your guide before real participants see it. Use it to test probe notes, edge cases, timing and the items you show.
Overview
A guide that reads well can still produce strange follow-ups. It can run too long or show a prototype at the wrong time. A pilot finds these problems while they cost nothing to fix.
In User Evaluation, click "Run a pilot first" to take the interview yourself. Nothing in a pilot is counted, paid or shown in the study report. When you click "End pilot", Eva reviews the guide and lists suggested fixes. You can apply a fix, run the pilot again or edit the guide.
For a text survey, "Send to 5 people first" sends it to five real people before the rest. Unlike a pilot that you take yourself, these five answers count toward the target and the report.
Rules
- 01
Take the pilot twice, as two different participants.
A talkative participant tests the stop rules. A participant who gives short answers tests the follow-ups. A guide must work for both.
- 02
Give a thin answer on purpose.
This is the only way to see if a probe note fires. Check that the follow-up asks for the detail you named in the note.
Do notPilot answer: a full, perfect story for every question.
DoPilot answer: "It was fine, I guess." Then check if the moderator asks for the moment you named.
- 03
Test skip, stop and off-topic paths.
Say "skip" once. Ask the moderator an unrelated question once. Say that you want to stop. Check that each response is polite and correct.
- 04
Time the pilot against the length you set.
Your own answers are usually shorter than a real participant's answers, because you know the questions. If the pilot is close to the limit, the real sessions will run over.
- 05
Check that items you show appear at the right question.
An image or prototype that appears too early can bias the earlier answers. In a video study, the pilot also asks for your camera and a window. You can check what gets recorded.
- 06
Read the suggested fixes, then decide.
An AI review can improve wording. It does not know your research goal as well as you do. Apply a fix only when it keeps the question on that goal.
Do notApply every suggested fix without reading it.
DoApply the fix that removes a leading word. Reject the fix that merges two questions you need to compare.
- 07
Read the transcript, not only your memory of the session.
In the session you focus on answering. The transcript shows the exact words of each follow-up and where the moderator went off the guide.
- 08
Run a small real wave before the full launch.
You cannot be a real participant in your own study. A few real sessions show problems with wording, screening and length that a pilot by the team cannot show.
Checklist before launch
- You took the pilot as a talkative participant and as a brief participant.
- Each probe note fired at least once on a thin answer.
- Skip, stop and an off-topic question each got a correct response.
- The pilot ended well inside the length you set.
- Images and prototypes appeared at the right question.
- You read every suggested fix and applied only the ones that fit the goal.
- You read the full transcript of the pilot.
- You plan a small real wave before the full launch.
Questions
A pilot is a test session before the real study. It checks that the questions, the follow-ups and the timing work.
In User Evaluation, a pilot you take yourself is not counted, paid or shown in the study report. The first five answers of a text survey sent with "Send to 5 people first" do count.
Run pilots until a full session goes without a problem in the follow-ups, the timing or the items you show. Then run a small real wave.
Ask real people your question
The AI moderator interviews screened people by voice or text. Each claim in the study report links to its clip. The first 3 interviews are free.