Use it for
- An export of survey responses with one row per respondent and one column per open question.
- The exact wording of each open question and the question before it, because earlier questions shape the answers.
- Closed answers for each respondent, such as plan or segment, so you can compare groups.
Steps
- Clean the answers
Remove blank answers, keyboard mashing and exact copies from the same person. Flag answers that repeat the question or that look pasted from a chatbot. Keep short real answers like "nothing".
- Build the codes from a sample
Take a random sample of 100 answers. Write a code for each distinct idea. Merge codes that overlap until each code has a clear include and exclude rule.
- Test agreement with a second coder
Two people code a new sample of 50 answers alone. Compare each answer. Where the codes differ, fix the rule that caused it. Repeat until the coders agree on most answers.
- Code all answers
Code the full data set. Allow up to two codes per answer, because survey answers are short. Mark the code that comes first in the answer as primary.
- Count and compare groups
Count the share of respondents per code, not the share of codes. Then compare the shares between groups, such as new and returning customers.
- Pick quotes last
Choose 2 or 3 quotes per code after you count. Pick quotes that are typical, not the most extreme. Note how many answers each quote stands for.
Codebook
Copy the codebook as a table. It pastes into a spreadsheet with one row for each code.
Cost not clear
The respondent hesitated because they did not know what they would pay.
- Include
- The answer mentions unclear pricing, fear of hidden fees, or not knowing what happens after the trial.
- Exclude
- The respondent knew the price and thought it was too high. Use Price too high for that.
- Example
- “Wasn't sure if I would be charged automatically after 14 days”
Price too high
The respondent knew the price and judged it high for them.
- Include
- The answer says the product is expensive, costs more than expected or is more than they want to pay.
- Exclude
- The respondent did not know the price. Use Cost not clear for that.
- Example
- “Pretty steep for what is basically a to-do list”
Unclear what it does
The respondent did not understand what the product does or how it works.
- Include
- The answer says the page or the product was confusing, or they could not see how it works.
- Exclude
- The respondent understood the product but doubted it fits their case. Use Fit to my case for that.
- Example
- “Took me a while to understand what it actually does”
Fit to my case
The respondent doubted that the product would work for their specific situation.
- Include
- The answer mentions their industry, team size, country, device or a special need that the product might not support.
- Exclude
- The respondent did not understand the product at all. Use Unclear what it does for that.
- Example
- “Not sure it works for a team of 2, looks built for big companies”
Trust in the company
The respondent doubted the company was real, stable or honest.
- Include
- The answer mentions few reviews, an unknown brand, a scam worry or fear that the company will close.
- Exclude
- The worry is about what happens to their data. Use Data privacy for that.
- Example
- “Never heard of you, had to check reviews first”
Data privacy
The respondent worried about how the company would use or protect their data.
- Include
- The answer mentions personal data, account access, permissions, data sale or security.
- Exclude
- The worry is about the company as a whole and not about data. Use Trust in the company for that.
- Example
- “Didn't love giving access to my whole Google account”
Setup effort
The respondent worried about the time or work to start.
- Include
- The answer mentions the length of the form, data entry, moving from another tool or learning time.
- Exclude
- The answer is about a long contract or a commitment. Use Commitment for that.
- Example
- “The thought of moving all my recipes over was a lot”
Commitment
The respondent hesitated because of a contract, a payment detail or difficulty to cancel.
- Include
- The answer mentions a card required for the trial, an annual plan, auto-renewal or how to cancel.
- Exclude
- The answer is about not knowing the price. Use Cost not clear for that.
- Example
- “Why do you need my card for a free trial?”
Comparing options
The respondent hesitated because they were still looking at other products.
- Include
- The answer names a competitor, or says they wanted to compare or try other options first.
- Exclude
- The answer names no alternative and only says the price is high. Use Price too high for that.
- Example
- “Was trying 2 other apps at the same time”
Nothing
The respondent says nothing almost stopped them.
- Include
- The answer is "nothing", "no", "all good" or similar.
- Exclude
- The answer starts with "nothing" but then names a concern. Code the concern.
- Example
- “Nothing, it was easy”
Report outline
- The question wording, the number of responses and the number removed in cleaning.
- The share of respondents per code, with the primary code shown apart.
- Differences between groups, such as segment, plan or traffic source.
- The top 3 codes, each with typical quotes and what the respondents mean.
- Actions for each top code and the next study to learn more about it.
- The codebook and the agreement result, in an appendix.
Mistakes to avoid
- Do not build the codebook from the first 100 answers in the file. Early respondents differ from late ones. Take a random sample.
- Do not count "Nothing" answers out of the total. They show how many respondents had no concern, which is part of the result.
- Do not use the same codebook for a different question. Codes depend on the question wording. Build new codes from a new sample.
- Do not trust long, well-written answers from a fast survey without a check. Some are pasted from a chatbot. Flag and compare them.
Guides that use it