The first 3 interviews are free · Start free or see the prices

Open-Ended Survey Coding: How to Code Survey Responses

Use this process to code the free-text answers in any survey. The example codebook is for the question "What almost stopped you from signing up today?". Replace the codes, keep the process.

Analyze a transcript free

Use it for

  • An export of survey responses with one row per respondent and one column per open question.
  • The exact wording of each open question and the question before it, because earlier questions shape the answers.
  • Closed answers for each respondent, such as plan or segment, so you can compare groups.

Steps

  1. Clean the answers

    Remove blank answers, keyboard mashing and exact copies from the same person. Flag answers that repeat the question or that look pasted from a chatbot. Keep short real answers like "nothing".

  2. Build the codes from a sample

    Take a random sample of 100 answers. Write a code for each distinct idea. Merge codes that overlap until each code has a clear include and exclude rule.

  3. Test agreement with a second coder

    Two people code a new sample of 50 answers alone. Compare each answer. Where the codes differ, fix the rule that caused it. Repeat until the coders agree on most answers.

  4. Code all answers

    Code the full data set. Allow up to two codes per answer, because survey answers are short. Mark the code that comes first in the answer as primary.

  5. Count and compare groups

    Count the share of respondents per code, not the share of codes. Then compare the shares between groups, such as new and returning customers.

  6. Pick quotes last

    Choose 2 or 3 quotes per code after you count. Pick quotes that are typical, not the most extreme. Note how many answers each quote stands for.

Codebook

Copy the codebook as a table. It pastes into a spreadsheet with one row for each code.

Cost not clear

The respondent hesitated because they did not know what they would pay.

Include
The answer mentions unclear pricing, fear of hidden fees, or not knowing what happens after the trial.
Exclude
The respondent knew the price and thought it was too high. Use Price too high for that.
Example
“Wasn't sure if I would be charged automatically after 14 days”

Price too high

The respondent knew the price and judged it high for them.

Include
The answer says the product is expensive, costs more than expected or is more than they want to pay.
Exclude
The respondent did not know the price. Use Cost not clear for that.
Example
“Pretty steep for what is basically a to-do list”

Unclear what it does

The respondent did not understand what the product does or how it works.

Include
The answer says the page or the product was confusing, or they could not see how it works.
Exclude
The respondent understood the product but doubted it fits their case. Use Fit to my case for that.
Example
“Took me a while to understand what it actually does”

Fit to my case

The respondent doubted that the product would work for their specific situation.

Include
The answer mentions their industry, team size, country, device or a special need that the product might not support.
Exclude
The respondent did not understand the product at all. Use Unclear what it does for that.
Example
“Not sure it works for a team of 2, looks built for big companies”

Trust in the company

The respondent doubted the company was real, stable or honest.

Include
The answer mentions few reviews, an unknown brand, a scam worry or fear that the company will close.
Exclude
The worry is about what happens to their data. Use Data privacy for that.
Example
“Never heard of you, had to check reviews first”

Data privacy

The respondent worried about how the company would use or protect their data.

Include
The answer mentions personal data, account access, permissions, data sale or security.
Exclude
The worry is about the company as a whole and not about data. Use Trust in the company for that.
Example
“Didn't love giving access to my whole Google account”

Setup effort

The respondent worried about the time or work to start.

Include
The answer mentions the length of the form, data entry, moving from another tool or learning time.
Exclude
The answer is about a long contract or a commitment. Use Commitment for that.
Example
“The thought of moving all my recipes over was a lot”

Commitment

The respondent hesitated because of a contract, a payment detail or difficulty to cancel.

Include
The answer mentions a card required for the trial, an annual plan, auto-renewal or how to cancel.
Exclude
The answer is about not knowing the price. Use Cost not clear for that.
Example
“Why do you need my card for a free trial?”

Comparing options

The respondent hesitated because they were still looking at other products.

Include
The answer names a competitor, or says they wanted to compare or try other options first.
Exclude
The answer names no alternative and only says the price is high. Use Price too high for that.
Example
“Was trying 2 other apps at the same time”

Nothing

The respondent says nothing almost stopped them.

Include
The answer is "nothing", "no", "all good" or similar.
Exclude
The answer starts with "nothing" but then names a concern. Code the concern.
Example
“Nothing, it was easy”

Report outline

  1. The question wording, the number of responses and the number removed in cleaning.
  2. The share of respondents per code, with the primary code shown apart.
  3. Differences between groups, such as segment, plan or traffic source.
  4. The top 3 codes, each with typical quotes and what the respondents mean.
  5. Actions for each top code and the next study to learn more about it.
  6. The codebook and the agreement result, in an appendix.

Mistakes to avoid

  • Do not build the codebook from the first 100 answers in the file. Early respondents differ from late ones. Take a random sample.
  • Do not count "Nothing" answers out of the total. They show how many respondents had no concern, which is part of the result.
  • Do not use the same codebook for a different question. Codes depend on the question wording. Build new codes from a new sample.
  • Do not trust long, well-written answers from a fast survey without a check. Some are pasted from a chatbot. Flag and compare them.

Guides that use it

Questions

Clean the answers and build codes from a random sample. Test the codes with a second coder. Then code all answers and count respondents per code.

Code all of them if there are fewer than 1,000. If there are more, a random sample of 400 to 500 usually shows the main codes and their shares.

Yes, with a check. Give the model your codebook with the include and exclude rules. Code 100 answers by hand and compare. Fix the rules where the two differ before you trust the counts.

Most single questions need 6 to 12 codes. If a code has less than 2 percent of answers, merge it into a related code or into Other.

Ask real people your question

The AI moderator interviews screened people by voice or text. Each claim in the study report links to its clip. The first 3 interviews are free.

Start free