The first 3 interviews are free · Start free or see the prices

Developer Experience Interview: Questions About Your API or SDK

Use this guide with developers who integrated your API or SDK in the last 3 months. It follows the integration from evaluation to production, with a focus on the first call, authentication, docs and errors.

Length
35 min
Questions
16
People
8 to 12
Sections
6
Run this study

What this guide finds out

  • The time and steps from sign-up to the first successful call, and what blocked it.
  • Where authentication, keys and permissions caused confusion or rework.
  • Which docs, samples and references developers used, and where they went outside the docs.
  • Which errors were hard to understand, and how developers debugged them.

When to use it

  • Many developers create keys but few make calls in production.
  • Your support queue has many integration questions that the docs should answer.
  • You plan a new SDK version, new auth method or a docs redesign.

Who to talk to

Developers who integrated your API or SDK in the last 3 months, with a mix of languages and company sizes. Talk to 8 to 12 people. Each interview takes about 35 minutes.

Screener for this guideSoftware Developer Screener: Questions to Recruit EngineersUse this screener to find people who write code as part of their job. A short technical check and a fake framework catch people who copy a developer profile.

The discussion guide

Each question has a probe note. The note tells the moderator, a person or an AI, what to do with the answer.

01

Warm-up

3 min

Learn the stack and the project of the developer.

  1. Tell me about the project where you used our API. What were you building?

    ProbeAsk for the language, framework and where the code runs. Record it as context.

  2. Was this your choice, or did someone else choose the API for you?

    ProbeIf someone else chose it, ask what the developer would have chosen instead.

02

Evaluation

5 min

Learn what developers checked before they wrote code.

  1. Before you wrote any code, what did you look at to decide if the API would work for you?

    ProbeAsk which pages they read: pricing, limits, reference, samples or status. Ask what answer they looked for.

  2. Which other APIs or libraries did you compare, if any?

    ProbeAsk what made the others fail. Do not defend our API.

03

Authentication and the first call

9 min

Rebuild the path to the first successful call.

  1. Take me through the steps from sign-up to your first successful call.

    ProbeAsk how long it took, in minutes or hours. Ask where they started: a quickstart, the reference, a sample or an AI assistant.

  2. How did you get and store your key or token?

    ProbeAsk where the key lives now, for example an environment variable or a secret manager. Ask if they were sure about the right scope or permissions.

  3. What was the first error you hit, and how did you solve it?

    ProbeAsk for the status code or the message if they remember it. Ask how long the error blocked them.

  4. Was there a moment when you thought about giving up?

    ProbeAsk what kept them going. If they say no, do not push.

04

Docs and samples

7 min

Learn which docs developers use, and what they look for outside the docs.

  1. Which parts of the docs did you go back to most often?

    ProbeAsk what they looked for each time. Ask if the page had the answer.

  2. When the docs did not answer your question, where did you go?

    ProbeAsk if they used search, forums, source code, support or an AI assistant. Ask what answer they found there.

  3. Did you copy a code sample? What did you change before it worked?

    ProbeAsk which language the sample was in and if their language had one. Record each change they had to make.

05

Errors and production

8 min

Learn how developers debug errors and what worries them in production.

  1. Tell me about the hardest error you had to debug with our API.

    ProbeAsk what the error message said and what the real cause was. Ask what message would have saved them time.

  2. How do you handle rate limits, retries and timeouts in your code?

    ProbeAsk how they learned the limits. Ask if they found them in the docs or in production.

  3. What worries you about our API in production?

    ProbeAsk about versioning, outages and breaking changes. Ask how they would want to hear about a change.

06

Close

3 min

Get one priority change.

  1. If you could change one thing about the developer experience, what would it be?

    ProbeAsk them to name one change. Ask which step it would remove or shorten.

  2. Is there anything else that a new developer should know before they start?

    ProbeRecord the advice word for word. Thank them and end the interview.

Example finding

ExampleWrong key type

Several developers used the publishable key on the server. The 401 error did not say which key type was needed.

“It just said unauthorized. I regenerated the key twice before I saw there are two kinds. The error could have told me that.”
In a User Evaluation study report, each claim links to the clip where a participant says it.

Mistakes to avoid

  • Do not ask developers to rate the docs on a scale. Ask what they looked for and if they found it.
  • Do not let the product team answer technical questions during the session. Record the question as a finding.
  • Do not ask developers to share keys, tokens or private code. If they share a screen, ask them to hide secrets first.
  • Do not interview only developers who reached production. Developers who stopped show where the path breaks.

Analyze the answers

Analysis templateUsability Issue Severity Rating Scale: Levels and RulesUse this scale to rate each issue from a usability test. Rate what the issue did to the task, then apply the modifiers. Two people who use these rules should give the same rating.

Questions

Time to first call is the time from sign-up to the first successful API request. It is a common measure of developer experience. Interviews show what makes it long.

It helps when you ask them to repeat the first call or show an error. Ask them to close files with keys before they share. A voice-only interview works for a story about a past integration.

Use your list of accounts that created a key in the last 3 months. Include accounts that never made a call, and accounts that reached production.

Measure time to first call, the share of keys that make a call, and the most common error codes. The interviews explain the numbers.

Let the AI moderator ask these questions

The AI moderator interviews screened people by voice or text. Each claim in the study report links to its clip. The first 3 interviews are free.

Run this study