There are now two very different ways to watch someone use your website. You can recruit real people and watch them try. Or you can send AI agents, software that drives a real browser, reads the page and acts on it, and watch them try instead.
It is tempting to treat these as competitors, and to ask which one is better. That is the wrong question. They answer different questions, fail in different ways, and cost very different amounts. Used together, they make each other more useful.
This guide is about choosing between them for a given job. It is written from our experience building both into User Evaluation: Swarm for AI agents, and AI-moderated interviews with real participants.
What each one actually is
An AI agent test sends software through your site with a goal ("sign up and share a first report") and a persona ("a busy founder on a laptop"). The agent clicks, types, reads and waits, and writes down where it got stuck. A good one also sees the screen and the errors behind it: console errors, failed requests, slow responses.
A real-user test puts a person in front of your product, either moderated live or recorded while they think aloud, and asks them to do the same kind of task. You get their behaviour and their reasons, in their own words.
The difference that matters most is simple. An agent can tell you that something is broken or confusing. Only a person can tell you whether it matters to them.
Where AI agents are the better tool
Reach for agents when the question is about the product's behaviour, not about anyone's feelings:
- "Does this journey work end to end?" Signup, onboarding, checkout up to payment, sharing, exporting. Agents try every step and report the one that fails.
- "What broke?" Agents that read the console and network catch the TypeError behind a dead button and the 500 behind a silent dialog. See Swarm agents see the page.
- "Did we break it again?" A run every day or every week on the same journey shows when a step that worked yesterday stops working today.
- "Is it broken for someone who reads everything, or for someone who skims?" Personas change how an agent behaves, so five agents with different personas find different problems.
- "Before we spend on recruiting, is the flow even usable?" A quick agent run clears out the obvious faults so real participants spend their time on the questions only they can answer.
Agents are also fast and cheap enough to use often. On User Evaluation, each plan includes Swarm runs every month (1 on Free, 5 on Pro and 25 on Team), and an extra run is $2, however many agents it has.
Where only real people will do
Reach for people when the question involves value, trust, emotion or a decision:
- "Would you pay for this?" An agent has never paid for anything and never will. It can't tell you whether a price feels fair.
- "Why did you leave?" Churn is a human decision with human reasons. Agents don't churn.
- "What did you expect to happen?" Expectations come from people's past experience with other products, not from reading your page.
- "Do you trust this?" Trust in a brand, a payment form or a privacy promise is a feeling only a person has.
- "Which of these problems matters most?" Agents can list friction. Only people can rank it by how much it hurts.
This matches the wider evidence on AI stand-ins for customers. Our piece on synthetic users versus real participants covers it: AI-generated respondents are shallow and too agreeable, and they cannot surprise you. Browser agents are a different tool from synthetic interview respondents, since they act on your real site instead of inventing opinions, but the same limit applies to anything that sounds like an opinion.
A quick way to decide
| The question you have | Start with |
|---|---|
| Does the flow work end to end? | AI agents |
| Which step throws an error, or is slow? | AI agents |
| Did this week's release break signup? | AI agents, on a schedule |
| Is the layout broken on a laptop or a phone? | AI agents, then people to confirm |
| Is this step confusing? | AI agents to find it, people to confirm it |
| Would people pay, trust or recommend this? | Real people |
| Why do customers cancel? | Real people |
| What should we build next? | Real people |
The combination that works
The strongest pattern is not agents or people. It is agents first, people second.
- Agents find the candidates. A run sends several personas through a journey and comes back with a triaged list of findings, each with the screenshots and steps behind it.
- You fix the obvious. A server error from your own API doesn't need a study to prove it.
- People judge the rest. The findings that depend on human judgement go to a small, fast study with real participants.
In Swarm, step three is a single button. Confirm with 5 people drafts one short AI-moderated interview that asks 5 everyday people about every open finding, and opens it for you to check before anything is charged. We wrote about it in Confirm with 5 people.
Mistakes to avoid
Treating agent findings as facts. They are good leads. On User Evaluation they stay labelled as agent evidence, at medium confidence, until people confirm them. Keep that discipline even when you use other tools.
Asking agents for opinions. "Did you like the onboarding?" is not a question for software. "Where did you get stuck?" is.
Skipping agents because you have a panel. Recruiting people to discover that a button throws an error is an expensive way to find a bug. Let agents clear the path first.
Using people only once. Five people once a quarter will not keep up with a team that ships every week. Small, frequent checks on the findings that matter keep research in step with the product. Our guide to continuous discovery with AI covers the habit.
The short version
Use AI agents to find out whether your product works and where it breaks, often and cheaply. Use real people to find out whether it matters, why, and what to do about it. Let the agents point, and let the people decide.
To try both in one place, see how Swarm works and AI-moderated interviews, and the prices for each on the pricing page.
