Use it for
- Notes or recordings from usability sessions, with the task each participant tried and whether they completed it.
- A list of issues, each written as one observed problem with the screen or step where it happened.
- The number of participants who tried each task.
Steps
- Write each issue as an observation
Describe what the participant did and what happened, not your guess at the cause. Write "3 of 6 tapped Save twice", not "the Save button is unclear". One issue per row.
- Merge duplicates
Merge rows that describe the same problem at the same step. Keep the participant IDs of each row. The number of participants per issue is the input for the frequency modifier.
- Rate the impact on the task
Give each issue a base level from 0 to 4. Rate the worst outcome that you observed, not the worst outcome you can imagine. Use the include and exclude rules of each level.
- Apply the modifiers
Raise the level by one for each modifier that applies, up to level 4. Do not lower a level with a modifier. Record which modifiers you applied, so readers can see why an issue moved.
- Calibrate with a second rater
A second person rates the same issues alone. Discuss each issue where the levels differ by more than one. Change the rule wording if the same disagreement happens twice.
- Sort and assign
Sort issues by final level, then by frequency. Give each issue at level 3 or 4 an owner and a fix date. Put level 1 and 2 issues in the normal backlog.
Codebook
Copy the codebook as a table. It pastes into a spreadsheet with one row for each code.
Level 0: Not a usability issue
An observation or comment that did not affect task success, time or errors.
- Include
- A participant states a preference or a design opinion, but their behavior shows no delay, error or confusion.
- Exclude
- The participant hesitated, asked for help or took a wrong step. Use Level 1 or higher for that.
- Example
- “I would make this blue, but it is fine. [participant completes the task without a pause]”
Level 1: Cosmetic
A small problem that the participant noticed but that did not slow them down.
- Include
- The participant comments on a typo, a misaligned item or odd wording, and the task time stays normal.
- Exclude
- The participant paused, reread or took an extra step because of it. Use Level 2 for that.
- Example
- “Oh, it says Recieve here. Anyway. [continues at once]”
Level 2: Minor
A problem that caused a delay or an extra step, but the participant recovered without help.
- Include
- The participant hesitated, tried a wrong path once or reread a label, then completed the task alone.
- Exclude
- The participant needed many attempts, needed a hint or made an error with a consequence. Use Level 3 for that.
- Example
- “Where is the... no, not this menu. Ah, here it is, under Settings.”
Level 3: Major
A problem that made the task hard. The participant needed many attempts or a hint, or had to repair an error.
- Include
- The participant completed the task, but only with a hint, after several wrong paths or after they fixed an error.
- Exclude
- The participant did not complete the task, or completed it wrong without noticing. Use Level 4 for that.
- Example
- “I added it twice, so now I have to find how to remove one. Can you tell me where?”
Level 4: Critical
A problem that stopped task completion, or made the participant finish with a wrong result that they did not notice.
- Include
- The participant gave up, asked to stop, or believed they had finished when the result was wrong.
- Exclude
- The participant completed the task correctly, even after a long struggle. Use Level 3 for that.
- Example
- “Done, I paid the invoice. [the payment was saved as a draft and never sent]”
Modifier: Frequent
The issue affected many of the participants who tried the task.
- Include
- Half or more of the participants who tried the task had the issue. Add one level.
- Exclude
- The issue affected only one or two participants in a large group. Do not add a level, but report the count.
- Example
- “[4 of 6 participants scroll past the Continue button and look for it at the top]”
Modifier: Costly consequence
In real use, the issue can cause loss of money, data, privacy or safety, even if the test caught it.
- Include
- The error sends money, deletes data, shares private information or cannot be undone. Add one level.
- Exclude
- The error only costs time and the participant can undo it. Do not add a level.
- Example
- “Wait, did that just send the report to the client? I only wanted a preview.”
Modifier: Repeats after learning
The issue happens again after the participant has seen it once, so practice will not remove it.
- Include
- The participant has the same problem on a later task or a second try. Add one level.
- Exclude
- The participant learned the fix and did not have the problem again. Do not add a level.
- Example
- “Again I pressed the wrong one. They look exactly the same.”
Report outline
- Tasks tested, number of participants and completion result for each task.
- Critical and major issues: the observation, the final level, the modifiers applied and a clip or quote.
- Minor and cosmetic issues in a short table.
- What worked well, so the team does not change it by accident.
- Recommended fixes with an owner, sorted by level.
- Method notes: who rated the issues and where the raters disagreed.
Mistakes to avoid
- Do not rate the issue by how much the participant complained. Rate what happened to the task. Quiet participants hide critical issues.
- Do not rate an issue higher because it is easy to fix. Severity and effort are two different columns.
- Do not use a modifier to lower a level. A rare critical issue is still critical.
- Do not mix opinions with issues. Level 0 comments go in a separate list, or they hide the real problems.
Guides that use it