The first 3 interviews are free · Start free or see the prices

Usability Issue Severity Rating Scale: Levels and Rules

Use this scale to rate each issue from a usability test. Rate what the issue did to the task, then apply the modifiers. Two people who use these rules should give the same rating.

Analyze a transcript free

Use it for

  • Notes or recordings from usability sessions, with the task each participant tried and whether they completed it.
  • A list of issues, each written as one observed problem with the screen or step where it happened.
  • The number of participants who tried each task.

Steps

  1. Write each issue as an observation

    Describe what the participant did and what happened, not your guess at the cause. Write "3 of 6 tapped Save twice", not "the Save button is unclear". One issue per row.

  2. Merge duplicates

    Merge rows that describe the same problem at the same step. Keep the participant IDs of each row. The number of participants per issue is the input for the frequency modifier.

  3. Rate the impact on the task

    Give each issue a base level from 0 to 4. Rate the worst outcome that you observed, not the worst outcome you can imagine. Use the include and exclude rules of each level.

  4. Apply the modifiers

    Raise the level by one for each modifier that applies, up to level 4. Do not lower a level with a modifier. Record which modifiers you applied, so readers can see why an issue moved.

  5. Calibrate with a second rater

    A second person rates the same issues alone. Discuss each issue where the levels differ by more than one. Change the rule wording if the same disagreement happens twice.

  6. Sort and assign

    Sort issues by final level, then by frequency. Give each issue at level 3 or 4 an owner and a fix date. Put level 1 and 2 issues in the normal backlog.

Codebook

Copy the codebook as a table. It pastes into a spreadsheet with one row for each code.

Level 0: Not a usability issue

An observation or comment that did not affect task success, time or errors.

Include
A participant states a preference or a design opinion, but their behavior shows no delay, error or confusion.
Exclude
The participant hesitated, asked for help or took a wrong step. Use Level 1 or higher for that.
Example
“I would make this blue, but it is fine. [participant completes the task without a pause]”

Level 1: Cosmetic

A small problem that the participant noticed but that did not slow them down.

Include
The participant comments on a typo, a misaligned item or odd wording, and the task time stays normal.
Exclude
The participant paused, reread or took an extra step because of it. Use Level 2 for that.
Example
“Oh, it says Recieve here. Anyway. [continues at once]”

Level 2: Minor

A problem that caused a delay or an extra step, but the participant recovered without help.

Include
The participant hesitated, tried a wrong path once or reread a label, then completed the task alone.
Exclude
The participant needed many attempts, needed a hint or made an error with a consequence. Use Level 3 for that.
Example
“Where is the... no, not this menu. Ah, here it is, under Settings.”

Level 3: Major

A problem that made the task hard. The participant needed many attempts or a hint, or had to repair an error.

Include
The participant completed the task, but only with a hint, after several wrong paths or after they fixed an error.
Exclude
The participant did not complete the task, or completed it wrong without noticing. Use Level 4 for that.
Example
“I added it twice, so now I have to find how to remove one. Can you tell me where?”

Level 4: Critical

A problem that stopped task completion, or made the participant finish with a wrong result that they did not notice.

Include
The participant gave up, asked to stop, or believed they had finished when the result was wrong.
Exclude
The participant completed the task correctly, even after a long struggle. Use Level 3 for that.
Example
“Done, I paid the invoice. [the payment was saved as a draft and never sent]”

Modifier: Frequent

The issue affected many of the participants who tried the task.

Include
Half or more of the participants who tried the task had the issue. Add one level.
Exclude
The issue affected only one or two participants in a large group. Do not add a level, but report the count.
Example
“[4 of 6 participants scroll past the Continue button and look for it at the top]”

Modifier: Costly consequence

In real use, the issue can cause loss of money, data, privacy or safety, even if the test caught it.

Include
The error sends money, deletes data, shares private information or cannot be undone. Add one level.
Exclude
The error only costs time and the participant can undo it. Do not add a level.
Example
“Wait, did that just send the report to the client? I only wanted a preview.”

Modifier: Repeats after learning

The issue happens again after the participant has seen it once, so practice will not remove it.

Include
The participant has the same problem on a later task or a second try. Add one level.
Exclude
The participant learned the fix and did not have the problem again. Do not add a level.
Example
“Again I pressed the wrong one. They look exactly the same.”

Report outline

  1. Tasks tested, number of participants and completion result for each task.
  2. Critical and major issues: the observation, the final level, the modifiers applied and a clip or quote.
  3. Minor and cosmetic issues in a short table.
  4. What worked well, so the team does not change it by accident.
  5. Recommended fixes with an owner, sorted by level.
  6. Method notes: who rated the issues and where the raters disagreed.

Mistakes to avoid

  • Do not rate the issue by how much the participant complained. Rate what happened to the task. Quiet participants hide critical issues.
  • Do not rate an issue higher because it is easy to fix. Severity and effort are two different columns.
  • Do not use a modifier to lower a level. A rare critical issue is still critical.
  • Do not mix opinions with issues. Level 0 comments go in a separate list, or they hide the real problems.

Guides that use it

Questions

It is a score that shows how much a usability issue hurts users. It helps the team decide which issues to fix first. This scale uses 5 levels and 3 modifiers.

Yes, but as a modifier, not as the base level. Rate the impact on one person first. Then raise the level if half or more of the participants had it.

Yes, if you can see or hear what the participant did. Rate from the behavior in the recording, not only from the answers to questions.

Yes. Reports in User Evaluation have a Usability findings template. It lays out the themes from the study reports with their clips. Add your severity levels to it by hand.

Ask real people your question

The AI moderator interviews screened people by voice or text. Each claim in the study report links to its clip. The first 3 interviews are free.

Start free