Findings, severity, and prioritization

Objectives

By the end of the lesson, you can write an evidence-based finding from test notes, estimate the severity of a problem, and order the fixes not by personal taste but by user impact, frequency, risk, and effort.

Why are notes not enough?

At the end of the test your notebook is full of observations, quotes, and impressions. The next step is not to fix them one by one, but to organize them: what the real problem is, how serious it is, and what is worth fixing first. The finding is the tool with which a raw observation becomes an item that can be discussed and fixed.

The structure of a finding

A usability finding is a usability problem supported by evidence, relating to a task and a consequence.

Finding template

Field	Content
Title	Short, factual (not "bad")
Task	In which task it occurred
Observation	What we saw (with how many participants)
Consequence	What it caused for the task / the user
Assumed cause	A hypothesis, not fact
Severity	See below
Improvement direction	A hypothesis, not a finished solution

Connect identical phenomena, but do not forcibly merge different causes. Two participants may get stuck in the same place with different expectations; the difference can also be an important research result.

Severity

Severity is an estimate of the user impact, frequency, and recoverability of a problem. Three factors shape it:

1. Impact: how large is the impact on the main task? (prevents, slows, only disturbs)
2. Frequency: how many people and how often run into it in relevant use?
3. Recoverability: can the user restore the situation, or is the consequence lasting?

A four-level scale that works well in practice (similar to Nielsen's scale):

Level	Name	Meaning	Example
4	Critical	Prevents the task or leads to a wrong decision	It is not clear that the registration did not succeed
3	Serious	Slows significantly or requires outside help	They cannot find their registrations
2	Moderate	Annoying, but can be worked around	A confusing filter label
1	Cosmetic	Aesthetic or easily worked around	A tiny alignment error

Do not confuse the spectacular problem with the serious one. A conspicuous animation can be disturbing, but a tiny, misleading cancellation condition can have a greater consequence.

Priority and fixing

Priority is the justified order of fixes. Severity and priority are not the same: priority weighs severity together with the implementation effort and the uncertainty. For every finding write a fix hypothesis, not an immediately final solution. "Let's make it clear that the booking is final only after the next step" is better than "let's put a green button here". We detail the structure of a hypothesis in chapter 9.

To be weighed:

• the expected user impact,
• the implementation effort,
• the uncertainty (are we sure of the cause?),
• the consequence for other areas.

A quick fix is not always the most important; a high-impact but uncertain decision may need to be tested again.

From findings to decision

flowchart TD
    A["Observed behavior"] --> B["Finding<br>with evidence"]
    B --> C["User<br>consequence"]
    C --> D["Severity<br>(1–4)"]
    D --> E["Fix hypothesis"]
    E --> F["Priority<br>(impact, effort, uncertainty)"]

Worked example: time zone in Mai program

Finding: participants do not understand whether the selected time is shown in local or central time (the event is a foreign online broadcast). Impact: they may arrive at the wrong time, so the severity is high (3–4). Frequency: it came up with every tester. Recoverability: a missed event cannot be made up. Fix hypothesis: next to the time, the time zone tied to the location should appear, and it should be repeated in the confirmation too. After the fix, in a new test task it can be checked whether the participant states the time correctly.

Priority: small effort (text), large impact: to be fixed in the first round.

Common misconceptions

Claim	Clarification
"Every finding is equally important."	Severity and priority help with the order; a cosmetic error should not come before a critical one.
"The finding is the solution itself."	The finding is the problem; the solution is a hypothesis that must be checked.
"If only one participant got stuck, it doesn't matter."	A serious, unrecoverable obstacle is significant even with a single participant, but indicate the sample in the wording.

In brief

• A finding contains an observation, a task, a consequence, and an assumed cause.
• Severity is the combination of impact, frequency, and recoverability; the four-level scale helps with classification.
• Priority weighs severity against effort and uncertainty.
• The improvement direction is a hypothesis, not a finished solution.

Review questions

1. Does your finding contain an observation or an opinion?
2. What is the highest-impact error in your project?
3. Which of your fix proposals requires a new test?
4. How would you formulate the problem without a solution proposal?