By the end of the lesson, you can prepare a neutral test task and a moderator script. You can tell the participant's behavior apart from your own interpretation, you know when to observe in silence and when you can ask follow-up questions without suggesting a solution, and you can choose the metrics that fit the question.
Why is moderation the hardest?
The hardest thing for a designer in a test is to keep quiet. When the participant gets stuck, the designer's reflex is to help: "The button is there on the left." But this removes the data, since the purpose of the test is exactly to see where someone gets stuck without help. Moderation is conscious self-restraint, and it has rules.
A task, not a route
A test task is a neutral instruction embedded in a concrete situation for trying out the solution. The scenario (chapter 7) gives the situation, the task is the request. A test task is not the same as a user story (chapter 4): the story is the direction of design, the test task is a tool of observation.
The characteristics of a good task:
it speaks in the language of the goal ("Find a program that..."), not the language of the interface ("Click the Filters button");
it contains no leading words that give away the label of the interface;
it contains as much background information as needed, but not more; if the task can be completed only if the tester thinks exactly like the designer, you are not examining the interface;
its success condition is fixed in advance, so that afterwards you do not fit the evaluation to the desired result.
In one test, three to five tasks are generally enough. Start with an easier, orienting task, then move toward the main goal and the critical decisions. Do not chain the tasks so that an earlier error makes the later one impossible; if necessary, reset the prototype or give a new starting point.
The role of the moderator
Moderation is leading the test situation and supporting the participant without suggesting the solution. The moderator helps make the situation safe, but does not help in solving the task.
What can the moderator do?
Ask for thinking aloud (you learned the method in chapter 2): "Tell me what you are looking for or what you expect."
Ask neutral follow-up questions: "What do you think would happen if you chose this?" "What would you have expected here?"
Reinforce participation, not the correct answer: "Thank you, this is very useful."
What can they not do?
They do not say: "Try the top right menu."
They do not explain how it should have been done.
They do not defend the design ("This is this way because...").
They do not correct the participant's mistake.
Getting stuck is data. Do not rescue the participant too early, because that makes the problem disappear. If the task is no longer progressing and must be stopped because of the time frame or emotional tension, briefly mark the point, say that this gives information about the design, and move on. At the end, ask whether there was a moment when they were uncertain, but let the observation remain more important than the after-the-fact explanation.
The skeleton of a moderator script
Introduction (consent, testing the design, no right or wrong).
Warm-up question about the participant's background.
Tasks one by one, reading out the scenario.
Post-task question for each task: "What was this task like? What was the most uncertain moment?"
Closing questions: overall impression, what they would change.
Thanks, closing.
Note-taking
A simple table is useful:
Task
Observation
Verbatim quote
Consequence
Assumed cause
Open question
Registration
Returned to the home page three times while looking for the change
"Where can I see what I registered for?"
Did not find the "My registrations" page
The menu item is under the profile
Would they expect it in the main menu?
"The navigation is bad" is not an observation. "The participant returned to the home page three times while looking for how to change the booking" is. A later interpretation can be: "the change is not in an expected place", but mark this as a hypothesis. We separate the same three kinds as in chapter 2: quote, observation, interpretation.
What should we measure?
The metric should belong to the question. The concepts of critical error and success criterion are familiar from earlier chapters; now we look at the tools of measurement. In an early usability test it is often useful to choose from the following:
Metric
What does it mean?
When is it useful?
Task success
Did the participant reach the goal (yes / partly / no)
Always
Requests for help
How many times they asked for help
To examine comprehensibility
Number and weight of errors
How many errors they made, was there a critical error
To examine error prevention
Task time
How long it took to complete
Only if speed is a real value and the tasks are comparable
Observable uncertainty
Stopped, went back, doubted
A qualitative signal
Comprehension
Can they correctly state the consequence in their own words
For content and trust questions
Satisfaction (SUS)
Subjective rating with a questionnaire
For comparison
A critical error is an error that prevents or seriously misdirects the main task. Time measurement is meaningful only if the tasks and participants are comparable, and speed is truly a value. In a health decision, too fast a choice can even be a risk.
The SUS (System Usability Scale) is a questionnaire of ten statements that the participant answers on a five-point scale; from the answers a score between 0 and 100 can be calculated. It is a useful quick, comparative number, but on a small sample (3–5 people) it cannot be generalized; check its description and the rules of calculation in the literature before using it.
A qualitative signal can also be that at the end of the process the participant can correctly state the consequence in their own words. For example, do they know that the registration is final, when the notification will appear, and how they can change it? This measures comprehension, not the number of clicks.
Worked example: the test task of Mai program
User story (chapter 4): "As a student looking for a spontaneous program, I want to compare today's events..., so that I can quickly decide which one is worth going to." Scenario: "It is Friday evening, you want to go out somewhere with four friends. You do not have much money, and not everyone speaks Hungarian well." Test task: "Find an option that you can decide for sure is suitable." Success criterion: the participant finds a suitable event within two minutes, names the time, place, cost, and registration requirement, and does not misinterpret critical information.
A participant does not find the confirmation after the booking. The moderator does not point out where it is, but asks: "Where do you expect this to be?" The participant says they would expect an email, then looks in their profile. The finding is not that "they did not find the button", but that the system did not make a clear promise about the channel and place of the confirmation.
Common misconceptions
Claim
Clarification
"A test task is made from the user story."
The direction of the story is design; the test task is a request embedded in a concrete situation that does not reveal the solution.
"A metric is always a number."
A clear, predefined observation or correct understanding can also be a valuable metric.
"The test task has to name the function."
This may hide whether the user would find or recognize the function at all.
"The moderator's job is to help the participant."
Help removes the data; the moderator creates the safe situation.
In brief
The test task gives a goal, not a control; the success condition is fixed in advance.
The moderator keeps quiet, asks neutrally, and does not rescue from getting stuck.
In the notes, keep the observation, the quote, and the interpretation separate.
We choose the metric for the question; on a small sample the qualitative signal is more important than the number.
Review questions
Does your test task name the goal or the control?
What sentence would you use to ask for thinking aloud?
When would you interrupt a case of getting stuck?
Write an observation and an interpretation separate from it.
Which metric would best show that the user understands the consequence?
Tasks, moderation, and metrics
Objectives
By the end of the lesson, you can prepare a neutral test task and a moderator script. You can tell the participant's behavior apart from your own interpretation, you know when to observe in silence and when you can ask follow-up questions without suggesting a solution, and you can choose the metrics that fit the question.
Why is moderation the hardest?
The hardest thing for a designer in a test is to keep quiet. When the participant gets stuck, the designer's reflex is to help: "The button is there on the left." But this removes the data, since the purpose of the test is exactly to see where someone gets stuck without help. Moderation is conscious self-restraint, and it has rules.
A task, not a route
A test task is a neutral instruction embedded in a concrete situation for trying out the solution. The scenario (chapter 7) gives the situation, the task is the request. A test task is not the same as a user story (chapter 4): the story is the direction of design, the test task is a tool of observation.
The characteristics of a good task:
In one test, three to five tasks are generally enough. Start with an easier, orienting task, then move toward the main goal and the critical decisions. Do not chain the tasks so that an earlier error makes the later one impossible; if necessary, reset the prototype or give a new starting point.
The role of the moderator
Moderation is leading the test situation and supporting the participant without suggesting the solution. The moderator helps make the situation safe, but does not help in solving the task.
What can the moderator do?
What can they not do?
Getting stuck is data. Do not rescue the participant too early, because that makes the problem disappear. If the task is no longer progressing and must be stopped because of the time frame or emotional tension, briefly mark the point, say that this gives information about the design, and move on. At the end, ask whether there was a moment when they were uncertain, but let the observation remain more important than the after-the-fact explanation.
The skeleton of a moderator script
Note-taking
A simple table is useful:
"The navigation is bad" is not an observation. "The participant returned to the home page three times while looking for how to change the booking" is. A later interpretation can be: "the change is not in an expected place", but mark this as a hypothesis. We separate the same three kinds as in chapter 2: quote, observation, interpretation.
What should we measure?
The metric should belong to the question. The concepts of critical error and success criterion are familiar from earlier chapters; now we look at the tools of measurement. In an early usability test it is often useful to choose from the following:
A critical error is an error that prevents or seriously misdirects the main task. Time measurement is meaningful only if the tasks and participants are comparable, and speed is truly a value. In a health decision, too fast a choice can even be a risk.
The SUS (System Usability Scale) is a questionnaire of ten statements that the participant answers on a five-point scale; from the answers a score between 0 and 100 can be calculated. It is a useful quick, comparative number, but on a small sample (3–5 people) it cannot be generalized; check its description and the rules of calculation in the literature before using it.
A qualitative signal can also be that at the end of the process the participant can correctly state the consequence in their own words. For example, do they know that the registration is final, when the notification will appear, and how they can change it? This measures comprehension, not the number of clicks.
Worked example: the test task of Mai program
User story (chapter 4): "As a student looking for a spontaneous program, I want to compare today's events..., so that I can quickly decide which one is worth going to." Scenario: "It is Friday evening, you want to go out somewhere with four friends. You do not have much money, and not everyone speaks Hungarian well." Test task: "Find an option that you can decide for sure is suitable." Success criterion: the participant finds a suitable event within two minutes, names the time, place, cost, and registration requirement, and does not misinterpret critical information.
A participant does not find the confirmation after the booking. The moderator does not point out where it is, but asks: "Where do you expect this to be?" The participant says they would expect an email, then looks in their profile. The finding is not that "they did not find the button", but that the system did not make a clear promise about the channel and place of the confirmation.
Common misconceptions
In brief
Review questions