Research questions and choosing a method

Objectives

By the end of the lesson, you can turn a design uncertainty into a research question, choose a research method proportionate to the question, and recognize when a research result gives behavioral evidence and when it gives only opinion or assumption.

Why do we research?

In the first chapter you formulated a problem and marked a few risky assumptions. The job of research is to check them. It is important not to collect data just because "we have to do research". Research is good when it reduces uncertainty before a decision. If the answer to a question does not change the plan, it is not worth asking.

The purpose of research: reducing uncertainty

UX research (user research) is a set of methods with which we study users' behavior, needs, and motivations so that we can make better-founded design decisions.

A research question is an open question whose answer influences a design decision. Suppose the designers of Mai program want to know how people choose an event. The useful question is not "Would people like a new app?", because it is easy to answer that with a polite yes. Much better questions are:

• How does a student find a suitable program today?
• Based on what information do they choose?
• Where do they become uncertain?
• What happens if they cannot find anything suitable?

Structural, content, and interaction decisions can follow from these answers.

What makes a research question good?

A question is good if it is narrow enough to be researchable, but does not suggest the answer in advance. Let us compare:

Weak question	Why it is weak	Better version
"Would you use an app like this?"	Hypothetical; we get a polite yes	"Tell me about the last time you looked for a program!"
"Why don't they use the calendar view?"	Assumes that the calendar view is the problem	"How do they compare the available time slots?"
"What is wrong with event pages?"	Suggests that something is wrong	"What does an average search look like from beginning to end?"

The common feature of weak questions is that they either ask about future intent (which people predict badly) or decide the cause in advance.

Exploratory, evaluative, and measuring research

Based on the purpose of the research, we can distinguish three common types.

Exploratory research is needed when we do not yet understand the situation. Here we are interested in goals, habits, obstacles, language use, and workarounds. A good method can be a semi-structured interview, contextual observation, or analysis of existing customer service feedback. At the start of the project this is the most important.

Evaluative research is needed when there is already a design or prototype. Here we examine whether the user reaches the goal, understands the content, and finds the function. This includes usability testing, tree testing, or walking through a clickable prototype. We deal with this in detail in chapter 8.

Measuring research tries to understand how frequent something is, or compares two options. A questionnaire, analytics, or an experiment can be useful, but only if the sample and the way of measuring are suitable for the question. You cannot claim a percentage from three conversations; but three similar obstacles can be a strong signal for improving a prototype.

Qualitative and quantitative data

Research data falls into two main groups.

	Qualitative	Quantitative
What does it show?	Why and how something happens	How much, how often
Typical method	Interview, observation, diary	Questionnaire, analytics, A/B test
Sample	Small (a few people), in depth	Large, for statistical inference
Strength	Discovering new problems, causes	Proportions, trends, comparison
Danger	Not generalizable	Does not reveal the cause

The two complement each other: quantitative data shows where there is a problem, qualitative shows why. In a semester student project, qualitative methods are almost always the right ones, because they are instructive quickly and with few resources.

Choose a method for the question

A method is the procedure chosen to answer the research question. The following are the most common:

• Interview: suitable for understanding past situations, motivations, and people's own words. Quick, flexible, but people rationalize afterwards.
• Observation: shows what someone actually does when performing a task; often different from what they say afterwards.
• Questionnaire: well usable for checking known patterns on a larger sample, but poor at discovering new problems, because we have to come up with the questions in advance.
• Examining existing solutions (competitor or desk research): you look at how others solve the same task today. Cheap, fast, and clears up many questions in advance.
• Usability test: does not tell whether the product is needed, but whether the chosen solution works in the task.

In a small student project, research time is often the biggest constraint. In that case you do not need to increase the number of methods, but to refine the question. Three relevant, well-documented conversations can be worth more than fifty answers to a questionnaire that only measures whether people in principle like the idea.

A research plan in minimal form

Even for a short study, make a one-page plan. It contains:

1. The decision that needs knowledge.
2. The research questions and the assumptions related to them.
3. The description of the participant from the point of view of the task.
4. The method, the time frame, and the way of handling data.
5. What counts as enough evidence for the next design step.

This is not bureaucracy. It helps avoid the conversation suddenly turning into a product demo in the middle of the interview, or making a decision afterwards based on the most memorable quote.

Worked example: the research plan of Mai program

Based on chapter 1, the Mai program team marked one risky assumption: "From event pages it is not quickly clear what the price, start time, language, or registration is."

• Decision that needs knowledge: what data should be on the event card, and in what order.
• Research questions: How does a student look for a program today? Based on what information do they decide? Where and when do they get stuck?
• Method: four semi-structured interviews, 25 minutes each, recalling the most recent search; in addition, two observations where the participant shows a real search.
• Participants: students who looked for an evening program at least once in the past month.
• Data handling: anonymous notes, no recording, the team keeps the notes until the end of the semester.
• Enough evidence: if at least two out of three participants mention the same data as missing.

Common misconceptions

Claim	Clarification
"The goal of research is to confirm my idea."	Research is useful if it can also refute or modify the assumption.
"A questionnaire is objective, an interview is subjective."	Every method has biases. In a questionnaire, the wording of questions and the selection of respondents, in an interview, the behavior of the interviewer can influence the result.
"You cannot learn anything from a small sample."	You cannot estimate general proportions, but you can indeed identify recurring task obstacles.
"Users tell us what to build."	We learn their situation, difficulty, and words from them; designing the solution is the designer's job.

In brief

• The goal of research is to reduce uncertainty before a decision, not data collection in itself.
• A good research question is open, narrow, does not suggest an answer, and is tied to a design risk.
• The method is determined by the question: exploration, evaluation, or measurement.
• Qualitative methods show the causes, quantitative the proportions; in a small project the former is usually enough.
• Even a short study should have a one-page plan.

Review questions

1. Which of your design decisions does not yet have enough evidence?
2. Why is "Would you use an app like this?" not a good research question?
3. Which method would best show the current workaround in your project?
4. What would make your research proportionate to the available time?