If you’re running moderated user research you need to learn how to ask the right questions, in the right way.
But what does that actually does this mean and why is it important?
The ‘right’ questions are ones that do not influence or lead or coax a test participant. Instead they reveal truthful opinions and behaviours without your own bias. This is important because if your tests are loaded with bias, the insights won’t lead to a true user-focused product.
To help us understand this further and uncover how we put this into action, we sought the advice of a variety of senior research leaders on how they frame their user research in order to surface the most useful and valuable insight. Here are their own detailed recommendations on how to ask the ‘right’ questions in the ‘right’ way.
For an in-depth, practical guide to successfully planning and conducting Remote Moderated Research check out our free-to-download, comprehensive ebook.
I think the hardest part about asking the right questions in the right way is being able to clearly identify any bias and internal/external influences you are bringing into the research project.
Vivianne Castillo, Salesforce
Senior Design Researcher
Some examples of this include (but aren’t limited to):
Belief bias (e.g. religious, political, socioeconomic, etc.)
Hindsight bias (e.g. seeing past actions by people more predictable than what they actually are; the “I knew it all along!” effect).
Crafting leading questions in order to please certain stakeholders.
Knowing this research project might influence your next promotion.
Lately, before I begin crafting a research plan, I’ve been jotting down (yes, with pen and paper) what biases/influences I am bringing into the questions I want to ask. I ask myself questions like, “How do I view people who always desire luxury?” or “What do I believe about the leaders of Brand X versus Brand Y?”
After crafting my questions, I always run them by another co-worker or two, because I know that I’m human and am not immune to blindspots.
I find that this practice goes beyond helping me gut check my questions: it increases buy-in and collaboration with my team and helps me notice patterns in how I ask questions, which influences how I approach future projects.
You’ve definitely hit on some of the biggest challenges UX Researchers face. Given my background as a trial attorney, I’ll focus on the topic of ensuring you’re asking the right questions in the right way.
Bradley Miller, Honeywell
UX Research Manager
What are the ‘right questions? As UX Researchers, we need to focus on understanding user behavior. That means observing what users actually do, and attempting to understand the thought processes behind why users do or don’t take certain actions. In order to do this, first focus your questions around tasks. What are the major tasks a typical user would need to complete?
Of those, which ones are the most critical decision points for your team or business to understand? For those latter tasks, be sure to ask the user follow-up questions, such as why they made the choice they did.
What’s the right way to ask questions? There are two components to asking questions the right way – flow and form.
Flow refers to the order of your questions. An ideal flow follows the user’s natural path through the experience. It’s important to follow, rather than lead, as much as possible. Only by following can you observe a user’s natural decision making.
The art to doing this is to avoid dictating or suggesting a specific path whenever possible. Instead, if you’re trying to test or observe a specific portion of an experience, ask the user to imagine she is trying to complete a task which would naturally include the thing you want to observe.
Form is just as important as flow. UX Researchers should always use direct, open-ended questions, and avoid leading questions unless absolutely necessary. What is a leading question? A leading question is any question that suggests an answer. By contrast, an open-ended question does not suggest an answer.
For example, the question “Did you find this experience delightful?” suggests the participant should either agree or disagree with your statement; you’ve therefore influenced her answer by limiting and characterizing how she should respond. A better question would be, “How was the experience?” This question is entirely open-ended: the participant is free to answer in her own words.
By allowing participants to answer in their own words, you’ll get a higher fidelity insight into how they characterize the experience.
There are a few ways to ensure you are asking questions in the most unbiased and least leading way possible. As researchers, we are constantly trying to understand the why behind someone’s thoughts, emotions and behaviors. The method I teach most of my students, which is the easiest to remember, is the TEDW method.
Nikki Anderson, From A to B
Senior UX Researcher
The TEDW method promotes conversations over interviews, by ensuring you are asking open-ended questions. TEDW stands for: “Tell me about…?” Or “Explain what you mean by…?” Or “Describe…?” Or “Walk me through…?”
Stating questions in this way ensures you are not leading the participant down a path only you care about, and allows them to answer beyond yes or no, which gets you closer to the why.
My second piece of advice is to combine the TEDW method with always asking about the past. The best way to predict future behavior is through past behavior, so asking a participant how they have done something in the past is the best way to figure out how they would do it in the future.
I start every project at the end – what do you need to solve for with this research? What decisions do you need to be able to make when it’s completed? It is critical that everyone involved is crystal clear on those things.
Angie Amon, Mailchimp
UX Research Manager
Having this info allows you to decide on the right methodology or methodologies and then from there, the specific questions that need to be part of the research.
Whether you’re talking about qualitative questions or questions in a survey, avoiding any bias is critical. Questions need to be as neutral as possible – no using inflammatory words or phrases that might push a respondent in one direction or the other.
Here’s an extreme ‘don’t’ example: “How horrible is this website?” Consider instead a more neutral approach: “Which of the following words best fits how you feel about this website?”
I always suggest running your questionnaire or discussion guide by a neutral party who isn’t invested in the outcome to make sure that they understand the questions. There’s no confusion or compound meaning in a question. If you have an open-text question, don’t ask it in such a way that a yes or no answer will suffice because that’s what you’ll end up with.
I don’t know that you can absolutely ensure that you are always asking the right question in the right way but having a clear vision of what you need from the research is a great starting point. Ask yourself and those who need to take action from the results, “If you get a majority of people answering this question a certain way, will it be clear to you what to do or what not to do with the information?”
Understand why we are asking the questions that were proposed.
Samantha Alaimo, GrubHub
Senior UX Researcher
Start with broad questions and then narrow in on the particulars.
Think of your users as the experts, you are just there to learn from them.
Follow the users down their rabbit holes if their answers go in a different direction than you expected.