What are ‘leading questions’ in UX tests and how can you avoid them?
A guide to leading questions, with advice on how you can be mindful of your phrasing and avoid wording pitfalls that can cloud your UX test.
What are leading questions?
In social science research, a leading question encourages a person to give a particular answer. Leading questions are not ideal for research because they introduce bias and influence the way participants respond.
Leading questions are also relevant to courtroom eyewitness testimonies. For example, if an examiner asks a witness whether he was home on the night of the murder, that’s a leading question. The phrasing assumes a murder indeed took place, and leads the witness to answer in a way that directly relates to his home. Hmm, sneaky.
Why do leading questions matter for UX testing?
Your job as a UX researcher is to uncover truth and honesty. Your job is to gather user feedback that isn’t coloured by your own hopes or expectations. Your job is to listen, and to be deliberate with your words and actions.
People often tell you things they think you want to hear. It’s driven by natural kindness, which is theoretically very nice! But if your survey or interview questions lead participants to give inaccurate answers, you might miss important insights.
Imagine the following scenario. You meet a research participant, Sally, and introduce yourself as a UX researcher. You say, “Thank you so much, Sally, for agreeing to participate in this research. I’m so excited to get feedback on this app. The team has worked very hard on it, and we know you’re going to love it.” Then, imagine you conclude your research session by asking Sally if there was anything she did not like about the app. She answers, “No, not really!”
Notice a problem? Sally went into this UX test already thinking a few things. First, you’ve probably given her high expectations for your app, and second, she doesn’t want to disappoint you or hurt your feelings!
Although it’s great to be polite and personable, it’s not a good idea to introduce too much emotion and personal expectations. Even worse, leading questions, like the one that gave Sally a hunch that you wanted her to respond favorably, led to responses that are not particularly true or useful. Strive to remain neutral and ask open questions.
Let’s look at specific examples
It’s easy enough to understand how leading questions are bad for UX tests, but leading questions can be tricky beasts. Here is a list of questions. Which ones do you think are leading?
- Would you rather use the old version or this improved version of the website?
- What did you expect would happen when you pressed that button?
- How would you normally perform this task, without this tool?
- Do you find this feature frustrating to use?
- Would you use this product if it were in your house?
- What are some pros and cons of these designs?
Now, let’s break them down.
- Would you rather use the old version or this improved version of the website? This is a leading question. By calling one version “improved,” you are already tricking participants’ brains into thinking that one is better than the other. It’s probably better to just call them Version A and Version B, or something like that.
- What did you expect would happen when you pressed that button? This is a good question, especially when the user has already pressed the button so you are not suggesting they do so. This question keys into the user’s mental model for how the product works.
- How would you normally perform this task, without this tool? This is another perfectly good question. It’s actually a really useful one, too, because it gives you extra context and juicy information about the problems you are trying to solve with the product.
- Do you find this feature frustrating to use? A bit leading. As a researcher, you can deduce that a feature was frustrating through your own data or by the user explicitly saying so. But by asking the question directly, users are likely to think too hard about it and you might not get genuine answers.
- Would you use this product if it were in your house? This question is not awful, but it is leading. Most people answer “yes,” so it doesn’t give you a lot of useful information. Try to get at this answer a different way. People really aren’t good at guessing about the future, so focus on past experiences and pain points.
- What are some pros and cons of these designs? Another fine question.
How can you avoid leading UX research participants?
1) Avoid direct questions with single-word answers
They may not always be leading, but avoid questions that can have single-word answers, like yes/no questions. Many participants give the shortest answer possible, especially for written surveys. Remember that you really want the juicy stuff: stories, explanations, and examples!
Whenever I talk about yes/no questions, I think about a scene from the Beatles movie A Hard Day’s Night in which the fab four respond cheekily to a barrage of reporter questions during peak Beatlemania. One reporter, obviously expecting a newsworthy response, asks George, “How has success changed your life?” and George responds with a curt “Yes.”
Consider how to word questions to get richer, non-leading feedback. Rephrase direct questions into open questions, like “Please describe an example of…” or “Tell me about a time you…” People are good at speaking on their own experiences, and not very good at elaborating on direct questions.
2) Practice and plan ahead
It takes practice for UX tests and interviews to come easily, and we all make mistakes along the way. But you can plan ahead to minimize these pitfalls. While prepping for user interviews, write out the questions you intend to ask beforehand, and group similar questions together. Get feedback from team members or other UXers. Just be flexible about when the questions might naturally come up during the tests.
If you’re writing a survey, have others look it over and double check your wording. And if you have the resources, consider a pilot test of the survey questions—give it out to a handful of people before deploying it to a large audience. Essentially, there’s no reason why you can’t usability test your test itself.
3) Be mindful of your nonverbal behaviour
Finally, beyond your words, your physical behavior can also be leading. Be encouraging and grateful for user feedback, but avoid responding too positively when participants say things you are hoping to hear, and avoid getting defensive in the face of product criticisms. Your poker face should be fully engaged.
Neutral behavior is easier said than done, so be mindful! During UX tests, I’ve seen research observers smile and nod when a participant behaves as expected, or point to features the participant didn’t see, or intervene to explain things the participant finds confusing. That’s all well and good during normal human interactions, but during a UX test? Please, don’t.
Of course, once the research session is over and you’ve collected your unbiased insights, feel free to open up, shake the seriousness off, and become human again with your participants. I actually really encourage that part!
Becca Kennedy — Guest Writer
Becca Kennedy is a Human Factors Psychologist and co-founder of Kennason, a UX consulting company in Upstate New York. Say hello on Twitter: @becca_kennedy
Thank you for signing up!
You should receive an email from us shortly