What is a Think-Out-Loud Study and how can you improve the feedback you get during a usability test?

“Thinking aloud maybe the single most valuable usability engineering method.” Dr. Jakob Nielsen wrote the above line way back in 1993, and so far, it has stood the test of time.

This is mainly due to the fact that when users are allowed to speak for themselves you can hear how they truly feel about your product or site, as well as gain an understanding of their misconceptions and why they have them. This kind of direct feedback, to paraphrase Jakob Nielsen, softens and convinces even the most hard-boiled developers, arrogant designers, and tight-fisted executives.

The only difference is that now instead of sitting down next to participants in a lab with a pen and a pad, companies are getting think-aloud feedback via remote usability tests. There are several reasons why, including being cheaper to run, geographical distribution of users, testing users in their natural settings, as well as the ability to recruit live users directly from a site, but this also presents a new challenge for companies. Namely, asking users to perform something most of us don’t normally do, which is keep up a continuous stream of talk as we perform an online task.

The good news is that there are several ways to encourage participants to give think aloud feedback during a remote usability study, it just requires a little bit of planning ahead. Here are a few tips to get you on your way.

1. Identify your goals beforehand

This is true for all studies, really. Before beginning any study make sure that you’ve had an in-depth conversation with all stakeholders involved regarding the business and research goals of the project. Business goals should help provide the horizon line for A) how to best structure any recommendations post analysis and B) what the research goals should look like.

Research goals should be granular and study specific, like: identify any glaring usability issues with the signup process, e.g. With these goals in mind, you can begin to craft tasks and prompts that will elicit feedback. 

2. Organize your design

For qualitative researchers, spreadsheets and tables might seem like the antithesis to creative free flow. Yet, I can’t stress enough the value of tabling your prompts and matching each question to a research objective and intended output. Take the following image:

Screen Shot 2016-08-29 at 11.30.29 AM

In unmoderated remote study design, it’s very important to be methodical, and precise about your questions and prompts. Unlike a moderated study, the ability to ask a follow up question to probe for further feedback is non-existent. Therefore, you must spend a bit more time on the front end ensuring your questions will provide the needed data.

Tip: If you’re unsure whether or not your question will generate the desired output, find a co-worker in your office, simply ask them the question, and see what they say!

3. Explain the technology very clearly

Don’t expect everyone to understand what a remote unmoderated study is, or even how to properly use the device they’re completing the study on. Guide the participant step by step: “If you’re reading this sentence, that means we’re recording you! Hold your phone in front of your face like you’re taking a selfie.”

One of the things we do at UserZoom is show participants a quick video that explains, and more importantly shows, them how to get set up and what what we’d like them to do.

4. Ask questions about them

In general, people love to talk about themselves. Use this natural human characteristic to your advantage. Warm up with questions like: “Tell me what you’re passionate about, and why” to get someone talking and in the mood to be chatty.

TipAvoid formal language like “what are your perceptions of this site element?” Even though the study is automated, remain approachable! Instead say: “What do you think about this part of the site titled XYZ? Remember, there are no right or wrong answers.” 

5. Tell a story

Every story has a beginning, middle, and end, and every unmoderated remote study should follow suit. This means not asking questions about checkout UX at the beginning of a study, or first impressions at the end. The questions and prompts should feel natural, and should appear at logical times within an experience.

This will help the user talk about them in a meaningful way since they won’t be struggling to recall events and how they felt at the time. 

Similar to asking appropriate questions at logical times, you should also be conscious of leading a participant towards answering in a particular way. Even though the study is unmoderated/automated, the participant will still want to please (even if subconsciously) the interviewer in one way or another.

For example if you’d like to gather any critiques a participant might have about a site, consider: “How do you feel about this website?” rather than “What did you dislike about this website?”

6. Try not to box in your users

While a usability test might lend itself to more task focused instructions, don’t be afraid to let participants explore. For example, imagine you want to test a website builder tool where participants must go through a number of different steps to ‘successfully’ build their site.  Rather than guide participants through every single step (e.g. first click here, then click here), simply provide them with an end goal:

“I’d like you to take all the steps necessary to build a website. Talk us through your process and any thoughts that come to mind as you do so.”

Later in the flow, you might provide a hint to ensure that those who perhaps couldn’t figure out how to build a website get back on track:

“You should now be on the page that says: ‘Your Website is Complete!’ HINT: If you aren’t there click the template module, then select the add to site button, and finally select complete. Once you are on the correct page, move to the next task.”

By asking a question in this way, you allow participants to show you their personal journey, without boxing them into your assumption about how someone should go about completing a particular task.

TipEnsure you capture the intended objective even if someone may have already touched on it prior. You can include language like: “You may have talked about this already, but please tell us what you think of XYZ.”

7. Shoot for the stars

The beauty of qualitative research is that you can really pull out meaningful feedback participants share about your experience or product. Ask creative questions, like:

  • “If you could wave a magic wand and make any changes to the site, what would you change?”
  • “Imagine you find yourself in an elevator with the CEO of [brand name] company. What advice would you give him or her?”

Give participants the “ok” to delve into their imagination, and speak from the heart.

Conclusion

It’s important to remember, and remind gung-ho stakeholders, that think-out-loud studies are not a panacea and shouldn’t be your go-to study for everything. They are simply a powerful tool to be used at the right time and in the right way. Should you decide that a remote think-aloud study is your best option, however, do keep these tips in mind.