Everything you need to know about planning, running and analyzing usability tests.
Welcome to our comprehensive guide to usability testing, where you'll learn everything from deciding when to conduct usability tests, to preparing tasks and questions, to deciding whether to go moderated or unmoderated, to analyzing and presenting your insights.
A usability test uncovers insights and issues that people may experience when using your product or service. The test comprises a series of tasks completed by a test participant, while you capture their feedback, actions, behaviors and/or spoken-aloud thoughts.
A well-conducted usability test will uncover rich insights like why users aren’t navigating past a certain point of your website and what information they might expect to find to help them make key decisions.
Usability tests are not the same as explorative research or market research. You can include explorative research questions before showing users the product, but usability testing is about understanding the ease-of-use of your designs.
Using these insights you can start to understand how a design needs to change to better suit your customers’ needs and increase engagement and/or conversion rates for your business.
Brace yourself to use a lot of quotes instead of statistics when advocating for change based on these insights. Analytics can help uncover where users are leaving the website, but usability testing will uncover insights into why users are leaving the website and their experiences and impressions of your product or service.
When embarking on usability testing, you’ll have to decide whether you want to run the sessions moderated or unmoderated.
Moderated sessions are where you observe the user in a live setting (this can be done in-person or via a remote video link) and you can facilitate the conversation.
You would likely choose this moderated if you need to further explore why someone is behaving in a particular way, rather than just seeing what users do on their own and the issues they come across.
Unmoderated means that the user is free to do the tasks on their own at any time of their choosing. This is a great choice because it completely automates the lengthy process of data collection, and as a budget consideration: you won’t have to cover travel and refreshment expenses for people coming to your office.
The major factor to be aware of when deciding between moderated and unmoderated testing is the style of your questions.
For unmoderated, you need to be clear in your instructions and explain the context well so that the script, scenarios and tasks make sense to the user without any further clarification.
Moderated tests are more forgiving in that you can clarify questions throughout the session and help users if they become stuck on the site.
While running a pilot session to test a discussion guide itself is highly recommended, doing so for remote testing is a must because no one will be there to help the user in the session. A whole session can be lost if a user misinterprets the task or question.
For a demo of a remote moderated usability test in action, check out our brand new product walkthrough:
You can perform a usability test at any point during the product development lifecycle, and we normally refer to testing done early or later in development as either ‘formative or summative’ respectively.
Formative usability testing is done early in product development to help form the product’s shape and design.
This testing answers the why and how questions of the design’s usability. It answers why something is not working and involves iteratively evaluating a product during design and development.
The earlier you conduct a usability test, the better for your business and your users. If you conduct usability testing on designs as early as a prototype or sketch fidelity, usability issues can be caught early and won’t require development to fix.
Of course, this is not always feasible. Usability tests may come later in the product development lifecycle…
Summative usability testing is usually performed later in the product development process when a product is fully developed.
It is often conducted when a design is reasonably complete and involves evaluating the design against quantitative goals or a competitor’s products.
If you’re conducting usability tests later in the product life cycle, you’ll need a more holistic approach across the product or service, rather than testing one or two features at a time in siloes.
For this article, we’ll mainly focus on formative usability testing.
The beauty of moderated usability testing is that it grants you time with users.
While the usability testing part of the session focuses on your product, you can still ask some explorative questions beforehand to uncover insights into how users behave, make decisions, and think and feel in the context of your product or service.
Follow-up questions after completing the tasks are also valuable, as they can help you dig deeper and learn more about why users struggled with a task or their overall perception of an experience.
These insights can also serve future designs because you and your team will have more empathy for your users and a better understanding of their needs.
A general rule of thumb is to test frequently with around five users, as insights start to become repetitive after the fifth user. Based on the assumption that you will be testing iteratively and after each round of testing, your revised design will be tested again with another five users.
However, bear in mind that five is only enough to uncover 85% of issues. So the higher the sample size the more issues you will find. If your design has matured and lots of optimization is needed, try testing with larger sample sizes of 20, 30, or even more per segment.
If you are testing with more users for a larger test, be sure to recruit as many types of users or market groups as possible. You may also need to consider recruiting for one or two more participants in case there are no-shows or last-minute cancellations, especially if you are committed to delivering insights from a set number of users in a contract.
As John Romadka states in his guide to preparing a simple usability study,
“Crafting tasks and scenarios for a usability study is a balancing act. In order to get realistic insights, you need to provide enough information so users aren’t guessing what they need to do, while also not providing too much information and biasing their chance of succeeding. Therefore it’s important to clearly define what each task is and isn’t.”
You also need to make sure you have an agreed-upon plan which includes a clear definition of the overall study’s purpose, the business and/or research objectives, and the research questions you want to ask.
To aid this process, John has developed a template that helps guide you through the task writing process.
Example of the pre-filled task template:
Now that you know what the goals are of your usability test and how many users you’re interviewing, you can start to plan what you’re going to say in the sessions by writing your discussion guide.
This is deliberately not called a script, because it is completely fine to deviate from your questions to explore or probe user behaviors and their answers. Just try to roughly stick to the topics you want to cover to ensure you achieve what you set out to do in the session.
You can share your discussion guide with stakeholders and colleagues as a way to get their investment in the test by adding their own questions.
The structure of your discussion guide will need to be tweaked depending on your chosen technique of usability testing, which will be touched on in the next sections.
There are many resources on how to write questions for a discussion guide, like Sarah Doody’s User Research Interview Guide and Steve Portigal’s Interviewing users, but here are some common tips to quickly bear in mind as a general rule of thumb for writing questions:
Remote testing, as we are all now probably familiar with, is interviewing users online, via video software.
In-person testing is where you’re sat next to the user, in an office or usability lab environment.
For usability tests involving more sensitive topics, you may want to consider remote testing. This allows users to participate from somewhere they know and be more comfortable than in a lab with cameras they’re not familiar with.
Although, with remote calls, you lose control over your testing environment. Users can dial in from anywhere, like a home with howling pets or an airport with poor WiFi. Be sure to tell users an ideal location from where to take the call in their invitations!
Another advantage to remote testing is that a user is likely to use their own devices to take the call for the usability test. This gives you rich insight into how users set up their desktops, how many tabs they have open and applications they use side-by-side. You get to see what users do, rather than just hear what users say they do.
The downside to this is that users have all their information open on their machine. One click to their email tab and you capture their sensitive information and most likely without a business need for it. This may be a privacy issue related to regulations like GDPR and may require extra video processing time to edit out this information later.
In-person testing is also a lot more personal than remote because you’re physically closer and you have body language to help build rapport. In-person testing also has the advantage of seeing gestures. So, when users say “I like this information here”, you can see where they’re pointing on the screen.
And if there are any technical issues with the machine during the session, you are on-hand to fix them, meaning less time of the call is spent on watching someone experience technical issues.
The next essential technique for any formative moderated or unmoderated usability test is the ‘think-out-loud’ instruction. This is where you encourage users to tell you what they’re thinking throughout the session. From first impressions, what they expect to happen, their feelings along the way, and whether the product or service matched their expectations.
What the user says is where you will uncover a lot of insights, because they will tell you what issues they encountered and the impact of the issue.
Talking through a thought process is not necessarily natural for everyone. Some users may need extra encouragement. Mentioning at the beginning of the session “Feel free to tell me any positive and negative thoughts as we go along” will help clarify the type of feedback you want without leading users.
Definitely highlight this as an initial instruction for remote unmoderated sessions as you won’t be there to probe users to think-out-loud.
No matter which techniques you choose or how much you plan, things may not always go according to plan. Usability testing is not exempt from the diversity and the complexities that make us human.
Some users will tell you every thought they have, while others will hardly say anything at all. Some will be excited to tell you their life stories, while others will be anxious about your questions.
Know that moderating a wide variety of people and conversations takes a lot of cognitive and emotional effort. There are books to help you understand how to moderate different situations, but you won’t know how you will react until you’re in that situation.
Remember to be flexible, adapt to the situation and most importantly… don’t forget to breathe!
Usability tests can lead to hundreds of insights being uncovered, so knowing how to analyze these insights is important to tell your users’ stories in an impactful way.
The first part of understanding an insight is to understand the context of your participant and what led to the usability issue. For example, if the participant within the first five minutes declared they would never buy a car on finance as they would buy it outright, and you’re testing the finance options of a car website, so their insights may not be relevant.
The most impactful insights are where you can safely bet that users would have experienced the issue at home and without you there because they would have behaved the same way.
Once you’ve identified outputs as insights, coding each insight across participants will then help draw out overall themes. Use the number of participants who experienced each issue along with the impact to rank each issue by severity.
Tools can be used to physically tag words in interview notes and automatically highlight themes across participants. The use of color-coding your insights across categories or issue severity can also help provide a visual understanding of insights and themes.
The last step is telling the user’s story by presenting your findings to stakeholders to truly champion your users!
Slide decks are great to linearly tell the story and aid you to physically present the findings. Although decks can lack detail without your dialog when others read it at their discretion. But the succinctness of a deck is great for stakeholders who don’t need the low-level details.
Reports are another option, offering a lot more detail than a slide deck. However, these can be quite long and difficult to read. While you have a detailed record of the test, they may not be as impactful if fewer people read the information.
Regardless of the format, when presenting usability insights tell the story of your participants; begin with who they are. Then go on to explain their goals in the context of what they wanted to achieve with your product, their pain points along the way and the impact this has on the business.
User quotes and video clips are indispensable for directly sharing research and invoking an emotional response in your audience. Seeing a participant struggle first-hand is more impactful than reading there was an issue.
Consider presenting the insights based on a scale, starting with the most severe through to observation level categories. A severe issue would have a large impact on both users and businesses – such as loss of custom.
An observation would be something that improves the user experience but won’t necessarily improve conversion, like grouping pieces of information closer together.
Prioritizing insights enables your team to understand where to start actioning insights, and improve the usability and overall user experience, which is the first step to preparing for your next usability test!