So much more than creating Frankenstein’s monster!

As part of our UX 101 education series, where we discuss the different types of studies and research methodologies you can use with our own user research platform, we’d like to introduce our readers to the exciting world of in-lab testing!

What is in-lab testing?

As the name implies, in-lab testing is when you conduct a study with participants that physically come into your usability lab. Whether or not you choose to wear a lab coat is entirely up to you (but be aware it might weird out your participants).

In-lab testing is often used to identify and mitigate usability issues, conduct formative usability testing, and observe participants as they interact with your product.

What are typical use cases for in-lab testing?

In-lab testing seeks to answer the question: How does the product perform in the context of use with different user groups?

This prevents the release of a product that is flawed or unsatisfying to users while allowing businesses to make changes that will enhance the usability, and therefore value, of the product. Without testing in the context of use the risk of releasing unsatisfying products is present.

Take this wireless mouse as an example, the Magic Mouse 2.

Having a charging cable eliminates the need to replace batteries in a wireless mouse, however, the location of the lightning port renders it useless while charging. You can see why testing this product in the context of use would uncover this design flaw.

It should be mentioned that another main advantage of using in-lab as your method is the ability to control and vary the testing environment (e.g. playing background sounds, furniture placement, downloading and installing, etc.) to better match that of the real world usage of your product. The goal is to uncover how users interact with products given realistic environmental situations such as low lighting, interruptions, sounds, glare, etc. These factors can be introduced and controlled in a lab setting.

Also, since in-lab testing is typically done in a moderated setting, it allows researchers and observers to watch the unconscious behaviors at play in participants that they might not otherwise be able to experience. It also allows researchers to ask follow up questions or spur of the moment deep dives into what’s happening with the participant.

As far as when this method is typically used in the product/design lifecycle, we most often see customers implementing it at one of these two times:

  • Identifying and solving usability issues early on in the design process
  • Validating final designs at the end of the design process

How does in-lab testing work?

To don my Captain Obvious hat for a moment – the key ingredient to conducting an in-lab study is having a *drumroll* usability lab. Don’t have a usability lab? No worries! Remote moderated is a methodology available to you that can deliver much of the same high quality results without the overhead of a lab.

The general process for setting up an in-lab study is the same as most research engagements. First and foremost outline what you want to test in a research plan (e.g. tasks, environment and metrics) and decide who you want to test with and how many participants you’ll need.

An important detail to note is that you will have to inform testers that they will be coming to your lab and give instructions as to how to get there, what time/date they’re expected as well as discuss compensation.

From there, of course, you’ll need to design and build your study.

Keep in mind that these are most likely going to be moderated sessions so you’ll want to brush up on moderated best practices such as not leading the participant and being careful not to introduce bias.

Also be aware that depending on the tasks you’ll be asking participants to do the session will most likely take between 30-60 minutes. Make sure to include time for housekeeping items like checking in, consent forms and introductions. We recommend not pushing participants to go beyond an hour as user fatigue can set in, resulting in less actionable insights. If you find that it is necessary, however, to go over an hour make sure to include a break and try to plan that break time in the overall session time you communicate to participants.

When should you use in-lab testing?

Here are essential facts to consider while deciding on whether or not in-lab testing is the right approach for your research goals.

Pros:

  • You have the flexibility to test with a variety of contexts/simulated environments that can impact usability. Oftentimes, products are QA tested in perfect conditions with the best lighting, optimal internet connections, free from distractions, etc. By testing in realistic settings you can find usability issues that otherwise would not have been found.
  • If the product is a tangible hardware item, in-lab testing is ideal for observing interactions (e.g. watching them open packaging and set it up and actually using it). Let’s not forget that amazing one-way mirror which allows observers to watch from a dark room, giving participants some “privacy” to behave naturally without the pressure of someone standing next to them. 
  • Can closely observe unconscious behaviors from participants who wouldn’t otherwise verbalize the why and how.
  • A moderator can observe and note subtle behaviors and ask insightful questions to probe participants and/or make their own analyses and conclusions for why this behavior occurred and what aspect of the product may contribute to this behavior.
  • Moderator can dig deeper and ask clarifying questions to follow up on participant comments and responses to interview questions.
  • If the product requires certain system requirements you can control for that by having the right equipment in the lab that is tested and quality controlled. Otherwise, in the wild, you can run into incompatibility issues with participants who may not always have the right browser, software etc.

Cons:

  • Participants tend to blame themselves for errors and lean toward positive statements about a product, however, their behavior (e.g. performance/comments/facial expressions) tell otherwise. To mitigate this, it’s important in the beginning to let participants know that honest feedback will only help designers improve the usability of the product and will not hurt anyone’s feelings. Feel free to remind them throughout if you sense they’re hesitating to tell what they really think.
  • Requires human factors specialists to analyze and interpret the why and how. Post-task interviews with participants to discuss why they took certain actions should be taken with a grain of salt and must be analyzed in light of human factors analysis.
  • In-lab testing can be more time consuming as sessions are typically for a single individual. If there’s a no show, you lose that time and opportunity to gather data unless a replacement can be found right away.
  • This methodology can be overkill for software-based products where not a lot of tangible interaction is necessary.

What results do you get?

The results of your study will depend on how you build it, but can largely be broken down into the following types.

Quantitative:

  • Performance: Percentage of success for each task, time on task, page views, clicks, etc.
  • Counting observations of task difficulties, errors and failures
    • Examples: 8 out of 10 struggled to remove a product out of packaging, 10 out of 10 skipped step 7 in the instructions, 2 out of 10 failed to insert a cord properly, etc.

Qualitative:

  • Subjective ratings: How satisfied were they? (not at all satisfied, slightly satisfied, moderately satisfied, very satisfied, and completely satisfied)
  • Comments: Capturing comments is really important especially if you receive the same comment from multiple people (e.g. the landing page is too busy or it’s confusing to understand). Hearing the same sentiment over and over should be documented and shared with stakeholders.
  • People tend to draw associations, both positive and negative, when experiencing a product for the first time. First impressions and attitudes are good to capture here. It’s important to determine whether their comment lands on the positive or negative side of the spectrum. At times, it can be difficult to tell when participants try to sugarcoat the truth and it’s up to the moderator must ask for clarification.
  • Behavior (e.g. facial expressions, sighing, cursing, hesitation, starting over)
  • Human factors analysis of contributing factors (e.g. human error or product design flaw)

Tips for analyzing your results

Before beginning a study it’s a good idea to gather a few colleagues in UX to identify all known and/or assumed usability issues of the product.

Use this list as a reference when taking notes on each participant as they go through the tasks, noting what errors they made and recommendations for mitigating those errors. Having this list beforehand makes it easy to begin categorizing errors (e.g. tabulating how many committed those errors and why they happened) and supporting your recommendations for changes.

Of course there is the classic UX adage that what participants say and do can sometimes be contrary to one another. This is why it’s up to UX researchers to analyze the whole picture. Were observed errors or difficulties a result of poor design or just human error? Running through multiple people who make the same mistake strongly point to a design flaw.

Finally, if you have run in-lab studies before or are planning on running some in the future, keep in mind that these two methodologies complement in-lab testing quite well:

Diary Studies: Diary studies are great to complement in-lab sessions. You can give participants a product to live with for a week or more and give daily diary assignments to take note of their experience. It’s a great exercise leading up to the in-lab session. You can use their homework assignment data as a launching pad for discussing their experiences with a product in their environments at work, home or traveling. You can also have them demonstrate what they wrote, illustrating the problems and issues they encountered.

Competitor Analysis: In-lab testing is a great way to compare two products and control the exposure and tasks with each product for fair side-by-side comparison. Each product can also be controlled so as not to bias participants by the order they are presented through counterbalancing.

This concludes our introduction to in-lab testing. Thank you for reading and don’t forget to check out the rest of our UX 101 education series to help you on your way!