A very handy guide to moderated and unmoderated testing

When you have the full blessing of your stakeholders to run some user research on your product, and you know the areas in which you want to test and/or improve (all fully aligned to business goals, of course) you’ll then have to decide whether to run your chosen studies moderated or unmoderated.

Within this debate, you’ll also have to figure out why one might be better suited than the other in your specific context, and when both would complement each other very nicely.

Is the distinction between moderated and unmoderated as difficult to grasp as figuring out the difference between flammable and inflammable? Possibly not, but whether you’re trying to surface valuable user insights or set fire to your latest failed DIY project, it’s good to have these things straight in your mind so you can access the right tool (i.e. card sort or bottle of paraffin) as quickly as possible.

Unmoderated vs. Moderated research: [tl;dr version]

The difference between ‘moderated’ or ‘unmoderated’ is quite simply whether a researcher (a moderator) is going to be present during the test, or whether the test participant is left to carry out the task without anybody else in the room (or on the other end of a computer).

But if that’s too tl;dr, let’s dig deeper into the benefits and challenges of both unmoderated and moderated:

What is unmoderated testing?

Unmoderated tests are how we describe unobserved tests, where a participant is left alone to complete tasks without the presence of a moderator. These sessions can be recorded for later viewing as part of a qualitative study (the results are the spoken out-loud thoughts and feelings of the participants), or the data is collected and analysed as part of quantitative research (cold hard numerical based results – how many, how often or how much).

Unmoderated testing methods

Typical unmoderated tests include (but are not limited to):

• A/B Testing
• Card Sorting
• Click Testing
• Diary Study
• Surveys
• Tree Testing
• True Intent Studies
• Usability Testing

Also bear in mind that many of these can also be run as moderated sessions. Ah UX testing, you flexible, wonderful thing.

Benefits of unmoderated testing

• Since a majority of the study is being run automatically with multiple testers, you can get more responses in a shorter amount of time
• Allows you to collect statistically significant amounts of data that give you high confidence in your decisions
• Gives you access to people over a much more broad geographical location
• Enables the democratization of research at organizations by letting multiple teams conduct research engagements without needing to be trained as a moderator

Challenges of unmoderated testing

• Will need a tool to conduct unmoderated testing, can’t be done with low-tech options like a pen and paper
• You get results after the fact, meaning if there was something you wanted to look into in more depth you would need to run another round of qualitative testing
• Relying on people to take studies from remote locations means that you remove some of your own quality control

What is moderated testing?

In moderated testing, the participants are observed by a moderator, either in-person or remotely via computer.

The key reason for running moderated sessions is so that you can be in a live setting with a participant. This allows you to have a conversation with your users as you’re observing what they are doing to better understand their behavior and dig deep into usability issues and attitudes.

Plus, you can modify your test script on the fly to probe more if there are confusing areas and ask ad-hoc follow-up questions.

Speaking of ad-hoc questions – another advantage to conducting moderated sessions is that you can have stakeholders and teammates anonymously observing the sessions. This is a great way to include people who are not usually involved in usability testing to observe the process firsthand, which can foster empathy with users and reinforces the need for further user research.

Moderated testing methods

Typical moderated tests include (but are not limited to):

• Card Sorting
• Click Testing
• Ethnographic Study
• Focus Groups
• Interviews
• Lab study
• Remote Usability Testing

Again, some of these can also be run unmoderated. Apart from the Interviews, that would be uh… weird.

Benefits of moderated testing

• Moderators can ask additional questions as needed during the test, and go as deep into those follow up questions as is required
• Allows for in-the-moment discussions into unexpected or unplanned behaviors from your testers that would be missed with unmoderated or automated testing
• Seeing is believing, and having key stakeholders present as note takers or observers can help them build empathy for your users

Challenges of moderated testing

• Moderated testing can be lengthy and time consuming since you can only be with one person at a time. This can lead to…
• Smaller sample sizes. This means that there can be a lack of statistical significance to your data, as well as smaller confidence intervals when looking at your data
• Moderated testing can be costly since you typically have to compensate your participants more for a moderated session and/or pay an agency to do the recruiting for you
• It’s possible to introduce bias through a moderator if they aren’t properly trained in correct research techniques, therefore altering the actionability and rigor of your results

And what about UX metrics?

Luckily with both unmoderated and moderated studies, the UX metrics you can measure can be behavioural (what they do – task success, task time, abandonment rate) and/or attitudinal (what they say – usability, credibility, appearance).

So you can run a basic usability test, collecting behavioural and attitudinal data with a moderator, or it can all be recorded automatically along with the user’s spoken-out-loud thoughts and feelings by screen recording software.

Again, this is methodology dependent, as not every testing method can be run both moderated and unmoderated.

Which is better: moderated or unmoderated?

Basically if you want the ability to probe and ask in-the-moment questions, and speed and statistical significance aren’t an issue, then moderated could be the choice for you.

However if you want to scale your user research and speed up the time to insights, then unmoderated is a sure fire way of achieving this.

No really, which is better?

Hand on heart, we would genuinely recommend a blend of both methodologies, and it really is determined by context.

If you have a real sticking point in your customer journey, and you need to drill down to where that is and why users are getting stuck, then running a usability test where you can observe and interview a smaller group of relevant users may be the most effective.

However to validate these proposed changes or improvements, you could then send out a survey to hundreds of users to back up the insight with statistically significant quantitative data.