Whether you’re a grizzled UX expert who “has seen everything in the line of duty”, or you’re a complete newbie who has been told to “do some of that UX stuff that’s so popular these days” – this is the guide for you.

We’ll explain the difference between all the most common UX research methodologies, how they are either quantitative and/or qualitative (and what that actually means), and we’ll also discuss the benefits and challenges of running moderated and unmoderated UX studies.

Basically all the stuff you’ll hear about when entering the exciting and very detailed-orientated world of UX research and testing.

As for the more experienced UXers, think of this as a way to refresh your UX tool kit. Perhaps you’ve been using the same few methods for the last few projects and you need some inspiration. Perhaps you’ve been firmly planted in either the qual or quant camp, and you can’t see sense in the other. Perhaps the insights you’re revealing just aren’t moving the needle enough. Maybe something here will give you the nudge you need.

It’s unrealistic and probably unnecessary to use every single methodology on any given project, however a blend of methods across the ‘quant and qual divide’ can ensure you’re getting the maximum number of helpful UX insights possible.

Jump to a section:

What is user research?
Quantitative vs Qualitative
Unmoderated vs. Moderated
Guide to UX research methods
A/B Testing
Card Sorting
Clickstream Analysis
Click Testing
Concept Testing (or Prototype Testing)
Customer Feedback (or Voice of the Customer Study)
Desirability Study
Diary/Camera Study
Ethnographic Field Study
Eyetracking
Focus Groups
Information Architecture Testing
Intercept Surveys
Interviews
Lab Study
Participatory Design
Remote Moderated Usability Testing
Remote Unmoderated Usability Testing
Surveys (or Questionnaires)
Tree Testing
True-Intent Study
Usability Testing
UX Benchmarking Study

What is user research?

The whole point of UX research is to understand who your customers are, what is important to them, how they actually use your products, what their biggest impediments are and how to fix them. And the most useful UX research is well-planned and organized, which should give you the confidence that the insights are derived fairly and accurately.

The very first stage of planning, and the one that will help you decide which research methodologies to use, is to decide on your research question.

Why are you running UX research? What goal are you trying to achieve? What are are you trying to improve?

These questions can include:

  1. Why is our checkout abandonment so high?
  2. How do we improve the navigation of our website?
  3. What CTA yields the largest conversion?
  4. How can we increase app downloads?
  5. In which category should a product or piece of content sit?
  6. How do we compare to our competitors?
  7. How are design iterations moving the UX needle?

Some of these questions can be answered by running usability tests (#1), analysing data from an A/B test (#3), conducting card sorting or tree testing (#2 and #5), some can be answered by jointly using quant and qual methods (#6 can discovered by conducting competitive UX benchmark studies and #7 through longitudinal benchmarks on your site) or through a combination approach of observing and collecting behavioral and attitudinal feedback (#4.)

Quantitative vs Qualitative

To help you decide where your research question is best answered, here’s more about the two main categories of research methods to choose from:

Quantitative research

Quantitative refers to any kind of research where the results can be presented in numbers, i.e. ‘how many, how often or how much’. It is often used to add context (e.g. it took users less time to complete the task on prototype A than prototype B) and can be thought of as the “What” data.

For a complex example of quantitative data, just take a look inside the analytics of your website – pageviews, sessions, bounce-rate, frequency of visits over time are all quantitative data and show what happened on the site. Net Promoter Score, ease of use, completion rates, brand perception, time on task, mouse clicks and more are typical examples of quantitative data in UX.

Benefits of quantitative research

  • Quantitative data is relatively easy to gather, doesn’t necessarily take very long and you’ll find it easier to assemble a larger group of respondents.
  • Fast results.
  • Because of the numerical nature of the results it’s simpler to tie UX improvements back to your company’s KPIs, therefore justifying investment.
  • Quantitative research tends to be objective (i.e. it’s harder to ‘lead’ participants with more complex questions) and more ‘scientific’ i.e. less prone to human bias.
  • Statistical significance – because quantitative studies are run with a much larger volume of participants, they are more representative of the population as a whole.
  • It’s easy to make nice visualisations from the raw data.
  • People love statistics!

Qualitative research

Qualitative refers to any kind of research where the results take the form of observations, comments, thoughts and feelings and allow you to tell a story about the participant’s experience. If quantitative answers the “What” then qualitative is the answer to “Why.”

This can be further subdivided into behavioral and attitudinal methods. For instance, a behavioral method examines how people use your product. An attitudinal method would then investigate what people think of your product.

Benefits of qualitative research

  • You’ll be inside the heart and mind of the person using your product.
  • You should hopefully see things that quantitative data can’t reveal – for instance if there’s a page on your website losing traffic, you’ll be able to witness first hand what happens when a genuine user visits the page.
  • Participants may find it easier to reveal their feelings about something, rather than assigning a number or ticking a ‘yes or no’ box.
  • This anecdotal evidence can be more persuasive than hard data – especially if it’s emotionally driven and your stakeholders can see and hear these observations for themselves.

Quite simply, qualitative research can answer why or how to fix an issue, quantitative research is better at answering how many and how much.

Specific research techniques can belong to either the qualitative or quantitative category, depending on how they are used. You will likely find that a blend of quantitative and qualitative research techniques will help you cover all possible bases.

Unmoderated vs. Moderated

Another thing to consider when choosing from the variety of research and usability testing methods at your disposal, is whether you’re going to run the tests ‘moderated’ or ‘unmoderated’. So, whether yourself or a researcher (a moderator) is going to be present during the test, or whether the test participant is left to carry out the task without anybody else in the room.

Here’s a little more detail on both options:

Unmoderated testing

Unmoderated tests are how we describe unobserved tests, where a participant is left alone to complete tasks without the presence of a moderator. These sessions can be recorded for later viewing as part of a qualitative study, or the data is collected and analysed as part of quantitative research, or both.

In an unmoderated test, a participant will interact with a digital product while being prompted with questions or tasks. A UX researcher will then look at the results of the project and take note of where the participant is experiencing problems.

You will need a specific online platform or service to help you accomplish unmoderated tests. The service will also help you build the study, find participants and analyse the data. This can be a lot easier, cheaper and quicker than running moderated sessions.

However as you will be reviewing the results ‘after the fact’, you’ll be missing the chance to ask more in-depth questions. So if something unexpected crops up, you’ll have to launch another round of tests to investigate.

Moderated testing

In moderated testing, the participants are observed by a moderator, either in-person or remotely. The core reason for getting into a moderated session is so that you can be in a live setting with a participant.

This allows you to have a conversation with your users as you’re observing what they are doing to better understand their behavior and dig deep into usability issues and attitudes. Plus, you can modify your script on the fly to probe more if there are confusing areas and ask ad-hoc follow-up questions.

Speaking of ad-hoc questions – another advantage to conducting moderated sessions is that you can have stakeholders and teammates anonymously observing the sessions. This is a great way to include folks who are not usually involved in user research and usability testing to participate and send you questions to ask the participants.

Guide to UX research methods

Here are examples of some of the most common techniques used to carry out UX research. Many of the following methods can be run as either moderated or unmoderated and can be used to gather quantitative or qualitative feedback depending on how you set up your study, but we’ve given the most common approaches that we and our customers use them for.

A/B Testing

(Quantitative / Unmoderated)

A/B testing requires you to use a third-party piece of software that helps you set up two different web pages, where one page has an element that’s slightly altered from the other. For instance, if you can’t decide on the text for a ‘buy’ button, you could use an A/B test to present one version of the button that says ‘add to cart’ to half your traffic, and the other version that says ’buy now’ to the other half of your traffic.

Then you can then see whether or not changing the text of this button has made any difference to the number of people clicking on it.

Card Sorting

(Quantitative / Moderated or Unmoderated)

In a card sort, participants are presented with a list of items (for example, all the products featured in an online supermarket) and asked to group them in a way that makes the most logical sense to them. Depending on the type of card sort, participants can also choose names for the groups they’ve put together, forming the potential categories and subcategories of a website.

card sorting
Example from a UserZoom card sort test

There are three types of card sort:

  • Open card sort: Participants are asked to group cards into categories that makes sense to them, and then they label each category in a way that they feel accurately describes the content.
  • Closed card sort: Participants sort cards into category groups that you’ve already labelled and defined. For example, this is handy if you’re launching a new page for ‘watches’ and you’re not sure whether to put it under a parent category ‘Accessories’ or ‘Jewellery’.
  • Hybrid card sort: This is a mixture of open and closed. Participants can sort cards into categories you’ve already defined and then create their own categories if they think your categories are inaccurate.

While the popular image of conducting card sorting involves post-it notes stuck on whiteboards, it’s worth noting that collecting, collating and analysing all those sticky bits of paper can be quite a challenge. Online card sorting removes the sticky notes and all the tedium and manual efforts required to organize them.

Clickstream Analysis

(Quantitative / Unmoderated)

Clickstreams are a record of the aggregated paths (URLs) followed by participants during their navigation.

Clickstreams allow you to view and analyze the paths participants took while performing their tasks, which percentages followed that path, and their final status (complete, error, abandon or timeout). Certain software even allows you to analyze the heatmaps (the aggregated areas where users clicked on the page) on each page for further behavioral analysis.

Often clickstreams are used to help you evaluate sales funnel and calculate the conversion rate. The clickstream below is from a study done on a Norwegian Cruise Website and shows the dominant path users took to accomplish their task of booking a cruise.

Click Testing

(Quantitative or Qualitative / Moderated or Unmoderated)

Screenshot click tests are a quick and simple way to test static images to see where users would click. This can be used to validate site design, as well as test wireframe prototypes, by asking participants questions such as, “Where would you click to access specific content.” This can be done for everything from high-res images to scanned doodles on a bar napkin.

Concept Testing (or Prototype Testing)

(Quantitative or Qualitative / Moderated or Unmoderated)

Concepts or prototypes are inexpensive versions of a design for engineers to test how the real thing will function. For product team members, prototypes facilitate discussion and understanding. UXers can test prototypes with users to get early feedback before sinking resources into a bad design. In the long run, it will save you time, money, and headaches rather than doing all of your UX testing when a product is nearly complete. Or, worse, doing no UX testing at all.

To get the most out of your prototype testing, you should remind participants that the prototype is not the finished product, but more like a blueprint. Encourage them to look past any unpolished aspects that might be missing or incomplete.

Concept testing can be done one-on-one or with larger numbers of participants, and either in person or online.

Customer Feedback (or Voice of the Customer Study)

(Quantitative or qualitative / Unmoderated)

A Customer Feedback or VoC study is aimed at collecting ‘true’ data on participants who visit a site. In general, Voice of the Customer studies are really just another way to round out the research you are already conducting by being ‘always on’ and gathering ongoing feedback for you in the background. This data can be used to segment visitors and create/flush out your user personas.

Conducting this type of study will help you gain true insights to data that you could not otherwise capture in an artificial environment or with artificial task scenarios:

  • You can obtain valuable statistics about who your users are and what they want from your website.
  • You can find out if visitors leave your site while having accomplished their goals.
  • You can analyze whether a visitor’s online experience was positive, neutral or negative.
  • You can get the Net Promoter Score and find out if visitors would recommend your site to others.

Users can be selected through a feedback link, button, form or email. They then visit the relevant website with a clear idea of what they intend to do. Visitors are invited to participate in a short survey as they arrive to the site’s homepage or any target page of your choice. Before users actually participate and fill out the survey, they are told to continue doing what they came to do as they normally would on the site. Users spend as little or as long as they wish on the site accomplishing their goal. Only when they are done, they can choose to start the questionnaire.

During the survey participants can be asked a series of qualitative questions aimed at better understanding their experience, such as:

  • What did you come to do?
  • Can you tell us a bit about yourself?
  • Did you accomplish your goal?
  • Were you happy with your experience?
  • How likely are you to recommend the site to others?
  • How can we make our site better?

That being said, these tend to be with very large sample sizes and as such can take a lot of bandwidth to go through them all.

Desirability Study

(Quantitative or Qualitative / Moderated or Unmoderated)

Desirability studies help you identify and define some quality of your product or brand. You will show your participants your product (whether it’s a prototype, live website or even some marketing copy or images), you then ask them to describe what they see using a list of pre-selected words. With this data you can see what percentage of your respondents consider your product to be “awesome” or your tone of voice to be “weird.”

Diary/Camera Study

(Qualitative / Unmoderated)

Diary studies gather information about a user experience over an extended period of time. Participants write about their experiences with a particular product or service in a diary. They may also take photos or perform other activities to record their experiences. Once the study period is over, the researcher analyses the findings.

Diary studies remove the influence of both the researcher and the unnatural out-of-home setting, but they’re also useful for understanding long-term behaviour.

According to NN/g, these long-term behaviours can include:

  • What time of day do users engage with a product?
  • How do they share content with others?
  • In what capacity do users engage with a product?
  • What are their primary tasks?
  • What are their workflows for completing longer-term tasks?
  • What motivates people to perform specific tasks?
  • How learnable is a system?
  • How loyal are customers over time?
  • How do they perceive a brand after engaging with the corresponding organization?
  • What is the typical customer journey and cross-channel user experience as customers interact with your organization using different devices and channels?
  • What is the cumulative effect of multiple service touchpoints?

Ethnographic Field Study

(Qualitative / Moderated)

Ethnographic studies involve talking with people and observing them perform their tasks in their own natural context. Its aim is not just to gather information on how people behave and interact, but also how their location, environment and other contexts affects their day-to-day lives. UX designers take this ethnographic research and use it to solve a problem through a product or technology.

Eyetracking

(Quantitative / Unmoderated)

Eyetracking lets UX researchers see precisely where participants look on a screen when performing tasks. It requires a special piece of equipment that tracks the user’s eyes as they look around the screen, and this information generates a heatmap of where on the page the user concentrates the most. This information can help you determine where your most valuable content or CTAs should reside.

eye tracking
Example of a heatmap generated by eyetracking software, courtesy of Usability.gov

Focus Groups

(Qualitative / Moderated)

A group of participants from your target market are gathered together in one room with a moderator, where they discuss topics around your potential product or service. These thoughts and feelings are collated and used to inform the direction of your product.

Information Architecture Testing

(Quantitative or Qualitative / Moderated or Unmoderated)

Information Architecture (IA) refers to the way content is presented and accessed from any given page on your website – whether through menus, breadcrumbs, categories, links – whatever takes you from one page to another.

Information architecture testing can help you define navigation, improve information taxonomy and maximize findability across your website. This is usually done through both card sorting and tree testing.

Intercept Surveys

(Quantitative or Qualitative / Unmoderated)

By adding a few lines of JavaScript code to your website or integrating your mobile app with an SDK, you are able to intercept site visitors on your site or app, with an automatically triggered survey. You can find our what your visitors are planning to do on your site, if they are able to achieve what they came for, where they are thinking of going after their visit, who they are in general and if they are satisfied by the experience you provide.

Interviews

(Qualitative / Moderated)

Participants meet with a researcher one-on-one to discuss topics around your potential product or service. These thoughts and feelings are collated and used to inform the direction of your product.

Lab Study

(Quantitative or Qualitative / Moderated)

In a traditional lab-based study, between 6-10 (varying according to needs and perspective) are brought into a ‘lab’ environment to run through a series of tasks. Participants work on a pre-configured computer or mobile device while being observed in a separate room either via monitor or through a one-way mirror. During the study, participants are given tasks and asked to perform them with a researcher sitting next to them or in the other room.

If using a think aloud protocol participants are asked to express their thoughts out loud and the researcher can feel free to probe or ask further questions while the participant is walking through their task and after. Alternatively, participants can walk through their task with no interruptions and questions can be left for probing after the task or after the study in order to gauge time on task.

Participatory Design

(Qualitative / Moderated)

As UXmag states, Participatory design brings customers into the heart of the design process. Also known as “co-creation”, “co-design”, or “cooperative design”, the end-users of a product, service, or experience take an active role in co-designing solutions for themselves.

Remote Moderated Usability Testing

(Quantitative or Qualitative / Moderated)

When carrying out remote moderated usability testing, also referred to as online moderated research, you are live online with participants but from your own location, connected to them with screen share technology and an audio bridge.

Other than that it’s the same premise as in-person or in-lab testing. The moderator is there to ask participants questions, respond to their questions and feedback, and guide them through the tasks.

The testing software will collect the quantitative and/or qualitative data as participants go through your test. It will also allow you to collect and triangulate different kinds of data and allows for the combination of different kinds of methodologies within a single study. This way you can collect as much data as possible with your participants and make the most out of their and your time.

Remote Unmoderated Usability Testing

(Quantitative or Qualitative / Unmoderated)

Also known as remote panel studies, remote unmoderated usability testing is a flexible and versatile methodology that caters to various research needs. It is also known for its economy with time and money in comparison to a lab-based study, hence its growth in popularity and increased adoption in recent years.

Remote UX testing will provide statistically valid quantitative data, as well as immediacy to usability issues with video, qualitative, and behavioral data.

This method requires having participants, chosen from a panel of testers who fit into your particular demographic, interact with your prototype, website, app or any other digital product in order to capture their feedback, actions, behaviors and/or spoken-aloud thoughts.

Surveys (or Questionnaires)

(Quantitative or qualitative / Unmoderated)

You can get feedback from users, and collect satisfaction metrics and ratings, with online surveys. Users can be intercepted directly from your website or app, using advanced survey capabilities such as logic, conditioning and branching, task randomization and advanced screeners to gather insights.

Typically, surveys are quantitative because you are asking rating scale type questions, and the goal is to get statistically significant sample sizes to ensure your decisions are backed by the numbers. You can, however, get qualitative insights from asking open-ended questions to get information that wouldn’t necessarily fit into a typical rating scale.

Tree Testing

(Quantitative / Unmoderated)

Tree testing is typically used to validate the results of a card sort. In tree testing, the main categories and subcategories for a website are already established.

Your participants are asked to explore these categories in order to find a particular item or piece of content. They click through the various links until they find the category where they expect the item to reside.

True-Intent Study

(Quantitative / Unmoderated)

True Intent studies enable you to intercept organic visitors on your website, mobile site or mobile app and ask them what their intentions are for visiting as well as getting actionable information about their experience.

True Intent is a cost-effective research approach for you to obtain a more holistic view of how well your site is doing by asking questions such as: Who is coming to your site? Why are they visiting? Did they get what they needed from the site? How are they using the site? How do they feel about it?

Usability Testing

(Quantitative and/or Qualitative / Unmoderated or moderated)

The bread and butter of user research. This method requires having participants interact with your prototype, website, app or any other digital product in order to capture their feedback, actions, behaviors and/or spoken-aloud thoughts.

Typically this is done on either their own personal devices if remote (see Remote Unmoderated Usability Testing above) or your devices if in-lab.

Observing or analysing this behaviour will give you immediate insight into what works and what doesn’t work on your site, allowing you to make improvements and focus on what to iterate next.

UX Benchmarking Study

(Quantitative and qualitative / Unmoderated or moderated)

UX benchmarks allow you to measure your digital products baseline performance and measure how changes are affecting the UX needle over time. Typical benchmark studies are either longitudinal, in which you continually measure your own products over time, or competitive in which you measure your products performance against others.

With benchmarking, you can send your participants off to carry out the same identical tasks on a variety of websites (including your own) in order to determine:

  • How your site performs relative to your competitors
  • Interpret your usability standing, feature set and more within your industry
  • Learn from their success and failures, what works or does not for your competitors
  • Industry best-in-class examples to emulate
  • Benchmark with other industries

In order to score the sites, you can combine various measurements, collecting both behavioural data (such as task success, task time, page views) and attitudinal data (such as ease of use, trust and appearance).

For more information on any of the above methods, please get in contact. We’re more than happy to chat.

Take our survey to win an iPad Pro or one of 10 Kindles!

We’re conducting a survey to help people just like you at companies just like yours understand the trends in enterprise UX and UX Research, and where they’re leading us.

We just need 10 minutes of your time to complete the survey and as well as entering you into our prize draw, we’ll also send you an early copy of the study findings as an extra thank you for helping us out.

Take the Survey

Main image by Hal Gatewood