Knock-knock, who’s there?

As part of our UX 101 education series, where we discuss the different types of studies and research methodologies you can use with the UserZoom platform, we’d like to introduce our readers to the exciting world of true intent studies!

What is a true intent study?

A true intent study is a type of survey in which you intercept a live visitor and ask them questions once they are done browsing your site or app. True intent studies are mainly used to better understand who is visiting your app or site and if they are able to accomplish what they came to do.

True intent studies are also helpful to consistently capture such usability metrics as Satisfaction, NPS, SUS, etc.

Intercept Survey Example

What are typical use cases for true intent testing?

True intent studies are typically conducted during the optimization phase of a site or app, although they are sometimes done during the discovery phase as well. The most frequent use cases that we run into over the hundreds of studies we conduct for customers every year are:

  • Better understanding why users are coming to the site
  • Seeing if visitors are able to do key tasks
  • Wanting to know what needs to be improved as well as how to prioritize them

True intent studies are like a researchers version of a canary in the mines. You want to have some clue of an impending disaster before it happens, yes? By conducting true intent studies you can test the waters and see if your live users are having any issues, as well as learn why they’re coming to your site or app in the first place.

How do true intent studies work?

A true intent study typically has three parts:

First, live visitors are intercepted on your app or site and asked if they would participate in your study. If they agree to participate in the study, they are taken to an initial questionnaire where you ask them what their main reasons were for visiting your site or app that day.

Second, after they answer the “why are you here today” question they will get some instructions telling them to minimize the survey window, go back to the website, do what they came to do, and come back once they’ve finished the visit.

Lastly, once they are done with the visit they will come back to the survey. Here is where we have open-ended questions to understand if they accomplished what they came to do, and if they couldn’t or had any kind of difficulty what those were. You can also add rating scale questions to capture metrics such as NPS, satisfaction, SUS, etc.

Once you start uncovering where problems are coming from by looking at the open-ended comments, you can conduct follow-up usability tests to dig into the origin of these problems.

Practical advice for running a true intent study

  • A JS code needs to be placed on the page where you want to intercept the visitors.
  • This code could be re-used for future studies, so if you want to run true intents periodically it’s enough with placing the code once.
  • The ideal number of responses we would like to get is 100 per main journey, usually adds up to about 400.
  • The survey should have in total a maximum of 10 questions since participants are typically doing it without being incentivized.

When should you use true intent studies?

Here are essential facts to consider while deciding on whether or not a true intent study is the right approach for your research goals.

Pros:

  • If you have a good amount of traffic you can get a lot of results very quickly with a relatively low cost per participant, if not outright free.
  • You can periodically reuse your true intent studies to easily capture the status of your website/app both in the moment and longitudinally over time.
  • Not sure where to start with your research? True intent studies are a good method to get an understanding of which problems your users are facing.

Cons:

  • Participants self-validate if they have been successful or not.
  • If you ask open-ended questions (Which difficulties did you have, for example) there can be quite a lot of open data to go through.
  • If you want to intercept participants in pages with low traffic it could take a long time to get the data you’re after.

What results do you get?

This is largely dependent upon the types of questions you are asking. Most often, however, we see:

  • Percentages. For example, X% of people came to do XYZ; Success Rate, X% of people were able to do the thing they came to do.
  • Rating scales with the different metrics you gathered: NPS, satisfaction, ease of use, etc.
  • Open-ended responses. “I couldn’t do Y because…”

Tips for analyzing your results

You’re going to have a jumble of data in the beginning and that’s okay! To make the most sense of your data we recommend filtering for the reason they decided to visit and looking at the success rate for that task individually. E.g. if multiple people said they came to sign up, take a look at the success rates for that.

Also, when looking at participants reasons for visiting, start looking at the problems/difficulties that the visitors are reporting and group them in themes. Is the problem in the search? Did they get stuck in the booking process? Are they reporting technical issues?

Keep in mind that if you’re a UserZoom customer you can integrate your true intent study with your analytics tool and cross the data you are getting from UserZoom with your analytics data. For example, what was the intent of people who are dropping in this page? What is the NPS score of the users who completed the purchase vs. the one who dropped off in the process?

This concludes our introduction to true intent studies. Thank you for reading and don’t forget to check out the rest of our UX 101 education series to help you on your way!