A short guide for setting up your own user test.

An online user test will generally have the following flow:

1. Welcome Page

Start with a welcome page that gives your participants a general idea of what to expect from your study. (i.e. How long it may take, expectations, etc.)

Quick Tip:
We recommend recruiting between 100-200 participants for a typical task-based study. If you are planning to segment your results, recruit at least 50 per segment. While lab-based think-aloud testing (qualitative) testing can run effectively with 5 participants, online task-based studies are best used for quantifying aspects of the user experience and require a higher sample size to generate reliable metrics.

2. Initial Questionnaire

Usually, before giving the task, a few introductory questions may be asked to help gather demographic or task-related information from your participants.

Quick Tip:
These questions can be used later to help filter results based on certain criteria (ie- gender, qualifications, etc)

3. Creating Your Tasks

You should instruct your participants to carry out navigational tasks on a website or prototype to complete a goal/task you would like insight on. For example, look for information, register, make a purchase or reservation, etc.

Quick Tip:
Your task description should clearly communicate to the participants what they’re supposed to accomplish. Try to avoid any site-specific terminology in your task description that could lead them to the correct answer.

4. Setting Up Validation Criteria:

When designing your task, you’ll want to setup validation criteria that will allow you to determine who successfully completed the exercise in your results. (Seen below)

Validation methods include:

  • Validation by question
  • Validation by URL
  • Self-Reported Validation

Quick Tip:
Below, you can see the UserZoom Task Bar. Depending on your Online User Test vendor, a task bar like this will be present while the participant completes the task to remind them of what they are being asked to do.

5. Capturing Data:

The user performs the task (as you can see below). The UserZoom system automatically captures effectiveness, efficiency, and other behavioural data

6. Follow-Up Questionnaire:

After performing the task (validated via question or URL), your participants should be directed to a follow-up questionnaire (as seen below) depending on task success or task error/abandon.

  • Repeat steps 3 – 5 for each task you have. A final questionnaire is used to gather feedback on the overall experience


7. Analyzing Results

The analysis of an online task-based usability study is focused around two areas: performance and satisfaction. When evaluating performance, you are evaluating if participants were able to successfully complete the tasks (effectiveness) and the effort required (time, number of clicks, errors etc.). If the users in your study struggle to complete the tasks, then it is likely real users who use your system will have issues too.

Satisfaction gives insight into how participants feel (gathered through your follow up questions). While you might see that a task was completed by a high number the participants, the experience may or may not of been satisfying.(1)

2 Quick Tips: Don’t design a study that is greater than 30 minutes long because that could yield high drop off/participant fatigue AND always remember to check and clean your data. Metrics like mean times are susceptible to skew from outliers values.

1 Tullis, Tom, and Bill Albert. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Amsterdam: Elsevier/Morgan Kaufmann, 2008. Print.