A Bit About Our Experience…
At UserZoom we have spent over nine years helping brand companies and their expert UX teams manage and conduct Online Usability Testing, a.k.a. ‘remote unmoderated task-based usability testing‘. We are considered a pioneer for this unique remote survey solution and have used and applied its method to our clients’ research long before it became popular and widely accepted in the UX community. Today, we can confidently say that we are the leading company in our unique field. To that end, we are very positive about both what we’ve accomplished in the past and what’s to come in the future.
Over the past few years, many customers have asked us to share more information and insight into how unmoderated online usability testing works. We answered this request with creating a more in depth How-To article on Online Usability Testing. To those of you who already are experts in what we’re about to discuss, this article may not teach you much that you don’t already know, but to those of you who are new to this type of research, please read on as it will give you a new perspective on where the User Experience is headed next.
Feel free to leave us feedback in the comments section about anything you would like more information or follow-up on!
WHAT is Online Usability Testing?
One of the most common types of online usability studies is a ‘task-based online study’. In an online, task-based study, a large sample of participants (typically 100 to 200) are asked to complete navigational tasks on a website or prototype such as looking for information, registering, making a purchase or reservation, etc. While lab-based think-aloud (qualitative) testing can be run effectively with 5 participants, online task-based studies are best used for quantifying aspects of the user experience and require a higher sample size to generate reliable metrics.
Task-based online studies focus on two main aspects of user experience: Performance and Satisfaction. Performance includes effectiveness, (are participants able to successfully complete the task) efficiency, (time on task, number of clicks, etc) and behavioral data (such as click streams and click heatmaps). Satisfaction answers how the participants feel about the experience and is typically captured through survey questions. (1)
As in other online surveys, users participate in their natural context (home or work) using their own computer. Unlike in person usability testing, no real time human moderation is needed. In this way, the interaction and real behavior of the users is analyzed while the participants carry out the proposed tasks.
See Online Usability Testing in action in the video below:
WHY is Online Usability Testing so Valuable?
Online Usability Testing can be extremely valuable in a variety of situations. Some of the most common use cases include:
- Completing a transaction
- Comparing products
- Evaluating frequent use of the same product
- Evaluating navigation and/or information architecture
- Creating an overall positive user experience
- Increasing awareness
- Discovering a problem
- Maximizing usability for a critical product
- Evaluating the impact of subtle changes
- Comparing alternative designs (1)
Some benefits derived from the methodology and technology include:
- Data that can be used for significance testing.
- Geographically disperse samples, locally or internationally.
- Reduced time to perform the studies, greater quickness in results and analysis.
- Ability to segment results.
- Greater cost efficiency (over lab testing).
- Respect of the habitual navigation environment of the real user (in their home, at work, etc.).
When is Online Usability Testing Useful?
So when would you use Unmoderated Online Usability Testing? Here a list with a few ideas:
- To quantify your usability research: Consider you have a large customer base. This customer base includes different personalities, different usage patterns and different perspectives. Quantifying your usability is the only way you can ensure that you are reaching a true representation of your diverse population. Not only do you gain valuable data that solidifies your true population but also you can validate your lab findings or alternatively target which critical tasks you need to be probing in a lab-based study.
- To test users in their natural context: My computer and environment is different from my friend’s computer and environment and most likely different than a good portion of the population. Testing participants in their natural context accounts for different systems, configurations, and setups. The data you gain not only accounts for a mix of these various environments and setups but also encourages participants to act as they normally would, as they are not being “observed.”
- To understand users’ behavior: You want to understand why users are coming to your site and what they do once they come there. URUT uses a combination of web analytics (where users go) and surveys (the why) to create a complete picture and provide valuable data in providing the best user experience for your site.
- To validate or define your lab-based research: You want to ensure that the research you are currently conducting is valid and a true representation. With URUT not only do you gain valuable data that quantitatively solidifies your current research but also alternatively you can use URUT to target key critical issues and tasks to bring in the lab for further probing.
- To test internationally without traveling: International research is very expensive and at times put aside due to the cost and time commitment. URUT allows you the flexibility to conduct a study in many internationally locations without taking a step out of your home. Not only does it removes the expense of travel but also removes the need for all your data collected to be translated in order to analyze it. URUT removes the barriers that have traditionally impeded this very critical research.
- To conduct benchmark studies: URUT allows researchers to obtain statistically significant usability metrics on how a website performs vs. other versions of the site or vs. competing sites. Therefore, it’s the optimal way to take usability testing a step further to actually measure user experience and compare results either across time or through ‘industry benchmarking’.
How to Implement an Online Usability Test
Prior to starting your online usability test, you’ll need to do some planning. At a high level, you’ll need to know: who you want to recruit to take your study, which tasks to evaluate, and what type of follow up questions to include (what metrics are most important).
- Recruiting: Who you will want to recruit for your study will vary depending on your goals. For a simple evaluation of an existing website’s usability, you’ll want to recruit a sample that is representative of the real users. By testing with a representative sample you’ll have more reliable results and be able to generate better predictions that can be extrapolated to your entire user population. For an online usability study, you can recruit participants from your own database (website members or registered users), from a professional panel vendor (like Survey Sampling International) or by intercepting and inviting visitors that come to your site.
- Task Selection: Again, the tasks you’ll be evaluating will depend on your study goals. For a simple evaluation of an existing website the focus will be on “core tasks”. A core task is a task that is a required or commonly performed task on your website. If you are not sure what the most commonly performed tasks on your website are, consider running a true-intent study. (*Many firms, by the way, are now using task success as a strategic metric when measuring overall online service quality and performance.)
- Formulate follow up questions: The survey elements of a study will vary depending on the metrics of importance. Commonly used follow up questions include the SEQ (single-ease question) which is a likert scale question that asks participants to report how easy they felt completing a task was. You also have the opportunity to gather data on a more holistic level in the final questionnaire. Asking a net promoter score (NPS) will give you a metric on the likelihood a participant would recommend your website.
Online Usability Test in 7 Simple Steps:
An Online Usability Test will generally have the following flow:
1. Welcome Page
Start with a welcome page that gives your participants a general idea of what to expect from your study. (i.e. How long it may take, expectations, etc.)
Quick Tip: We recommend recruiting between 100-200 participants for a typical task-based study. If you are planning to segment your results, recruit at least 50 per segment. While lab-based think-aloud testing (qualitative) testing can run effectively with 5 participants, online task-based studies are best used for quantifying aspects of the user experience and require a higher sample size to generate reliable metrics.
2. Initial Questionnaire
Usually, before giving the task, a few introductory questions may be asked to help gather demographic or task-related information from your participants.
Quick Tip: These questions can be used later to help filter results based on certain criteria (ie- gender, qualifications, etc)
3. Creating Your Tasks
You should instruct your participants to carry out navigational tasks on a website or prototype to complete a goal/task you would like insight on. For example, look for information, register, make a purchase or reservation, etc.
Quick Tip: Your task description should clearly communicate to the participants what they’re supposed to accomplish. Try to avoid any site-specific terminology in your task description that could lead them to the correct answer.
4. Setting Up Validation Criteria:
When designing your task, you’ll want to setup validation criteria that will allow you to determine who successfully completed the exercise in your results. (Seen below)
Validation methods include:
- Validation by question
- Validation by URL
- Self-Reported Validation
Quick Tip: Below, you can see the UserZoom Task Bar. Depending on your Online Usability Testing vendor, a task bar like this will be present while the participant completes the task to remind them of what they are being asked to do.
5. Capturing Data:
The user performs the task (as you can see below). The UserZoom system automatically captures effectiveness, efficiency, and other behavioral data
6. Follow-Up Questionnaire:
After performing the task (validated via question or URL), your participants should be directed to a follow-up questionnaire (as seen below) depending on task success or task error/abandon.
- Repeat steps 3 – 5 for each task you have. A final questionnaire is used to gather feedback on the overall experience
7. Analyzing Results
The analysis of an online task-based usability study is focused around two areas: performance and satisfaction. When evaluating performance, you are evaluating if participants were able to successfully complete the tasks (effectiveness) and the effort required (time, number of clicks, errors etc.). If the users in your study struggle to complete the tasks, then it is likely real users who use your system will have issues too.
Satisfaction gives insight into how participants feel (gathered through your follow up questions). While you might see that a task was completed by a high number the participants, the experience may or may not of been satisfying.(1)
2 Quick Tips: Don’t design a study that is greater than 30 minutes long because that could yield high drop off/participant fatigue AND always remember to check and clean your data. Metrics like mean times are susceptible to skew from outliers values.
Check out some of the results you can get with UserZoom’s Online Usability Testing Solution here.
- How-To: Click Testing & Timeout Testing
- How-To: Card Sorting & Tree Testing
- How-To: Voice of Customer + Analytics
