Knowing how to prepare your test script before conducting an unmoderated usability test is essential for a successful study.
A framework helps you to structure all the information you need to include in your usability test.
Frameworks usually include a set of concepts, practices, and criteria that facilitate dealing with a common process or problem and serve as a point of reference for future projects.
In this post we will present a framework which helps you define your remote usability test; everything from creating the initial concept all the way to conducting your research. By the time you have your finished script you will be able to quickly set up and run your test in a testing tool.
Knowing how to prepare your test script before conducting an unmoderated remote task-based usability test is essential. A framework helps you to structure all the information you need to include in your usability test. Frameworks usually include a set of concepts, practices, and criteria that facilitate dealing with a common process or problem and serve as a point of reference for future projects.
1. Hypotheses and global questions: According to the objective of your test, you will first develop hypotheses and questions to ask participants.
2. Scenarios: To help users understand what you want them to test, you will need to provide them with context e.g. “Winter is approaching fast and you want to buy new boots.”
3. Task(s) for each scenario: Asking users to perform a task in-line with your scenario allows you to validate your hypothesis and/or provides you with further context.
4. Questionnaires and their logic, i.e. where users are sent depending on their answers. During a test, there are three moments where asking questions comes in handy:
5. Invitations and participants: All information available related to each participant. In particular, the user ID your testing tool assigns to each person, the invitation message and design and status of each participant e.g. cheaters, abandoned study, successfully completed, paused etc.
6. Data, results, and analysis: Includes all study results. The UX metrics you obtain depends on the testing tool you use. You should at least gather: qualitative data (user feedback via video & audio, browser videos including screen interaction) and quantitative data (success rates for each task, number of clicks needed to perform a task, time on task, clickstreams, satisfaction rates, difficulty ratings and other issues reported by users).
Remote Moderated 101
What global questions do you want users to answer? What hypotheses do you want to validate?
Defining only one or two test objectives will help you focus on the results you want to obtain. Your objectives should aim at answering one important strategic question. Your hypothesis and global questions will allow you to derive scenarios, tasks and questionnaires you will need in order to gather the feedback you want from your participants.
Questions that can help you define your objectives are: What first impression do users have? Do they understand your product and services? Does your business cover user / customer needs? Is your website easy to use? How do users evaluate your new design? How do users rate you compared to your competition?
Answers to these questions may turn into your test objectives:
If users perform a test in real-world context you will be able to see how users behave under real-world conditions, such as how they behave while browsing on their smartphone. Therefore, it is important that your testing tool allows you to conduct tests on different device types. The closest you can get to realistic usage is by inviting real users to conduct a study either from your database or by means of a panel provider. If your participants are also potential customers they will know how to best describe their needs and problems.
By working with real user context you also obtain more actionable data and get valuable information on the time of use. This is not possible if you conduct a test in a usability lab since the context can sometimes distort the data.
The total number of participants depends on your website or app and the type of information you want to obtain. If the approach is purely qualitative a small sample might suffice but if you are after quantitative data, your goal will be to find out how many of your users are affected by a certain UX issue.
After having defined the objective, it is time to focus on potential questions. They can be classified according to whether they are general (to collect psycho- demographic information such as: gender, age, professional profile, family members, habits, etc.) or whether they require a context. For example, you will want to include questions users can only answer by interacting with your application and by performing a task. Tasks enable users to experience what you are asking about.
For example, imagine you want to test the purchasing process of your site. The winter collection has arrived and you’ve made changes to the check-out procedure, especially when users shop for winter shoes. You now want to invite users to test the new check-out. With this context, possible questions could be:
After having formulated research questions, you need to think about how to put them into context. This is why you need scenarios. Participants need to feel like they’re in a real context of use. Through scenarios users are able to understand why you ask them a certain question.
These are some example scenarios for our fashion website:
In order for users to answer your research questions they must interact with your app, mobile site, website or prototype first. The way you ask users to perform a task is important because it will influence how users behave while doing the test. You should link scenarios and use a consistent “story” to create the right testing context.
Make sure you establish a way for validating task success e.g. by requesting users to find a price, a certain page or specific information:
Now that you have created the context (scenarios and tasks), it is time to define the questionnaires in order to collect not only results, but to answer your hypotheses and strategic questions. Usually these questions are about opinions and perceptions and help gather in-depth information on the user experience.
During a study, three different stages for asking questions usually exist:
In the following graphic we provide an overview of all framework parts to help you when reviewing your final script:
When you are done planning your study along this framework, it is time to enter everything in your UX research software. At this point it is important to highlight that the more your tool meets the presented framework the easier it will be to set your project up.
In other words, your remote testing tool should let you enter different types of questionnaires, scenarios, and tasks and should allow you to define when certain information or questions are shown to participants. If your testing tool is set up the same way as this framework, you won’t need to spend extra time and effort on adapting your script to your research tool. This will accelerate your research process.