So you’re about to run a usability test on some function of your website, app or digital product. Great! But how can you guarantee that you’re inviting the *right* people to participate in your study?
It doesn’t matter what test you’re running, whether it’s a card sort, an online survey, or a remote usability test. Nor does it matter if it’s moderated (the researcher is present) or unmoderated (the participant flies solo) – if the people you recruit for the test aren’t the sort of people who will use your product in the first place, then their insight can be a pointless waste of time at best, or at worst – steer your product in completely the wrong direction.
That’s not to say that you should only recruit your actual customers for a study – after all, you’ll want to grow the number of new customers using your product, however there are people you’ll want to filter in or out, depending on what you’re trying to achieve.
Ready to begin? Let’s start with the basics…
With huge thanks to Chris Spooner (our Resourcing Project Manager), Clare Burroughs (our Customer Marketing Manager), Becca Kennedy (Human Factors Psychologist and author of excellent UX articles) and Andrea Peer (our VP Product Strategy of Research) for their invaluable input.
Before running any kind of user testing platform, you’ll want to find people to participate. People who will carry out the tasks you sent them.
A screener (or screener question) is an opportunity for you to have bit more control over who carries out your test before they begin.
This will also help you filter out anybody who wouldn’t necessarily be right for it.
For instance, if you’re testing the navigation of a website that has a sole audience of civil engineers, and you want to make sure all the relevant categories are represented and sit in the right menu, there’s little point in recruiting a load of people who literally just this second had to Google the term ‘civil engineers’.
Therefore you’ll want to find out what level of knowledge a participant has around civil engineering terms.
You’ll likely have two types of question available:
Be careful when selecting the type of screener question you want to use and think about how people might want to answer it. If the participant is likely to give more than one answer, then give them that option.
Here’s an example by Clare Burroughs of a screener question for an online car dealership:
Which of the following statements apply to you?
This is created to not lead the user towards one answer over another.
The ‘Next’ or ‘Screenout’ messages are for internal use only and show whether a participant would be selected to take the test or not. Next means they will proceed, Screenout means they won’t be suitable for this particular study.
An example of a Screener question from UserZoom
Try to ask ‘open questions’, as yes/no questions are commonly seen to be too leading. Chris Spooner, says, “The risk of this is that participants may answer falsely by what they assume to be the required answer, in order to qualify for the study. So, for example, if you are looking to recruit ‘Ford Drivers’, then rather than asking – ‘Are you a Ford driver?’ it would be more effective to ask which brand of car they drive and provide a list of car brands to choose from.“
Before setting these questions, you’ll want to actually ascertain who exactly your ideal test participant is. And if your team or organization has already defined your business’s key personas… awesome, the hard work has already been done!
Personas are a way to help organisations understand their potential and existing audience in a more personal way. In essence, personas are detailed profiles of a particular audience member, who represents a distinct group of people – in that they share similar behaviour, attitudes, personalities and preferences of your product, but are the ‘figurehead’ for a larger demographic.
You could therefore tailor your questions to whittle down potential participants to the most valuable people.
However as Chris Spooner states, it may benefit you to cast your net wider:
“In my opinion it is important to know why you need certain participants for the study. If you believe you can still obtain useful data from a more general specification of users then it is advised to do so, as it will work out cheaper and speed up field time.
I have experienced a few cases where we are given detailed personas and we have to narrow it down to identify a common theme between them. This can be a tricky process, and whilst I recognise their importance in UX research; when considering screener questions, it is helpful to centre on common attributes such as ‘works in finance’ or ‘has children under 18’, based on what is most important for your research.
Demographics can be helpful, and they are simple to set up as questions or pre-target on. When deciding to use them or not it is worthwhile considering how important other criteria is, as they naturally limit the sample available to you.”
With thanks to Chris Spooner and Clare Burroughs for the following insight…
If there’s one thing you should definitely keep in mind, it’s this piece of advice from Dr. Andrea Peer, our VP Product Strategy and Research…
Getting the best fit for your research involves juggling between three critical variables: representativeness, time in field and cost.
If exact representation is critical for your research questions then it will take more time in the field and often cost more. Usually when running research that requires assessment of a user’s ability to make a decision on specific things, high representation is important.
If getting results quickly is most important, then relaxing the representation criteria and being willing to pay a little more will ensure the fastest possible turn arounds. Classic usability type of questions could fall into this category.
To learn more about UserZoom’s all-in-one UX research platform, please get in touch!