Unmoderated research is a very powerful tool. You get speed, efficiency and quick insights that can accelerate your team and move your product forward.
However, one of the challenges of unmoderated research is that when relying on people taking studies from remote locations, you remove some of your own quality control.
Therefore, there are a few steps that need to be taken when conducting online user research to ensure your speed and efficiency is met with quality.
First, you need to work with a partner that manages the panel of testers with rigor. They’ll ensure bots aren’t included, that ‘professional’ testers are screened out, and they’ll continuously review the panel’s quality and engagement with different studies.
Once you have that partnership, ensure you make data cleaning part of the process.
While you work with a panel partner, some respondents may still not provide quality responses when answering questions. That is par for the course in online user research. Market research has been managing this balance for years.
In fact, the industry norm is to expect that 10-15% of participants from a study will need to be removed. Jeff Sauro, founder of MeasuringU and the SUPR-Q user experience measurement, says researchers should expect to remove around 10% of respondents. “In our experience, around 10% of responses (often ranging from 3% to 20%) ‘cheat’ in surveys and unmoderated UX studies, and need to be tossed out or ‘cleaned’.”
User Experience Research 101
So it’s important to make data cleaning part of your process for any unmoderated research you conduct. If you’re conducting a think-out-loud study with 10 participants, you’ll likely need to remove one person due to them not talking or poor audio quality. If you run a survey with 100 participants, you’ll likely need to remove about 10 participants due to nonsensical open-ended commentary.
In the market research industry, it is actually a best practice to include an open-end question in surveys, to be able to easily filter out low-quality respondents. Again, it’s all part of the process.
So when conducting a research plan, try to follow the steps below to ensure your final results are actionable, based on quality data:
One thing to keep in mind when removing respondents is asking yourself, “Is their answer really low quality, or does it make sense in the scenario?” For example, if they respond, “I don’t know” to a question, that answer may make sense and be sufficient. While you may not like the response, if you remove them you may bias the data. Also, try to look at their answers holistically, across different questions.
In a study I ran recently, someone said “The instructions are not clear,” in one open-end response, in another they provided a long, thoughtful answer, and in another they said, “I don’t know.” Therefore, looking at their responses together I decided to keep this participant in, as they were likely answering honestly for each question.
In another study I reviewed, there was a follow-up question to a click task asking, “Why did you click in that area?” I had to ask myself, how would I answer that question? I would probably say, “Because it makes sense” or, “It’s similar to other sites I’ve seen before”. So part of getting thoughtful open-end responses, is to write good and specific questions. Encourage participants to provide detail, and type one-two sentences.
So when you run your next unmoderated project, be sure to embed data cleaning into your process. This step will not only ensure you have high-quality data, but will also save you time in the long run and ensure you’re making decisions with confidence.