How to avoid bias in usability testing

Before we get into the whole ‘bias’ thing, let’s just try and figure out the difference between fact, opinion and bias. It might sound like a simple exercise, but a cursory search will uncover huge arguments raging in all corners of the internet.

Well, the corners of the internet where dictionary definitions are hotly debated, and where most people only visit when they want to figure out the difference between affect and effect.

Here we go…

• Fact: Keane are a four-piece band from East Sussex, who have sold 10 million records worldwide.
• Opinion: Keane’s popularity is undeserved because I think their music is pallid at best, irritating at worst.
• Bias: I don’t like Keane because in general I don’t like the same music my dad likes.

and just for good measure…

• Prejudice: I don’t like Keane because they come from East Sussex.

Is that helpful? Possibly not, but it was worth spending the last 15 minutes coming up with some cheap shots at the expense of Keane.

Let’s just say for the sake of argument that in this article, when it comes to user testing, bias in all its opinionated or prejudiced forms will be discussed and be thoroughly interchangeable. Don’t blame me, blame the quarrelling indecisives at English Language & Usage.*

Bias on the blog

This is a blog – therefore you would forgive a large proportion of personal opinion. But this is also a blog about UX and research methods including usability testing, a field in which personal bias can be detrimental to gaining true insight.

That’s the contradiction at the heart of the UserZoom blog – we riff on the subject of UX, testing and usability, encouraging real-world insight from analysing user behaviour, while also blathering on about our own thoughts around the subject. But we do try to keep everything user-focused. So whatever our authors’ own opinions, it’s the behaviour of the user that should be the primary focus when deciding on improvements for your site.

Biased opinions are really hard to avoid though. If you run a remote UX test with our platform, we encourage you to observe the interactions and listen for where testers are articulating any problems they have with your site. Not necessarily whether they think the colour you chose for your background is nice or horrible, or whether they think running a costume store for hamsters is stupid. That’s just their own bias.

You keep those hamster millinery dreams alive, my friend.

Where does bias occur?

When testing, bias can occur for various individuals at different stages, manifesting in endlessly surprising ways.

Bias about the way someone *thinks* something should be done

Call it gut instinct, call it intuition, call it an inkling – either way, your own thoughts about the way something should be done are not necessarily the way other people think it should be done.

Bias from experience

If most of your favourite websites have the navigation on the left-hand side, that’s how you’ll prefer to see it on any new site you visit, and feel like it’s wrong if it’s on the right-hand side. This doesn’t mean you’re right and the other is wrong – it just means you have a different experience from someone else.

Bias from best practice guides, third party experts, or expert review

One website’s UX case study on how they got 60,000 conversions purely by making a button green is terrific for them, but it doesn’t mean your website will experience the same results. It’s a different product, with a different audience.

Similarly, if something is established as ‘best practice’ in one industry, it doesn’t mean it translates to another, and best practice has a habit of becoming defunct rather quickly.

Expert reviews, where a panel of UX experts gather to assess your product, are also prone to their own whims, experience and bias. Especially if you ordered in the *wrong* pizza from the cheaper place.

Your own mental model of the way something should work

A mental model is a tangible description of the way someone thinks something should work in the real world. Here’s a diagram of a mental model from Boxes and Arrows representing how someone might get up in the morning…

Each of these behaviours within the model are individual to that person, and won’t necessarily translate identically to someone else.

It’s worth bearing in mind if designing a product and plan on upending an established means of operation (i.e. making the ‘home’ button on an iPhone the self-destruct switch, when everyone knows you can only do that by saying, “Hey Siri, Alexa is better than you”).

Personal dislike

You might just take a personal dislike to the product, the service, the brand, your client or even the person testing. You know, like when you get a feeling about someone. You can’t put your finger on it, but you just know that if things were different and you were stranded on an island together, you’d have to be the one in charge of rations because they just look like they’d screw you over and eat everything in one go. But you can’t stay awake all night, can you? These are the kind of thoughts that often run through my head when listening to instructions given to me by any authority figure.

There are also certain brands and products out there that I wouldn’t go near, so put me in a room with them to interact with their platform and I wouldn’t be able to give them an even-handed assessment. You’ll probably have your own personal bête noir. And if you’d like to take my place when Keane ask me to UX test their next album, that would be cool.

Why is bias an issue in UX testing?

The purpose of user testing is to observe real-life interactions between a human being and your product, be it physical or digital. But this won’t be a pure observation if the tester has preconceived notions of how things should be, either through their own bias or the bias you’ve projected onto them. Bias can cloud the results of your test.

This also means that the user – i.e. your audience and your customer – isn’t the central focus of the experience. If you’ve bought in to UX testing and believe it’s the right path to improving your experience, then imposing the bias of yourself, your stakeholders or the HiPPO in the room will sway the results in a terribly non-user-centred manner.

Gut instinct is fine to a small degree, as is expert opinion and experience, however none of this can be backed up without honest to goodness user research and testing, because without this, your product could turn into a big expensive waste of time.

How do you avoid bias?

Unmoderated user testing

Unmoderated testing takes away the influence of others and removes outside distractions. People using your product are in their own natural environment, on their own with their own personal device, and best of all – comfortable. Certainly more comfortable than being sat in a lab, across from a ‘company representative’, being scrutinised on every decision you make.

This also means that people are less likely to tell you what you want to hear if they can’t see your face. People are more comfortable being critical from behind the privacy of their own computer screen (see: the opening 150 words of this article).

AND it also means that any stakeholders, product managers, investors or anyone else with an opinion on how the thing should work are kept well away from the test itself.

Screening questions

Screener questions can sometimes help filter out people with a certain opinion or political persuasion if you feel there may be a conflict of interest. But bear in mind there’s no strict guarantee that they won’t lie to get through anyway.

You should also remember that their observations are as valid as anyone else’s. I may not agree with the way Amazon goes about its business, but I could never deny the quality of its user experience.

Avoid leading questions

As Becca Kennedy writes in her post on how to avoid leading questions in UX testing, “Your job as a UX researcher is to uncover truth and honesty. Your job is to gather user feedback that isn’t coloured by your own hopes or expectations. Your job is to listen.”

So avoid questions like, “Would you rather use the old version or an improved version of the website?” or “Do you find this feature frustrating to use?” – basically avoid anything direct that can easily be answered with a one word answer.

Keep your own bias in check

No message could be any clearer, if you want to make the world a better place, then take a look at yourself and make a…

*key modulation*

… change!

Or to put it less plagiaristically, if you want your tests to be as useful as possible, keep your own language (body and speech) neutral as possible. Be professional, but not scary robot. Be open and receptive, but don’t stand there smiling and nodding while the test is progressing.

Also, even if it’s what you truly think, don’t say the test product is a “pile of crap.” That’s a real no-no.

*Please don’t tell them I pluralised indecisive. Or that I used the word ‘plagiaristically’.