Eight mantras to remember when UX benchmark testing
We live and breathe UX benchmarking here at UserZoom – it’s an incredibly effective way to prove the value of user research, tie user experience to business objectives and engage stakeholders.
To celebrate this awesome tool, over the coming weeks we’ll be publishing a bumper bounty of content relating to all things competitive and longitudinal benchmarking.
Kicking things off is a webinar mini-series, presented by our resident benchmarking expert and Senior Director of UX Research, Dana Bishop – the first part of which covers proving the value and quantifying UX benchmarking. You can watch it for completely free right now!
The above video is a fascinating and wide-ranging discussion on getting up and running with benchmarking while also proving its value to stakeholders. Dana also highlights some important things to remember when running benchmarking, both before launch and during testing.
I’ve pulled out eight of the most vital things to keep in mind…
What cannot be measured cannot be managed
You might hope or assume you’re product is a best-in-class, A+ example of modern usability, but you’ll never truly know until you actually measure the experience with your users, and then use this as a benchmark for further rounds of testing and improvements.
Without doing this, you won’t be able to quantify how effective your user experience is, and as you’re probably aware, stakeholders love simple, easy-to-understand numbers. Benchmarking helps you achieve this.
There’s no single perfect moment in time to begin either, just start today! There’s a lot of power in understanding where you are right now – what you’re doing well at, and where users are struggling and why. But you need to create a baseline first, which you’ll need to measure against going forward.
How often should you benchmark? You’ll want to set up regular intervals, whether it’s annually, bi-annually, quarterly, or monthly. It really depends on your development cycle and what works for your organization.
You also don’t want to be too rigid about your cadence. If you roll out a major update or redesign that’s outside your cycle, you should re-run your benchmark and see the effect it’s had. One of the biggest benefits of benchmarking longitudinally is you can measure and compare how design iterations are impacting the user experience.
Understand the ‘why’
While you’re tracking changes and KPIs, it’s also important to understand the whys behind significant shifts of performance. So when your quantifiable measurements change, you’ll be able to understand what’s affecting them. We recommend adding a small set of survey type questions to help monitor those types of changes, so you can gather qualitative feedback.
Invest the time upfront
When you’re designing your benchmark study, it’s really critical to invest as much of your time as possible before launch. As this is something you’re designing once, and then running many times into the future, the success of your benchmarking depends on getting it exactly right from the beginning.
It’s tempting to include too much. Often we see people who want to know absolutely everything. But all your questions should tie back to helping you answer your top objectives. You need to know exactly what you’re gonna do with every piece of information and every data point that you collect. Don’t overwhelm users with too many questions and too many tasks, as this just muddies the water. You want to stay lean and focused.
Competitive UX benchmarking is a way to compare some metrics of your product with your direct (or sometimes indirect) competitors. And this a huge part of optimizing the user experience of your website.
Rather than just benchmarking your own progress, competitive benchmarking gives you an entire industry view, helps you identify other useful examples of UX and you can give you insight into whether a competitor’s new feature is worth replicating or ignoring.
This also creates compelling proof to your stakeholders and executives when you can say “This is how we’re performing directly against our competitors and this is how we can beat them!”
Your research is only as good as your participants
Make sure that when you’re creating your participant profile for benchmarking, that you’re targeting either existing customers (if that makes sense for the task you’re benchmarking) or if your intended audience is prospects, ensure that the participants have the right background or experience to meet your important demographic criteria.
For instance, if you’re trying to understand how well your site is set up for customer acquisition, your existing customers aren’t your best target market. They’re already sold on your product. Instead you want to be targeting prospects. You can do this by creating a screener survey that allows you to get the right people testing your product.
Consistency is king
The validity of the data derived from your measurements overtime depends on the consistency of your process. You need to use the same screener questions, the same sample size, the same criteria for your participants and the same proportions of demographics.
Keeping everything consistent allows you to measure longitudinal data in a reliable way.
For more in-depth guidance, watch the full on-demand webinar on proving the value and quantifying UX benchmarking.
Christopher is the Content Marketing Manager for EMEA, which basically means the skipper of the good ship ‘UserZoom blog’. So far his requests for changing its name to the ‘USS-erzoom Blog’ have been rightfully denied. In his spare time, Christopher is a filmmaker and the editor of wayward pop culture site Methods Unsound. He used to be the deputy editor of Econsultancy, editor of Search Engine Watch, staff writer for ClickZ and features editor of CMO.com.