Anyone who works in user research knows their work can uncover an array of user insights which can inform business and design decisions, and ultimately help create a better experience for users. However, it can be difficult to quantify anecdotal experience, especially if you’re mainly running moderated tests and getting qualitative insight (i.e. people telling you why your product is rubbish or great).
But how can you communicate this data effectively and efficiently to stakeholders? How do you tie an ‘improved user experience’ to improvements in your company’s overall key performance indicators (KPIs)? You worked hard for those insights, dammit, and you should be rewarded!
And even more importantly, how can you measure user experience improvements over time and therefore justify more testing to make your product even more awesome and user-friendly?
One way is through UX benchmarking!!! (You probably guessed this already, because you looked at the headline.)
Benchmarking creates a baseline for understanding the current user experience on your website, app or any digital product. And the sooner you get started, the sooner you’ll be able to measure and improve the user experience. You may also hear this referred to as longitudinal benchmarking – which is basically measuring over time.
When you start benchmarking you need to create a baseline. A starting point. Otherwise you’ll have nothing to measure against.
You won’t know how fast you can run 100 metres until you actually put on a pair of running shoes and persuade a confused friend to measure the distance and time for you. The same is true for your website, app or other digital product.
Just pick a feature or tool you want to measure and improve on your website or app – let’s say it’s the checkout section on your retail site. You then run some testing to get some baseline usability metrics.
We discuss methods of testing and metrics in the UX benchmarking ebook, but for this example let’s just say we ran a basic usability test where we timed how long it took a group of 20 test participants to find a specific item and purchase it through checkout. We’ll also look at average number of pages they visited, and whether the task was successful.
You’d then input these figures in a table and then you will have your baseline data to begin tracking this particular user experience over time.
Perhaps within this initial test you got some in-the-moment feedback from your participants where they mention feeling “overwhelmed by so many options during the checkout.” You could then make some changes to the user experience based on this research (for example, simplifying or removing
some options), then you run another round of testing and see how you perform against your baseline.
REMEMBER: keep all the details of the test study the same for each round of testing otherwise it won’t be an accurate benchmark. For instance – the item being purchased, the device/browser being used, the number of test participants, the type of task and any questions you might ask.
You then start to build a picture like this (which is essentially a scorecard, and you can learn all about UserZoom’s UX scorecard program here).
As you make changes and run each round of testing, you’ll begin to see problem areas for improvement. You’ll then see exactly how your improvements are affecting these metrics, and if you tie these metrics to wider company KPIs (improve revenue, improve ease of use, improve customer satisfaction) your user research can be seen as an invaluable tool in the organization.
Measure UX and prove the value of research
So you know you want to benchmark something, but what exactly can you benchmark?
Or a combination of the above – basically anywhere your users and customers are accessing your site or product online can and should be UX benchmarked.
This applies to external sites as well as internal sites or improving employee intranets, because you want to ensure your online properties are as streamlined and optimized for your internal users as they are for your customers. Just imagine if Amazon had the same UX as your employee expenses portal? *Shudder*
Really you can can begin any time from concept, to live working product, to every iteration that follows. But basically… start NOW!
As for how often?
We work with many different customers who have different intervals in which they run their benchmarking. Sometimes it’s annually, sometime’s biannually and sometimes quarterly or even monthly. It really depends on your development cycle and how often you’re rolling out design changes.
We would also recommend testing before or after launching a major update or redesign that may not fall within your regular intervals, as you don’t want to find that something isn’t working six months after launch, because that’s when your timetable says you’re due another round of tests.
Why would you undertake UX benchmarking in the first place? Here’s our executive-friendly guide to why UX benchmarking is so rad.
Longitudinal UX Benchmarking creates a baseline for understanding the current user experience on your website, app or any digital product. And the sooner you get started, the sooner you’ll be able to measure and improve the user experience.
However a huge part of optimizing the user experience of your website is knowing how it performs in comparison to your competitors. Sure you can measure and influence improvements and declines in your own performance, and that’s awesome, but you’re not going to gain a competitive advantage if you’re not measuring against your opponents.
Let’s investigate further…
Competitive UX benchmarking is a way to compare some metrics of your product with your direct (or sometimes indirect) competitors.
This is great because your boss may often say to you, “Ooooh I like this fancypants new feature that COMPETITOR X has launched, let’s copy it” and even though your own gut might be saying, “No bad idea, bad idea, wrong!!!” and “When’s lunch?” you don’t really have a concrete argument against copying the feature.
So you just steal it, because they’re the boss and you haven’t discovered the joys of competitive UX benchmarking yet and breaking for lunch feels increasingly like a foreign country.
But thankfully now you’re reading this article, you can say, “Good idea BOSS X, I like what you’re suggesting, but before we spend too much money, let’s build a prototype of the feature and UX benchmark it against our competitors, then we’ll see if our users actually like it or not?” That way you’ve either saved the company some money and proved yourself very wise, or you’ve backed up your boss’s idea with concrete data and proved them as very wise, and hopefully in both cases boosted the value of your user research efforts.
Comparing a prototype or concept with a competitor’s similar feature isn’t the only thing you can do. With competitive benchmarking you can test anything that’s live on your site and compare with any number of competitors or non-competitors (after all, you may find inspiration from outside your own industry).
You could compare the usability of your checkout against another, or you could compare softer metrics such as asking participants, “What is your perception of this company?” before and after the task.
You could conduct competitive benchmark as one snapshot in time, or perform longitudinal competitive benchmarking, where you monitor the performance of your product against your competitors over a longer period of time.
Just remember that if you’re comparing a particular tool or functionality – such as adding a lawnmower to a shopping basket – you have to be confident that you and your competitors will continue selling lawnmowers over the length of your study period. After all, it might just be a phase that FOREVER 21 is going through.
Also bear in mind, the more competitors you benchmark against, the more expensive, time consuming and unwieldy the data becomes. So try to keep the group small and meaningful.
Why you would undertake competitive UX benchmarking in the first place? Here’s an executive-friendly guide to why it’s so rad.
This creates one heck of a compelling argument when you can say to your stakeholders and executives, “This is how we’re performing directly against our competitors and this is what we can (or have done) to beat them!”
This is really helpful, as you can be sure your customers aren’t just your customers (I know, heartbreakers) but also your competitors’ customers. Therefore their expectations of your site and its UX are shaped by this.
Sometimes this might be you, sometimes it’s not. Either way, it would be good to know, so you can either celebrate or optimise your stuff so you can meet and exceed these SO CALLED best in class examples.
Sure it may look shiny and fancy, but maybe it’s also a huge waste of time. Copying competitor features is a favourite topic of executives, so anticipating these conversations with this type of research can help you save time, money and a few heated discussions.
Our Senior Director of UX Research, Dana Bishop, highlights some important things to remember when running benchmarking, both before launch and during testing.
You might hope or assume you’re product is a best-in-class, A+ example of modern usability, but you’ll never truly know until you actually measure the experience with your users, and then use this as a benchmark for further rounds of testing and improvements.
Without doing this, you won’t be able to quantify how effective your user experience is, and as you’re probably aware, stakeholders love simple, easy-to-understand numbers. Benchmarking helps you achieve this.
There’s no single perfect moment in time to begin either, just start today! There’s a lot of power in understanding where you are right now – what you’re doing well at, and where users are struggling and why. But you need to create a baseline first, which you’ll need to measure against going forward.
How often should you benchmark? You’ll want to set up regular intervals, whether it’s annually, bi-annually, quarterly, or monthly. It really depends on your development cycle and what works for your organization.
You also don’t want to be too rigid about your cadence. If you roll out a major update or redesign that’s outside your cycle, you should re-run your benchmark and see the effect it’s had. One of the biggest benefits of benchmarking longitudinally is you can measure and compare how design iterations are impacting the user experience.
While you’re tracking changes and KPIs, it’s also important to understand the whys behind significant shifts of performance. So when your quantifiable measurements change, you’ll be able to understand what’s affecting them. We recommend adding a small set of survey type questions to help monitor those types of changes, so you can gather qualitative feedback.
When you’re designing your benchmark study, it’s really critical to invest as much of your time as possible before launch. As this is something you’re designing once, and then running many times into the future, the success of your benchmarking depends on getting it exactly right from the beginning.
It’s tempting to include too much. Often we see people who want to know absolutely everything. But all your questions should tie back to helping you answer your top objectives. You need to know exactly what you’re gonna do with every piece of information and every data point that you collect. Don’t overwhelm users with too many questions and too many tasks, as this just muddies the water. You want to stay lean and focused.
Competitive UX benchmarking is a way to compare some metrics of your product with your direct (or sometimes indirect) competitors. And this a huge part of optimizing the user experience of your website.
Rather than just benchmarking your own progress, competitive benchmarking gives you an entire industry view, helps you identify other useful examples of UX and you can give you insight into whether a competitor’s new feature is worth replicating or ignoring.
This also creates compelling proof to your stakeholders and executives when you can say “This is how we’re performing directly against our competitors and this is how we can beat them!”
Make sure that when you’re creating your participant profile for benchmarking, that you’re targeting either existing customers (if that makes sense for the task you’re benchmarking) or if your intended audience is prospects, ensure that the participants have the right background or experience to meet your important demographic criteria.
For instance, if you’re trying to understand how well your site is set up for customer acquisition, your existing customers aren’t your best target market. They’re already sold on your product. Instead you want to be targeting prospects. You can do this by creating a screener survey that allows you to get the right people testing your product.
The validity of the data derived from your measurements overtime depends on the consistency of your process. You need to use the same screener questions, the same sample size, the same criteria for your participants and the same proportions of demographics.
Keeping everything consistent allows you to measure longitudinal data in a reliable way.