A comprehensive guide to UX Benchmarking: Longitudinal and Competitive
Here’s our guide to why you should care about longitudinal and competitive UX benchmarking, an awesome, highly persuasive methodology that can help you tie user research to actual business KPIs.
Anyone who works in user research knows their work can uncover an array of user insights which can inform business and design decisions, and ultimately help create a better experience for users. However, it can be difficult to quantify anecdotal experience, especially if you’re mainly running moderated tests and getting qualitative insight (i.e. people telling you why your product is rubbish or great).
But how can you communicate this data effectively and efficiently to stakeholders? How do you tie an ‘improved user experience’ to improvements in your company’s overall key performance indicators (KPIs)? You worked hard for those insights, dammit, and you should be rewarded!
And even more importantly, how can you measure user experience improvements over time and therefore justify more testing to make your product even more awesome and user-friendly?
One way is through UX benchmarking!!! (You probably guessed this already, because you looked at the headline.)
What is UX benchmarking?
Benchmarking creates a baseline for understanding the current user experience on your website, app or any digital product. And the sooner you get started, the sooner you’ll be able to measure and improve the user experience. You may also hear this referred to as longitudinal benchmarking – which is basically measuring over time.
When you start benchmarking you need to create a baseline. A starting point. Otherwise you’ll have nothing to measure against.
You won’t know how fast you can run 100 metres until you actually put on a pair of running shoes and persuade a confused friend to measure the distance and time for you. The same is true for your website, app or other digital product.
Just pick a feature or tool you want to measure and improve on your website or app – let’s say it’s the checkout section on your retail site. You then run some testing to get some baseline usability metrics.
We discuss methods of testing and metrics in the UX benchmarking ebook, but for this example let’s just say we ran a basic usability test where we timed how long it took a group of 20 test participants to find a specific item and purchase it through checkout. We’ll also look at average number of pages they visited, and whether the task was successful.
You’d then input these figures in a table and then you will have your baseline data to begin tracking this particular user experience over time.
Perhaps within this initial test you got some in-the-moment feedback from your participants where they mention feeling “overwhelmed by so many options during the checkout.” You could then make some changes to the user experience based on this research (for example, simplifying or removing
some options), then you run another round of testing and see how you perform against your baseline.
REMEMBER: keep all the details of the test study the same for each round of testing otherwise it won’t be an accurate benchmark. For instance – the item being purchased, the device/browser being used, the number of test participants, the type of task and any questions you might ask.
You then start to build a picture like this (which is essentially a scorecard, and you can learn all about UserZoom’s UX scorecard program here).
As you make changes and run each round of testing, you’ll begin to see problem areas for improvement. You’ll then see exactly how your improvements are affecting these metrics, and if you tie these metrics to wider company KPIs (improve revenue, improve ease of use, improve customer satisfaction) your user research can be seen as an invaluable tool in the organization.
For an in-depth and entertaining guide to measuring UX, check out our free-to-download, 42 page guide to running both longitudinal and competitive benchmarking.
What can you UX benchmark?
So you know you want to benchmark something, but what exactly can you benchmark?
- Your website, app or any other digital product
- Any of the above versus those of your competitors’
- Different iterations of prototypes through development
- Different versions of your live website or app
- A prototype of a feature versus a live feature of a competitor
- User journeys from one point on your website to another
Or a combination of the above – basically anywhere your users and customers are accessing your site or product online can and should be UX benchmarked.
This applies to external sites as well as internal sites or improving employee intranets, because you want to ensure your online properties are as streamlined and optimized for your internal users as they are for your customers. Just imagine if Amazon had the same UX as your employee expenses portal? *Shudder*
When and how often should you benchmark?
Really you can can begin any time from concept, to live working product, to every iteration that follows. But basically… start NOW!
As for how often?
We work with many different customers who have different intervals in which they run their benchmarking. Sometimes it’s annually, sometime’s biannually and sometimes quarterly or even monthly. It really depends on your development cycle and how often you’re rolling out design changes.
We would also recommend testing before or after launching a major update or redesign that may not fall within your regular intervals, as you don’t want to find that something isn’t working six months after launch, because that’s when your timetable says you’re due another round of tests.
Why is UX benchmarking so valuable?
Why would you undertake UX benchmarking in the first place? Here’s our executive-friendly guide to why UX benchmarking is so rad.
- Engage stakeholders and secure buy-in. While video-based qualitative insights are compelling, you may struggle to get buy-in from more data-driven stakeholders. Therefore you may get increased stakeholder buy-in when you run studies specifically with benchmarking in mind, because the higher sample sizes increase statistical significance, and this kind of robust data brings much more confidence.
- Identify product strengths and weaknesses. Benchmarking across your key customer and user journeys on your digital products enables you to understand specifically where your strengths or weaknesses lie. This can help you to work on reinforcing your competitive advantages with further developments or changes to your messaging.
- Inform better design decisions. Making changes to a digital product without measuring the outcomes quantitatively (i.e. numerically) means that you are operating in a knowledge void. If changes are made without the context of knowing if they’re making an experience better or worse, changes become reactive and based on gut instinct. Dirty Harry operated purely on gut instinct, and although he got results, you’re not Dirty Harry. I’m sorry you had to hear it from this UX blog first.
- Validate designs against business objectives. Benchmarking key user journeys on your digital products prior to and after a significant change will enable you demonstrate the impact of user research on the most important business KPIs, such as conversion rates, basket abandonment or even the bottom line.
- Demonstrate the positive impact of your efforts. Adding quantitative, measurable metrics to your UX research not only helps support your findings in the moment, it will also help you understand trends and provide context to design and business decisions in the longer term.
Longitudinal UX Benchmarking creates a baseline for understanding the current user experience on your website, app or any digital product. And the sooner you get started, the sooner you’ll be able to measure and improve the user experience.
However a huge part of optimizing the user experience of your website is knowing how it performs in comparison to your competitors. Sure you can measure and influence improvements and declines in your own performance, and that’s awesome, but you’re not going to gain a competitive advantage if you’re not measuring against your opponents.
Let’s investigate further…
What is competitive UX benchmarking?
Competitive UX benchmarking is a way to compare some metrics of your product with your direct (or sometimes indirect) competitors.
This is great because your boss may often say to you, “Ooooh I like this fancypants new feature that COMPETITOR X has launched, let’s copy it” and even though your own gut might be saying, “No bad idea, bad idea, wrong!!!” and “When’s lunch?” you don’t really have a concrete argument against copying the feature.
So you just steal it, because they’re the boss and you haven’t discovered the joys of competitive UX benchmarking yet and breaking for lunch feels increasingly like a foreign country.
But thankfully now you’re reading this article, you can say, “Good idea BOSS X, I like what you’re suggesting, but before we spend too much money, let’s build a prototype of the feature and UX benchmark it against our competitors, then we’ll see if our users actually like it or not?” That way you’ve either saved the company some money and proved yourself very wise, or you’ve backed up your boss’s idea with concrete data and proved them as very wise, and hopefully in both cases boosted the value of your user research efforts.
What can you benchmark in a competitive study?
Comparing a prototype or concept with a competitor’s similar feature isn’t the only thing you can do. With competitive benchmarking you can test anything that’s live on your site and compare with any number of competitors or non-competitors (after all, you may find inspiration from outside your own industry).
You could compare the usability of your checkout against another, or you could compare softer metrics such as asking participants, “What is your perception of this company?” before and after the task.
You could conduct competitive benchmark as one snapshot in time, or perform longitudinal competitive benchmarking, where you monitor the performance of your product against your competitors over a longer period of time.
Just remember that if you’re comparing a particular tool or functionality – such as adding a lawnmower to a shopping basket – you have to be confident that you and your competitors will continue selling lawnmowers over the length of your study period. After all, it might just be a phase that FOREVER 21 is going through.
Also bear in mind, the more competitors you benchmark against, the more expensive, time consuming and unwieldy the data becomes. So try to keep the group small and meaningful.
If you’re looking for a single UX metric to clearly prove to stakeholders that your product is improving through user research, let us introduce you to the qxScore! Read all about it here:
Why is competitive UX benchmarking so valuable?
Why you would undertake competitive UX benchmarking in the first place? Here’s an executive-friendly guide to why it’s so rad.
It provides measures of how you’re performing directly related to your competitors
This creates one heck of a compelling argument when you can say to your stakeholders and executives, “This is how we’re performing directly against our competitors and this is what we can (or have done) to beat them!”
You’re getting an entire industry view
This is really helpful, as you can be sure your customers aren’t just your customers (I know, heartbreakers) but also your competitors’ customers. Therefore their expectations of your site and its UX are shaped by this.
Identifying best in class examples
Sometimes this might be you, sometimes it’s not. Either way, it would be good to know, so you can either celebrate or optimise your stuff so you can meet and exceed these SO CALLED best in class examples.
Understand if a competitor’s new feature is worth replicating and iterating
Sure it may look shiny and fancy, but maybe it’s also a huge waste of time. Copying competitor features is a favourite topic of executives, so anticipating these conversations with this type of research can help you save time, money and a few heated discussions.
Eight mantras to remember when UX benchmark testing
We recently launched a benchmarking webinar series presented by our resident expert and Senior Director of UX Research, Dana Bishop – the first part of which covers proving the value and quantifying UX benchmarking. You can watch it for completely free right now!
The above video is a fascinating and wide-ranging discussion on getting up and running with benchmarking while also proving its value to stakeholders. Dana also highlights some important things to remember when running benchmarking, both before launch and during testing.
I’ve pulled out eight of the most vital things to keep in mind…
1) What cannot be measured cannot be managed
You might hope or assume you’re product is a best-in-class, A+ example of modern usability, but you’ll never truly know until you actually measure the experience with your users, and then use this as a benchmark for further rounds of testing and improvements.
Without doing this, you won’t be able to quantify how effective your user experience is, and as you’re probably aware, stakeholders love simple, easy-to-understand numbers. Benchmarking helps you achieve this.
There’s no single perfect moment in time to begin either, just start today! There’s a lot of power in understanding where you are right now – what you’re doing well at, and where users are struggling and why. But you need to create a baseline first, which you’ll need to measure against going forward.
2) Be flexible
How often should you benchmark? You’ll want to set up regular intervals, whether it’s annually, bi-annually, quarterly, or monthly. It really depends on your development cycle and what works for your organization.
You also don’t want to be too rigid about your cadence. If you roll out a major update or redesign that’s outside your cycle, you should re-run your benchmark and see the effect it’s had. One of the biggest benefits of benchmarking longitudinally is you can measure and compare how design iterations are impacting the user experience.
3) Understand the ‘why’
While you’re tracking changes and KPIs, it’s also important to understand the whys behind significant shifts of performance. So when your quantifiable measurements change, you’ll be able to understand what’s affecting them. We recommend adding a small set of survey type questions to help monitor those types of changes, so you can gather qualitative feedback.
4) Invest the time upfront
When you’re designing your benchmark study, it’s really critical to invest as much of your time as possible before launch. As this is something you’re designing once, and then running many times into the future, the success of your benchmarking depends on getting it exactly right from the beginning.
5) Stay focused
It’s tempting to include too much. Often we see people who want to know absolutely everything. But all your questions should tie back to helping you answer your top objectives. You need to know exactly what you’re gonna do with every piece of information and every data point that you collect. Don’t overwhelm users with too many questions and too many tasks, as this just muddies the water. You want to stay lean and focused.
6) Get competitive
Competitive UX benchmarking is a way to compare some metrics of your product with your direct (or sometimes indirect) competitors. And this a huge part of optimizing the user experience of your website.
Rather than just benchmarking your own progress, competitive benchmarking gives you an entire industry view, helps you identify other useful examples of UX and you can give you insight into whether a competitor’s new feature is worth replicating or ignoring.
This also creates compelling proof to your stakeholders and executives when you can say “This is how we’re performing directly against our competitors and this is how we can beat them!”
7) Your research is only as good as your participants
Make sure that when you’re creating your participant profile for benchmarking, that you’re targeting either existing customers (if that makes sense for the task you’re benchmarking) or if your intended audience is prospects, ensure that the participants have the right background or experience to meet your important demographic criteria.
For instance, if you’re trying to understand how well your site is set up for customer acquisition, your existing customers aren’t your best target market. They’re already sold on your product. Instead you want to be targeting prospects. You can do this by creating a screener survey that allows you to get the right people testing your product.
8) Consistency is king
The validity of the data derived from your measurements overtime depends on the consistency of your process. You need to use the same screener questions, the same sample size, the same criteria for your participants and the same proportions of demographics.
Keeping everything consistent allows you to measure longitudinal data in a reliable way.
If you’d like to know more about how UserZoom can help test, measure and improve your own site’s UX, please get in touch!
Christopher is the Content Marketing Manager, which basically means the skipper of the good ship ‘UserZoom blog’. So far his requests for changing its name to the ‘USS-erzoom Blog’ have been rightfully denied. In his spare time, Christopher is a filmmaker and the editor of wayward pop culture site Methods Unsound. He used to be the deputy editor of Econsultancy, editor of Search Engine Watch, staff writer for ClickZ and features editor of CMO.com.