What metrics and KPIs do the experts use to measure UX effectiveness?

Why do we need to measure UX?

It’s all well and good us sitting around in our ivory tower yelling how great UX research is out of the window to passers-by, but this can only get us so far.

Occasionally some of those people look up and yell back, “yeah I know! It just makes common sense to make design decisions based on actual human behavior” but then they often make the following point…

“But how can we measure that? If we run usability testing and make a change to a website that presumably improves the user experience based on our observations, how do we really know the change has worked? What UX metrics can we use to measure success? How do we prove to our bosses that the investment is worth it?”

It’s normally around the ‘metrics’ mark where we start to close the window and mumble something about “having to keep it shut because of the air-con, sorry I can’t hear you.”

Metrics were traditionally a difficult discussion when it comes to measuring the success, failure of shrugging indifference of your UX. Every other discipline has it made!

• You want to measure how well your blog post did: Look at your traffic, see the time on page, notice how many times it has been shared, judge the quantity and quality of comments.
• You want to measure your social channels: Look how many followers you have. Is there growth? Are they influential in your niche? Do they comment? Do they share? Are they entirely bots?
• You want to measure the changes made to the order of the categories in your main menu: Well it definitely looks better to you! Have you run some more usability testing to see if people are still struggling with it? Perhaps traffic from the homepage to those categories has improved, but there’s no guarantee it’s because of the changes.
• You want to measure the quality of your Bakewell Tart: Did I eat the whole damn thing? Probably, but that’s not a testament to how good it is. I’ll often eat an entire Bakewell Tart with the same ease that I take a breath. Did I demand that you make me another one? Yes! Now that’s a quality Bakewell Tart.

As we already know, data only shows part of the story. Google Analytics can tell you what’s happening but not why it’s happening. If you’re only going by analytics, you’re essentially guessing. Sure, it can be an educated, highly informed guess – but you won’t know exactly why things are happening on your site until you see real people using it.

But UX measurement doesn’t have to be an intangible mystery. As you’ll see below, there are many ways to prove the value of UX research.

What’s the difference between behavioral and attitudinal UX metrics?

We work with various companies across all industries and have noticed certain metrics that are most commonly used for benchmarking (either over a period of time or compared against competitors). We broadly divide them into two categories:

Behavior (what they do)

In the user research world, it’s critical to understand what people are doing, and how they are using your products. Task-based usability testing is a standard method to gather this information across the industry. We don’t mean just ‘in-lab’ think-out-loud studies, but also remote moderated studies, which will help you access larger sample sizes in an efficient way.

Typical metrics you could capture include these task-level behavioral measurements:

• Abandonment Rate
• Pageviews
• Problems and Frustrations
• Task Success
• Task Time

Attitude (what they say)

How users feel, what they say before, during or after using a product, and how this affects brand perception.

To measure this, you might want to capture these attitudinal metrics:

• Loyalty (using scores such as SUS or NPS – more on these further down)
• Usability (or ease of use)
• Credibility (taking things like trust, value and consideration into account)
• Appearance (“oooooh pretty!” or “OW MY EYES!!!” etc.)

But how do you quantify opinion? How do you take these “oooooh pretty” or “OW MY EYES!!!” hot takes and turn them into a simple score that any busy executive can understand?

Let’s take a deeper dive into these individual metrics, and we’ll see how they can help to form a bigger picture.

Behavioural UX metrics

Abandonment Rate

Quite simply, how many people have come to your online retail store, put a bunch of products in their basket, and then just left without checking out. Behaviour that would make a real world IKEA visit a treacherous assault course. The abandonment rate is the ratio of the number of abandoned shopping carts to the number of initiated transactions.

AOV: Average Order Value

AOV means average order value, and this is simply your total revenue / number of checkouts. According to VWO this is a “direct indicator of what’s happening on the profits front.” If your UX efforts directly tie into increasing cross-selling or upselling, then AOV can be an indicator of whether you’ve improved things or not.

Conversions

Helpful if there’s a specific thing triggered by a UX improvement. Say for instance a web-form completion, newsletter sign-up or some other task completion. If the site change directly impacts how many people are converting in that specific task, and you can measure that accurately, then you can be *fairly* confident you made an impact.

Just remember that having a higher conversion count may also be a result of marketing efforts, so be sure to measure the conversion rate (typically Number of Sales / Number of Visits).

As NN/g suggests:

“The conversion rate measures what happens once people are on your website. Thus it’s greatly impacted by the design and it’s a key parameter to track for assessing whether your UX strategy is working.”

And because we like to argue both sides of the… uh… argument… here’s ecommerce whiz kid Dan Barker on why you shouldn’t necessarily trust conversion rate as the solution to all your problems. Remember that not all visitors to your webpage have the potential to convert, or that conversion rates vary wildly based on visitor type.

Pageviews

Website page views and clicks are a common metric. For mobile apps, or even web applications or even single-page web apps, some combination of clicks, taps, number of screens or steps can be measured.

If you are running an in-lab study, counting these can be extremely tedious. But, if you are using a user research platform like ours, most of these metrics are captured automatically and significantly reduce analysis and reporting time. In most cases, combining these, or at least connecting them to analytics data (from the live site or apps) is beneficial.

Problems & Frustrations

These can be measured as a Number of unique problems identified and/or Number (or %) of participants that encounter a certain problem. We recommend conducting Think-Out Loud studies to identify problems, and then quantify them via a large-sample study to find the % of problems actually encountered by a large population (with confidence intervals).

Most of these Behavioral KPIs are collected ‘per task’ and then aggregated as an average for a given study, and/or digital product. These are then compared over a period of time (e.g. each quarter) or compared with competitors’ digital products.