Understanding how your website compares to your competitors’ websites is critical to winning customers, increasing conversion rates and providing a positive user experience (UX) that boosts your site’s credibility and improves customer retention.

Your site’s user experience is a powerful representation of how your company values its customers. It’s no secret to the UX world that customers who feel confident while engaging with a site are more likely to purchase and/or engage with your company both on their initial site visit and over time.

The bottom line is whether your company is already leading-edge or just starting out, it’s crucial to study how users interact with your own site as well as your competitors’ in order to obtain and retain a competitive advantage.

What is UX benchmarking?

UX Benchmarking involves testing your online properties to get baseline usability metrics for improvements, which are then tested and compared against the baseline over time in order to understand how your site, app or other digital product is progressing.

The outcome of benchmark studies are baseline performance metrics that help identify problem areas for improvement as well as building a vision for future releases.

What is competitive UX benchmarking?

An important component of optimizing the user experience of your website is knowing how it performs in comparison to your competitors. Competitive benchmarking is a research method where you compare your website’s key performance indicators (KPIs) with your direct or indirect competitors.

What makes UX benchmarking valuable?

You can’t manage what you can’t measure. UX Benchmarking allows you to measure and quantify your online user experience in order to manage it and make improvements over time. It helps you keep a finger on the pulse of your business in order to understand what’s working and what could be improved by having actionable data to inform your decisions.

  • Obtain competitive performance metrics on usability KPIs: Know precisely how your website fares against your competitors on usability KPIs such as: Response Times, Success Rate, Satisfaction, Brand Perception, etc.
  • Identify strengths and weaknesses: What is your competitive advantage? What are your weaknesses? Identify functional areas where you exceed your user’s expectations as well as areas that need improvement.
  • Gain market intelligence and domain knowledge: Gather key insights into your competitors’ product strategies. Get metrics on how users perceive your brand and product offerings to ultimately gain a competitive edge.
  • Develop new insights and inspiration: What are the product goals that will drive revenue for the next release? Get competitor insights that help establish clear design directions. Get data driven insights that help create a product that is superior and more innovative than the competitors’.

What can you benchmark?

You can test a desktop site, a mobile site, an app, a prototype or a combination of the aforementioned – basically anywhere your users and customers are accessing your site or product online can and should be benchmarked.

This applies to external sites as well as internal sites or intranets, because you want to ensure your online properties are as streamlined and optimized for your internal-facing users as they are for your customer-facing sites and external users.

How to choose which KPIs to measure in a UX benchmarking study

Different companies will be at different stages of implementing user-centred design (UCD) and optimizing the UX of their sites/apps, which means they will have different benchmarking objectives.

If companies are conducting preliminary benchmarking as a means of quantifying UX for the first time, then they will want to track KPIs that offer a baseline view of their digital product. What this does is allow for a starting place where teams can look at the metrics and decide which areas need to be improved upon.

The four essential KPIs to measure and track baseline performance are Task Success Rates, Time on Task, Unique Page Views and Clicks.

which kpis to choose in a ux benchmark

Task success rates allow you to gauge how effectively your users and customers are able to complete the core tasks they came to do. Time on task allows you to measure how quickly or efficiently they’re able to complete these tasks. Measuring efficiency against your competitors is a key step in beginning your competitive analysis.

Unique page views and clicks gives you insight into how users are navigating your site or app and whether or not the taxonomy is helping or hindering their search. You can use these effectiveness metrics to measure against the same user goals on competitor sites. Effectiveness can be gathered by way of data tracking a users’ path and their arrival on specific page(s) on a site. Success could also be self-reported via questions posed to users.

results1

You may already be using ease-of-use and satisfaction metrics for your site via surveys or SUS (Satisfaction Usability Scale). By collecting these same metrics from a study where users engage with competitor sites, they become valuable benchmark metrics.

NPS (Net Promoter Score) measures the likelihood of users to recommend a website to a friend or colleague. It’s a popular metric with many companies. When you conduct user research on competitive sites, consider measuring NPS for your site as well as competitor sites.

Screen Shot 2015-03-12 at 12.32.36 PM

Screen Shot 2015-03-12 at 12.32.45 PM

Another interesting metric for usability studies is to measure brand perception before and after users engage in certain tasks on a site. You’ll be able to understand how positively or negatively you or your competitors’ brands are perceived after interacting with a site.   This gets to the heart of how user experience impacts brand credibility and loyalty.

brandperception2

 

Determine which UX KPIs fit best with current business objectives.

Further refining your goals

Once a baseline benchmark has been measured there will be weaknesses and strengths. Depending on how companies address these areas they will have different benchmarking objectives. Clearly defining these objectives and tying them to business goals is the key to choosing the right KPIs to track.

Based on our research team’s experience we were able to draft a few examples of common business objectives and which KPIs to benchmark:

  • Are current business goals focused on increasing registration rates? Measure time-on-task in the registration flow
  • Increasing sales or signups? Measure task success rates, satisfaction with experience, # of clicks to complete a purchase, or likelihood to purchase
  • Decreasing customer service calls? Measure effectiveness of Help or FAQ content

Our researchers also had some more general advice to keep in mind when it comes time to choose which KPIs to track during a UX benchmark:

  • Lean toward KPIs that can be easily tied to, and presented with, other regularly reported business metrics or analytics data (to give your KPIs maximum visibility)
  • Choose a cadence and a period over which they will be measured (for 1 year, 2 years, etc.)
  • Have hypotheses about the direction you expect KPIs to move, and then check your hypotheses as you go

Benchmarking multi-task processes

It doesn’t have to be a single task that is benchmarked, either. There are likely processes that are vital to the success of your business such as registering a new account, getting a price quote, completing a checkout, or scheduling a visit with a doctor. These are multi-step tasks that can all be baseline measured and optimized to make the collective process better for your users.

How to collect competitive benchmarking metrics

You may be wondering exactly how to set up usability studies or other user research that will facilitate collecting competitive data and KPIs. Here are a few tactics to consider:

  • A popular and recommended approach is to conduct regular usability studies on both your site and competitors’ sites and gather feedback from users in-the-moment as they are interacting with each site.
  • Consider the number of metrics you want to collect, and the types of corresponding tasks you want users to attempt during a usability study. Then consider how long it may take a single user to perform those tasks on multiple websites (yours and competitors’). Asking a single user to perform multiple tasks on multiple sites during a single usability test session may be overwhelming and cause users to become fatigued, confused or ambivalent. You may consider designing studies where each user tests only one site, and you then compare the results across users (in the research world, this is known as between-subjects design).
  • Although by definition benchmarking focuses on quantitative data, you’ll want to supplement your benchmark metrics with qualitative data as well. This is easily accomplished by using tools such as UserZoom that allow you to record usability test sessions including screen recordings and audio & video feedback from users as they complete tasks. Use video footage to show your stakeholders examples of pain points on your site or competitors’ sites.
  • It’s worth highlighting that UserZoom allows you to facilitate remote usability testing with large numbers of users (e.g. in the hundreds) in order to obtain statistically relevant benchmarking data. Studies utilizing remote users can be conducted in a matter of days, allowing you to collect and share data quickly and regularly.
  • Finally, by conducting studies in automated remote usability testing platforms, you are easily able to copy and repeat studies as frequently as needed. A perfect match for benchmarking purposes.

It should go without saying that competitive UX benchmarking only works when comparing a task or tasks that are achievable on all sites. In general, the more specific the task the easier it will be to measure KPIs and to analyze the benchmark.

You’ll want to carefully test each task on each site to make sure that the measurements are captured consistently, such as the amount of clicks required or average page views.

Competitors’ websites are also prone to change without you expecting it, so careful monitoring is a must both while building the study and while the study is live to ensure your task or task validation still makes sense.

Speaking of validating, you also want to spend time ensuring how you validate the task includes all possible outcomes. For example, if you’re validating by URL make sure to test if it’s always the same URL or if it changes depending on how users navigate there.

And just like any other benchmark study, be sure to formulate hypotheses and questions that will validate them. A problems and frustrations list can help validate aspects of the site or app that aren’t working well.

UX Battle of the Week - TaxSlayer problems

Between-subject vs. Within-subject

Once the tasks have been tested and the study built, you’re going to have to make a decision on whether to do a Between-Subjects or a Within-Subjects study. Put simply, do you want the same participant to perform tasks on each site (within-subjects) or only perform a task on one site (between-subjects)?

There are a few factors to consider when deciding, namely sample size, participant time & fatigue, overall number of tasks, and learning curves.

Having participants do tasks on multiple sites cuts down on the sample size needed, but it also increases the amount of time participants have to be willing to invest in the study. Another potential issue would be learning curves – if a participant has never done a task before they are likely to be slower, but would apply their experience on the second site. This would require researchers to randomize which site participants visit first in order to counter the learning curve effect.

Having participants only do tasks on one site means you can ask them to perform more tasks, but you might have participants who are faster or slower than average which wouldn’t be applied equally across both sites. And as stated earlier, you essentially need double the sample size.

Jeff Sauro - pros and cons of within-subjects vs bewteen-subjects

Analyzing competitive benchmark results

In your analysis of the competitive benchmark results be sure to point out any areas where experiences across sites may not be apples-to-apples (e.g. different user flows), while at the same time, use those scenarios to learn about what your competitors are doing and watch for best practices that may be emerging.

If you did decide to do a within-subjects approach, don’t forget the obvious questions of which site(s) did users prefer and why. Also, be sure to share competitive UX benchmark data with others in your organization that conduct any other type of competitive analysis for maximum exposure.

Why conduct UX benchmarking with UserZoom?

  • Fast and cost effective: Convince stakeholders by conducting large-scale testing quickly and cost-effectively with UserZoom’s remote unmoderated testing methodology – no moderation or expensive lab equipment needed.
  • Study templates: When you create your first Benchmark study within UserZoom you can use it as a template for all your future benchmark studies, allowing you to quickly and easily test iterations.
  • Get the complete picture with both types of data: Get actionable insights by collecting both quantitative and qualitative data in a single study to better understand your users’ experiences.
  • Quick results: As soon as your study is active you can begin collecting results in real time, allowing you to see feedback as participants complete your study.
  • Recruit your target audience: Whether it’s testing participants who are geographically dispersed or whether you have a specific target audience in mind, UserZoom’s flexible recruiting options allows you to recruit the right participants into your studies.
  • Benchmarking all devices: Whether you want to Benchmark your website, mobile site, mobile app, or prototypes for all the above, UserZoom’s platform has you covered.
  • Benchmark against competitors – With UserZoom you can even test your competitor’s websites, allowing for a direct comparison of key usability metrics.

UX benchmarking is about letting real users help you reveal pain points on your own site and competitors’ sites. When you benchmark KPIs on your site vs. competitors over time, you keep the competition in view and benefit from data and inspirations that can drive future enhancements to your site – ultimately leading to an industry-leading user experience.