So much more than carving your name onto a bench (also, don’t make marks on benches).

As part of our UX 101 education series, where we discuss the different types of studies and research methodologies you can use with our own user research platform, we’d like to introduce our readers to the exciting world of competitive benchmarking!

What is competitive benchmarking?

Competitive benchmarking allows you to better understand how your site or product performs against your competitors. It is used to add context to your own results by understanding if your metrics and KPIs are higher or lower than those of your competitors, therefore highlighting areas for improvement.

Typically, this is seen as the next steps after you have gathered a baseline benchmark for how your site/product performs. If you want to learn how to get a baseline performance and how to benchmark your own site/product after time, check out this article on longitudinal benchmarking.

Keep in mind that the true power of competitive benchmarking increases as you repeat it. We recommend repeating at least quarterly as this would allow for a consistent understanding of your performance versus competitors. You might also want to conduct this after a large redesign/feature implementation has been completed by both you or a competitor.

What are typical use cases for competitive benchmarking?

Competitive benchmarking not only adds context to your longitudinal benchmark results by comparing and contrasting them with your competitors, but it can also help show you and your team what works and what doesn’t across your entire industry by uncovering industry-wide gaps.

Both the discovery of gaps and issues from an industry standpoint, as well as the contextualization of how your performance compares with your peers, combine together to help teams prioritize new features/necessary improvements and help to link UX to business KPIs (e.g if participants can buy the same item quicker in another store this can contribute to revenue loss/customer retention issues).

Here are more common use cases for conducting competitive benchmarks:

  • Want to know how you compare to your competitors? This is for you.
  • Want to know what makes you or your competitors better? This is for you.
  • Want to know if you offer a more user friendly experience compared with your competitors? This is for you.
  • Want to implement a feature similar to what your competitors have but are unsure of its merits? This is for you.
  • Have you just redesigned your site and want to see how it performs against competitors? This is for you.
  • Are you seeking a higher budget and need real metrics to show your stakeholders? Do I need to say it…THIS IS FOR YOU!

How does competitive benchmarking work?

The process is, for the most part, the same as a longitudinal benchmark you would run on your own product. First and foremost you focus on a specific task or set of tasks (booking a holiday; sending money; buying a product, etc.)

Next, figure out which research questions you will need in order to get the metrics you want to obtain (task success, task speed, ease of use, etc). After that you will need to gather the assets you need to show participants (remember, you need the site/pages of your competitors as well as your own.)

Once these steps are done you simply need to finish building the study and recruit your participants.

Practical advice for running a competitive benchmark

There are, however, a few tips that apply specifically to competitive benchmarking that we would like to share with you.

Keep in mind that competitive benchmarking can quickly become time consuming depending on a few different factors, such as: the number of competitors, the amount of qualitative questions that are being asked and the granularity or broadness of research questions.

Also, be aware that your important metrics are most likely going to be quantitative which is why you should be looking to recruit at least 50 participants per competitor (so if you are benchmarking your site vs 3 others, that’s 200 in total). Think of this number as a guideline to improve your statistical rigor.

Because of the aforementioned we recommend limiting the number of competitors to up to 5. Also, as an aside, because these tend to be quantitative studies it is a fantastic way to make use of the benefits of unmoderated testing.

When should you use competitive benchmarking?

Here are the essential facts to consider while deciding on whether or not competitive benchmarking is the right approach for your research goals.

Pros:

  • Allows you to establish measurable metrics in order to measure your performance versus peers.
  • Enables you to compare across a variety of competitors at once.
  • Assists you in remaining competitive and understanding the limitations/gaps that you or your competitors have.
  • Very easy to retest future iterations.

Cons:

  • Competitive benchmarking can be time consuming if a high number of competitors and tasks are used.
  • This research method requires a higher number of participants which can be costly and time consuming to recruit.
  • Access to competitor assets are required for this to work. This can prove difficult in some cases (i.e sites that require login/membership/ an account).

What results do you get?

As always it depends on what questions you write in, but typically speaking, there will mainly be quantitative findings (e.g. success rates, time on task, page views and clicks). These can be reported dashboard style in a single-page or in a report format that addresses each research question. The former is great for liaising with non researchers as the questions can be compared to company KPIs.

Qualitative findings, if you collect them, can be categorized into themes for more detailed insights (remember – quant only gives you the how many and how much).

Tips for analyzing your results

Always clean the data before analyzing. This means removing data (or participants) that seems invalid or inaccurate that can influence the results of the study.

Inline with the above point, another tip we implement when running an unmoderated study is to throw in an open-ended qualitative question which asks participants to input an answer. This is to catch those rushing through/using gibberish (speeders/cheaters in the UserZoom platform language) which can make it easier to clean your data. Plus, you now have some qualitative data to work with.

Finally, and this is true for many of your research engagements, don’t get lost within the data. Always refer to the research questions unless you find something crucial that is outside of the scope.

This concludes our introduction to competitive benchmarking. Thank you for reading and don’t forget to check out the rest of our UX 101 education series to help you on your way!