If you’re struggling to visualise qualitative results from benchmark studies in a clear and accessible way, try this method devised by Jen Lewis, Senior UX Researcher at UserZoom – Qualitative Heatmaps!

In the last year I have undertaken several large scale benchmark studies, typically comparing four or more sites at a time. Quantitative metrics from multiple sites from these benchmarks (such as ‘time-on-task’) can be quickly and effectively visualised in a single bar graph. However, condensing qualitative data into one slide presents much more of a challenge.

So let me recommend an easier way to deliver UX insights – qualitative heatmaps!

But first let’s take a look at your other options, and why they present a challenge.

The challenges of traditional thematic analysis

The standard method for analysing qualitative comments is to perform a ‘thematic analysis’. This is where each open comment is coded into a theme.

In the example below we can see that the first user simply reports a technical issue, whilst the second user reported both a technical issue and stated they were unable to enter their details in to the booking panel.

When complete, the column headings of your thematic analysis will show you the full range of usability issues that were encountered. At this point you may be tempted to just summarise these issues in a paragraph, however it is vital that you quantify how often each issue was encountered.

For example, if 25 participants encountered technical issues and 1 struggled to locate the search functionality, it would be misleading to list these two issues together in a manner that implies they occur with equal frequency. So how would you summarise the data?

Summarising thematic analysis data

The next step is to turn your counts in to percentages. Let’s say 5 of the 100 participants who completed a task on ‘Amazing airlines’ reported technical issues. Simply divide 5 (participants who reported a technical issue) by 100 (the total number of participants in that segment) to get a figure of 5%.

Then repeat this step for each issue encountered on each site.

Visualising thematic analysis data

When reporting usability issues for a single site you can simply list out the issues from the highest to the lowest frequency. For example;

  • 21% experienced technical issues
  • 15% struggled to enter details in to the booking panel
  • 10% struggled to locate the search functionality
  • 9% could not work out how to proceed to next page
  • 8% were unable to sort the flights by departure time
  • 8% struggled to understand content on pages
  • 6% were unable to enter their personal details in to the forms
  • 2% unable to navigate back to previous steps

However, because these lists are usually quite long, the data from one site usually takes up an entire slide. In benchmarks this means having to flick between four or more slides to compare the issues encountered on each site.

My initial solution was to display the data in a bar graph like the one shown below. This allows the all data to a be displayed on one page and gives a visual marker of how frequently the issue occurs on each site. However, the downside is that when you have a wide range of issues the width of each bar becomes microscopic!

Solution: qualitative heatmaps!

Recently a benchmark I was running had over 20 quantitative metrics that were being compared across four sites. The client wanted to me condense these findings in to one slide that would instantaneously highlight which sites performed better/worse and show if there was any particular area of the sites that needed improvement.

Their suggestion was what they called an ‘overview heatmap’ which would look something like this:

In the example above the ‘traffic light’ colour coding makes it instantly apparent that Amazing Airlines is the best performing site whilst Disastrous Airlines is the worst. And, when looking at the metrics for Disastrous Airlines, it would appear that a poor search functionality was driving this lacklustre performance.

So much insight on just one slide!

I instantly fell in love with heatmaps and began to wonder, could this technique be used to summarise qualitative data too?

After a few experiments I came up with the format below. It has the same layout as the quantitative overview heatmap – but instead of the traffic light colour coding it uses shades of red to indicate whether an issue is a high or low frequency occurrence.

In the example above the red shading makes it immediately apparent that technical issues are a high frequency occurrence across all sites.

In addition to this, the heatmaps highlight that: a high proportion of those using Better Airlines were unable to sort the flights by departure time, over a third of those using Crummy Airlines could not enter their details into the booking panel and a quarter of users on Disastrous Airlines had difficulty locating the search function.

While other issues were reported, the pale shading indicates that they occurred at a low frequency that does not warrant concern.

Qualitative heatmaps are now my standard format for presenting qualitative data in reports. They satisfy most clients desire for maximum insight/minimum slides.

If you want to embellish things a bit you can always add a few key participants comments in callout boxes. . . .

I’d love to hear your views on this new method so please get in touch or email me directly: jlewis@userzoom.com