The Task-Based Benchmark Study

UserZoom ran a quick task-based benchmark study between JetBlue and Virgin America, two well-known airline companies, to compare the experience of finding information about loyalty reward points on their websites.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to JetBlue and 30 went to Virgin
  • The Task: Find out how many reward points loyalty members are awarded per dollar spent when booking a flight on their website
  • We validated the task by asking how many reward points loyalty members receive per dollar spent on the site

Let’s Get Ready to Rumble!

Before participants visited the websites, we showed all 60 participants an image of JetBlue’s and Virgin’s homepage and asked them to choose which site they associated with several UX attributes.

jetbluevirginmonitors

Screen Shot 2015-10-05 at 11.24.13 AM

This was a study in contrasting design principles, minimal versus eye-catching, and participants had strong opinions regarding the homepage of each site. Most notably, the overwhelming majority of participants who ranked JetBlue’s homepage as being Lively. As we’ve seen in previous Battles of the Week, there was also a strong correlation between being regarded as Lively with being Welcoming, but also as being seen as Overwhelming.

What’s interesting is that participants were split fairly evenly between which site seemed Easy to use despite Virgin being seen as more Organized and JetBlue being ranked as more Informational.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand. Click on the image to enlarge.

Screen Shot 2015-10-05 at 12.03.06 PM

Both brands had positive initial ratings, with only 2% of participants on each who weren’t familiar with the brand. JetBlue had 67% of participants give them the highest ratings possible for a starting score of 6 out of 7.

Virgin had 43% of participants give them the highest scores possible for a starting score of 5.4 out of 7.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Click on the image to enlarge.

Screen Shot 2015-10-05 at 12.38.17 PM

This is where things get interesting from a research perspective (as you will see in the next section when it comes to task success rates.) After participants interacted with JetBlue’s site 60% of participants gave them the highest ratings possible, and with no negative ratings to bring their score down, they only saw a slight decrease to 5.7 out of 7.

Virgin saw a much more drastic decrease in brand perception post-task from their participants, dropping down to 4.6 out of 7. Only 20% of participants gave them the highest ratings possible while 40% ranked them negatively.

Task Outcomes

In order to validate whether users were successful at finding the reward points loyalty members received per dollar spent while booking a flight on the site, we asked them how many reward points are awarded after the task. If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or thought they had found the correct information but chose the incorrect answer, which we labeled as Error.

Screen Shot 2015-10-05 at 12.37.50 PM

Success

Participants were almost twice as likely to successfully find the correct amount of reward points on Virgin’s site than on JetBlue’s.

Screen Shot 2015-10-05 at 1.09.02 PM

This is why the above findings in brand perception post-task on both sites is interesting – JetBlue was rated more positively than Virgin America despite 70% of JetBlue’s participants failing to find the correct information and those who were successful taking longer to do so than on Virgin’s site.

Take note of the Maximum amount of time a participant took in order to be successful on each site, as well as the standard deviation on the average success time, page views and clicks needed to successfully find the information. Based on the numbers alone one would assume that Virgin would be rated as easier to use than JetBlue.

JetBlue user session (with audio)

Virgin user session (with audio)

Ease of Use

We asked users who successfully completed the task to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy. Click on the image to enlarge.

Screen Shot 2015-10-05 at 4.29.38 PM

Despite the differences in the overall amount of successful participants, as well as the time it took successful participants to find the information, JetBlue was still rated as being easier to use overall by users.

Comparatively, 8 out of 17 (47%) participants on Virgin’s site rated the ease of use positively versus 3 out of 9 (33%) participants on JetBlue’s site. The final factor is that only 22% of JetBlue participants felt negatively while 48% of participants from Virgin felt negatively about their experience.

Non Success

Screen Shot 2015-10-05 at 4.42.09 PM

A majority of JetBlue’s users missed the bonus points for booking on the site, choosing the base 3 points instead of the 6 points awarded for booking on JetBlue.com, which explains the 70% of non successful users. Virgin’s users, however, were split almost evenly between abandoning the task and in choosing a wrong answer based on credit card rewards.

It’s important to note the amount of time participants were willing to spend on each site. JetBlue’s participants spent four and a half minutes searching before abandoning while Virgin’s participants spent less than two before giving up. People who found an incorrect answer on Virgin’s site were only there for about a minute but JetBlue’s users searched for just under three.

We asked participants on both sites who chose an incorrect answer how confident they were that they found the correct amount of reward points:

error confidence both

As you can see, JetBlue’s users felt extremely confident that they had found the correct amount of rewards points, whereas Virgin’s users were decidedly more neutral.

JetBlue user session (video only)

Virgin user session (video only)

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues. Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10. Click on the image to enlarge.

Screen Shot 2015-10-05 at 5.16.02 PM

In our seven years of experience, we have found that the average NPS differs by industry and that it’s not uncommon for brands to have negative Net Promoter Scores. What’s striking about these results are the fact that despite having 70% of users fail to find the correct amount of reward points JetBlue still had a vastly better chance of being recommended than Virgin.

Filtering the Data

We filtered our data to participants who said that the last time they traveled via airline they didn’t book their trip themselves but used a travel agent instead. We wanted to see how their online experience compared with users who said they booked their previous airline flight themselves online. Here are the results:

Success Ratios

Screen Shot 2015-10-06 at 11.11.15 AM

Non Success Questionnaire

JB error confidence filtered


Screen Shot 2015-10-06 at 3.35.36 PM


Screen Shot 2015-10-06 at 3.32.59 PM

Success Questionnaire

Virgin success ease filtered

Net Promoter Score (NPS)

NPS - filtered

While looking at the subset of people who don’t usually book their flights online the result trends are the same as the overall trends we saw earlier. JetBlue still had a lower success ratio but with high confidence ratings among users who erred. The successful participants on Virgin’s site gave Virgin a very low Ease of Use score. When it comes to whether or not this subset of users would recommend the site to others, JetBlue’s NPS was evenly split whereas Virgin’s users were decidedly unlikely to recommend.

Conclusion

For this group of participants, both overall and in the subset of travelers who don’t usually book online, JetBlue turned out to be the clear winner in terms of whether or not participants would recommend the site and service to friends, family and colleagues as well as in being rated an easier site to use despite a majority of users not finding the correct answer.


Want to learn more about
UX Benchmarking with UserZoom?

xperience360-button