The Task-Based Benchmark Study

UserZoom ran a quick task-based benchmark study between PlayStation and Xbox, two well-known video game consoles, to compare the experience of finding the cost of an annual membership service on their websites.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to PlayStation and 30 went to Xbox
  • The Task: Find out how much a yearly subscription to their premier membership service costs
  • We validated the task by asking them what the price is

Let’s Get Ready to Rumble!

Before participants visited the websites, we showed all 60 participants an image of PlayStation’s and Xbox’s homepage and asked them to choose which site they associated with several UX attributes. Click on the image to enlarge. 

psxbox

Screen Shot 2015-09-08 at 4.51.25 PM

We’ve seen a trend in the UX Battles of the Week where people associate a scroll heavy homepage, i.e. a homepage that takes awhile to scroll from top to bottom, with negative attributes. It’s up to each brand to listen to their customers and target audience to decide how they want their brand to appear at first glance.

Xbox’s scroll friendly homepage won over this group of participants, ultimately outscoring PlayStation in every positive category. It should be noted, however, that both homepages were almost an even split when it came to being overwhelming despite the difference in page length.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Given that participants had not yet visited the website, we also included the option for participants to say they’re not familiar with the brand. Click on the image to enlarge.

Screen Shot 2015-09-09 at 12.04.59 PM

Both PlayStation and Xbox started with brand perceptions that were positive before participants interacted with their sites. PlayStation started with an average rating of 5.3 out of 7. Xbox started slightly higher, with an average rating of 5.6 out of 7.

Both brands had a majority of participants say they were familiar with them, with only 1 participant out of 30 saying they weren’t familiar with Xbox.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Click on the image to enlarge.

Screen Shot 2015-09-09 at 12.38.06 PM

PlayStation saw a decrease after participants interacted with their website, ending with an average brand perception of 4.3 out of 7. Xbox also saw a slight decrease in brand perception, down to 4.7 out of 7, after participants interacted with their site.

Task Outcomes

In order to validate whether users were successful at finding the cost of an annual membership we asked them how much it cost after the task. If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or thought they had found the correct information but chose the incorrect answer, which we labeled as Error.

Screen Shot 2015-09-09 at 12.45.48 PM

Success

Participants were more likely to successfully find the cost of an annual membership plan on PlayStation’s site than on Xbox’s.

Screen Shot 2015-09-09 at 12.52.18 PM

Despite the difference in the total amount of successful participants for each site, both brands had strikingly similar efficiency rates when it came to the average time on task, page views and clicks required to successfully find the information.

PlayStation  user session (with audio)

Xbox user session (video only)

Ease of Use

We asked users who successfully completed the task to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy. Click on the image to enlarge.

Screen Shot 2015-09-09 at 1.08.50 PM

Despite the similarities in time on tasks, page views and clicks, successful participants rated PlayStation’s site as being easier to use than Xbox’s, with 54% of PlayStation participants rating the site a 6-7 out of 7. Xbox only had 24% of its participants rate them at 6-7 out of 7.

Non Success

Despite the overall difference in the amount of non successful participants, both brands are again similar in the amount of time participants searched before abandoning the task.

Screen Shot 2015-09-09 at 1.23.25 PM

What’s interesting is that on both site’s there was an overwhelming amount of participants who simply gave up, and not after a brief time spent searching either. Given that both brands offer membership deals that should appeal to their customers (free games on a monthly basis, the ability to play online with other console owners, and discounts) it is interesting to note that users weren’t finding a wrong answer – they weren’t finding any answers.

Xbox user session (with audio)

PlayStation user session (video only)

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues. Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10. Click on the image to enlarge.

Screen Shot 2015-09-09 at 1.38.19 PM

In our seven years of experience, we have found that the average NPS differs by industry and that it’s not uncommon for brands to have negative Net Promoter Scores. What’s noticeable about these results are the similarities in NPS despite the differences in Success and Non Success rates.

Conclusion

This week we have no clear winner in the battle of PlayStation vs Xbox when it comes to finding out more information on their premier membership services. Despite double the amount of participants who abandoned the task on the Xbox site, both brands had strikingly similar time on task, page views and clicks for both successful and non successful participants as well as identical overall NPS.


Want to learn more about
UX Benchmarking with UserZoom?

xperience360-button