The Task-Based UX Benchmark Study

UserZoom ran a quick task-based UX benchmark study between HBO and Showtime, two well-known premium television and movie companies, to compare the experience of streaming episodes online.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to HBO and 30 went to Showtime
  • The Online Task: You want to stream a free full episode on the website. From the homepage, navigate to where you’d be able to watch an episode of Curb Your Enthusiasm/Billions. When you have reached the correct page we will automatically redirect you and ask a few questions.
  • We validated the tasks via URL validation

conclusion anchor button

Let’s Get Ready to Rumble!

We showed all participants an image of HBO’s and Showtime’s homepage and asked them to choose which site they associated with several UX attributes.

HBO v Showtime homepage

Brand Matrix

Showtime’s homepage is very long and requires more scrolling than HBO’s, which once again leads to the kinds of results we’re seeing here. Showtime’s scroll-heavy homepage is seen as being Overwhelming while HBO’s compact homepage dominated every other category.

Show & Movie Preferences

We asked all 60 participants the kinds of movies and shows they prefer to watch.

We asked the participants who chose other to explain, this is what they wrote:

  • Fantasy
  • Animation
  • Reality

Screenshot Click Test

We split the participants evenly and asked where would they click to start a free trial before purchasing a subscription.

HBO

23% correctly clicked on either Order HBO, or 30% clicked on HBO NOW in the footer. Both of these choices are for subscription packages. 33% clicked on Free Episodes, which is for episodes and not subscriptions.

Showtime

Showtime heatmap

The majority, 77% of participants, correctly clicked on Start Your Free Trial.

Online Task Outcomes

We asked participants to find the page where they would be able to stream a full episode for free – Curb Your Enthusiasm/Billions, respectively. In order to validate whether users were successful at the task we validated via URL.

If users got to this page they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or said they had found it before reaching the correct page, which we labeled as Error.

Success

Both sites had a majority of users successfully complete the online task, but users on HBO’s site were 8% more likely to successfully complete the task than Showtime’s users.

Participants on both sites were able to quickly complete the task with a majority of users being able to find the correct page in under a minute and with minimal page views and clicks. Even the maximum time on both sites is still under 3 minutes, however, we do see a massive jump in the amount of clicks. In the end, HBO’s participants were able to complete the task more efficiently.

HBO user session

Showtime user session

Non Success

Screen Shot 2016-06-15 at 11.56.30 AM

HBO’s non successful participants also spent less time overall – with the one abandon happening just under a minute mark and most errors happening under 2. We asked the participant who abandoned to explain why they felt the need to abandon and they said, “I couldn’t figure out where to go to find the free episodes.”

Showtime’s non successful participants spent more time searching and clicked many more times before they abandoned or made an error. The participants who abandoned on the Showtime site said, “I tried to search for ’Billions’ and I couldn’t exactly find a way how to watch the premiere. I clicked on watch now, but it was taking forever to load” & “I could not find the specified episode to watch.”

HBO user session

Showtime user session

Ease of Use

After the tasks we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.

Participants on both sites felt that the task was, on average, pretty easy to accomplish. In the end, however, HBO scored higher on the ease of use rating than Showtime.

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

HBO

Screen Shot 2016-06-15 at 1.34.59 PM

The participant who chose other said, “I got stuck on the subscription page and when I clicked the HBO logo it wouldn’t take me back to the homepage of the main site which was annoying.”

Showtime

The participants who chose other said:

  • “The free episodes is pretty hidden”
  • “At first it seemed all too clustered but it’s surprisingly easy to navigate after getting used to it – hard to get started though”
  • “Navigation wasn’t exactly confusing, but after finding the Billions page it was a tad difficult to see the Watch for free option”

Brand Perception

We asked participants to rate their perception of the brand before and after their experience with the site.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they weren’t familiar with the brand before the task.

Interestingly enough, both sites didn’t see that big of a change in brand perception with their ratings numbers staying pretty static both before and after the online experience.

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.

HBO’s participants were more likely to recommend the site to a friend, family member or colleague than Showtime’s participants.

Likelihood to Start a Subscription

After participants interacted with the sites we asked them how likely they were to start a subscription with the service based on their experiences.

Both brands had similar response rates in terms of the likelihood that their participants would sign up for a subscription based on their online experiences, but in the end, HBO’s participants were slightly more amenable to signing up for a subscription.

Conclusion

HBO winner

Both sites fought a good fight, with many similarities amongst their results. The winner, however, in this week’s usability battle is HBO! Not only did they have a higher task success rate with fewer abandons, they also had fewer reported problems & frustrations, and higher scores in terms of ease of use, net promoter score and likelihood to start a subscription based on their experience. 


Want to learn more about competitive
UX Benchmarking with UserZoom?

personalized demo