The Task-Based UX Benchmark Study

UserZoom ran a quick task-based UX benchmark study between T-Mobile and Verizon, two well-known wireless service providers, to compare the experience of finding information about setting up automatic payments online.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to T-Mobile and 30 went to Verizon
  • The Online Task: Find the page that tells you how to set up automatic payments for your bill
  • We validated the task via URL validation when participants got to the correct page

Let’s Get Ready to Rumble!

We showed all 60 participants an image of T-Mobile’s and Verizon’s homepage and asked them to choose which site they associated with several UX attributes.

tmobile homepage botw

T-Mobile v Verizon UX attributes

This week’s participants preferred T-Mobile’s homepage over Verizon’s in every category, particularly when it came to being seen as Easy, Welcoming, Lively and Helpful. The votes were much more close when it came to being seen as Organized, Trustworthy, and interestingly enough, Overwhelming.

Screenshot Click Test

We split the participants evenly and asked where they would click if they wanted to learn more about the carrier’s business solutions.

T-Mobile

T-Mobile heatmap UX BoTW

60% of participants correctly clicked on Business in the top navigation bar. Of the outside areas, Plans and Why T-Mobile were the highest clicked areas for a combined 30% of the clicks.

Verizon

Verizon heatmap UX BOTW

77% of participants correctly clicked on Business in the top navigation bar. Smartphones, Plans, Tablets and Why Verizon areas had a combined 20% of the clicks.

Tree Test

We asked participants where they felt they would be able to learn more about international coverage.

T-Mobile Tree Test UX BOTW

Verizon Tree Test UX BOTW

T-Mobile had a 93% success rate, with participants finding the correct selection in under 10 seconds. Verizon had a 33% success rate with an average time of 23 seconds.

Online Task Outcomes

In order to validate whether users were successful at finding the page for automatic bill payments, we did a URL validation on the page. If users managed to get to the page from the homepage they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website, or said they had completed the task without reaching the correct page, which we labeled as Error.

success rates

Success

Users were 20% more successful at finding information for setting up automatic bill payments on Verizon’s site than on T-Mobile’s.

T-Mobile v Verizon success efficiency

Successful users on Verizon’s site were able to find the page in about 47 seconds less time than T-Mobile’s participants and with less clicks. Both sites had similar amounts of average page views.

T-Mobile user session

Verizon user session

Non Success

T-Mobile v Verizon UX BOTW Non Success Rates

T-Mobile users were slightly more likely to abandon the task entirely than to make an error whereas Verizon users were more likely to err than abandon. What’s interesting is that participants on Verizon’s site tended to stay on the website far longer before abandoning or erring compared to participants on T-Mobile’s site.

T-Mobile user session

Verizon user session

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand.

T-Mobile v Verizon brand perception pre-task

T-Mobile started with a neutral brand perception of 4.6, while Verizon started with a slightly higher score of 5.3 out of 7.

Brand Perception Post-Task

T-Mobile v Verizon UX BOTW brand perception post task

After participants interacted with T-Mobile’s site their brand perception rose slightly to 4.8 out of 7. 44% of users giving them the highest score, which was up from 37% pre-task.

Verizon saw a slight decrease in brand perception after users interacted with the site, ending with 4.6 out of 7. 37% of users still gave them the highest score, but that dropped down from 53% pre-task.

Ease of Use

After the task we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.

T-Mobile v Verizon UX BOTW ease of use

Given the success rates this is probably not very surprising, but participants on the T-Mobile site felt that the task of finding information about automatic bill payments on the site was rather difficult – the average ease of use score was a 2.8 out of 7.

Verizon fared slightly better and ended with a neutral ease of use rating of 4.2 out of 7.

User Satisfaction

After the task we asked all the users to rate how satisfied they were with the site, with 1 = Very Unsatisfied, 4 = Neutral and 7 = Very Satisfied.

T-Mobile v Verizon UX BOTW satisfaction

Both brands had user satisfaction ratings that were fairly evenly spread out across the spectrum. T-Mobile’s users ended up giving an average satisfaction rating of 3.9 out of 7, while Verizon’s users average rating was 4.2.

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

T-Mobile

T-Mobile UX BOTW problems and frustrations

Verizon

Verizon UX BOTW problems and frustrations

If users chose Other they were given an open-ended question to describe their problem or frustration. The users who chose this for Verizon said:

  • “The video is a great idea but it was a little buried and hard to find”
  • “Should have a ‘videos’ section that’s easy to find in the main menu”

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.

T-Mobile v Verizon UX BOTW NPS

Both brands were incredibly close in regards to NPS, but in the end T-Mobile had one more Promoter than Verizon, which edged them in for a narrow victory.

Conclusion

T-Mobile v Verizon UX BOTW Winner

The winner this week is Verizon for having a higher percentage of users successfully find the automatic payment information, as well as for having a higher overall ease of use and user satisfaction score.


Want to learn more about competitive
UX Benchmarking with UserZoom?

personalized demo