The Task-Based UX Benchmark Study

UserZoom ran a quick task-based UX benchmark study between Fitbit and Microsoft Band, two well-known wearable fitness technologies, to compare the experience of finding information about their products and apps online.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to Fitbit and 30 went to Microsoft Band
  • The Online Task: You want to try out the free health app to experience the tracking dashboard on your iPhone before purchasing. From the homepage, find the link to download the app to your phone
  • We validated the task via URL validation when participants got to the page to download the mobile app

Let’s Get Ready to Rumble!

We showed all 60 participants an image of Fitbit’s and Microsoft Band’s homepage and asked them to choose which site they associated with several UX attributes.

monitors-fitbitmsband

Screen Shot 2016-01-08 at 12.27.49 PM

As we usually see in a matchup between a condensed homepage that requires almost no scrolling (Fitbit) versus a long, scroll-heavy homepage (Microsoft Band), participants’ initial reaction is that the condensed homepage is seen as being Easy, Welcoming and Helpful while the scroll-heavy homepage is seen as being Informational but Overwhelming. Interestingly, participants were split almost 50/50 on which website appeared more Organized.

Screenshot Click Test

We split the participants evenly and asked where they would click if they wanted to learn more about taking care of their wearable fitness tracker.

In this case we asked participants to only click once which is why the percentage of users and number of clicks is the same. Click on the images to enlarge.

Fitbit

Fitbit Heatmap

A third of the participants, 33%, correctly clicked on Help in the top nav bar while the remaining 67% clicked on outside areas. Of the outside areas, Products and Shop ‘Em Now were the highest clicked areas for a combined 20% of the clicks.

Microsoft Band

Screen Shot 2016-01-08 at 12.49.26 PM

Slightly more participants, 37%, correctly clicked on Support specifically for Microsoft Band. The next highest clusters were on Features and the Support link for all Microsoft products for a combined 30% of the clicks.

Tree Test

We asked participants where they felt they would find downloadable guided workouts using the site’s information architecture.

The numbers in the left column represent that section being clicked on, while the numbers in the right column represent how many participants selected that section as their answer. Using this we know if participants were able to successfully choose the correct section on their first try.

Fitbit

Screen Shot 2016-01-08 at 1.12.47 PM

17% of participants, 5 out of 30, were able to find the correct section using the site’s current information architecture. All of the successful participants chose the correct section, FitStar, on their first attempt. The incorrect section with the highest percentage of clicks was Setup.

Microsoft Band

Screen Shot 2016-01-08 at 1.25.56 PM

30% of participants, 9 out of 30, were able to find the correct section using Microsoft Band’s current information architecture. Of the 9 successful participants, 7 were able to find it on their first try. The incorrect section with the highest percentage of clicks was Features.

Online Task Outcomes

The companion health apps for both of these fitness trackers are integral to the use of the devices – so much so that both companies offer the apps for free (with limited capabilities such as step counting) in order for consumers to experience the tracking and exercise dashboards before buying a tracker.

In order to validate whether users were successful at being able to find the page for the companion health app, we did a URL validation of the companion app page in the iTunes store. If users managed to get to the app page from the website they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or said they had completed the task without reaching the app page in the iTunes store, which we labeled as Error.

Screen Shot 2016-01-08 at 1.36.10 PM

Success

Users were 16% more successful at finding the companion app on Fitbit’s site than on Microsoft Band’s.

Screen Shot 2016-01-11 at 12.12.54 PM

Although more participants were able to complete the task on Fitbit’s site, successful users on Microsoft Band’s site were able to find the companion app in about half a minute less time than Fitbit’s participants. Both sites had similar amounts of average page views and clicks required.

Fitbit user session

Microsoft Band user session

Non Success

Fitbit v MS Band non-success

Users on both sites were more likely to Error rather than Abandon, meaning participants felt they had reached the correct page but hadn’t clicked on the link that took them to the page where they would be able to download the app.

More worrisome are the people who could not find the app’s location and gave up the task. These participants spent similar amounts of time, mouse clicks and page views as successful participants but ended up searching other pages such as the store or the compatible apps. Or, as in the Microsoft Band example below, simply didn’t scroll down far enough on the page.

Fitbit user session

Microsoft Band user session

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand. Click on the image to enlarge.

Fitbit v MS Band brand perception pre

Fitbit started with a fairly high brand perception of 6.2 out of 7 due to not having any negative ratings.

Microsoft Band’s brand perception was more mixed across the spectrum. With roughly 30% of participants giving Microsoft negative, neutral and positive ratings they started with a neutral average brand perception of 4.5 out of 7.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Click on the image to enlarge.

Fitbit v MS Band brand perception post

Fitbit’s average brand perception after participants interacted with the site dropped slightly, but with 64% of users giving them the highest ratings possible Fitbit’s final brand perception was 5.8 out of 7.

Microsoft Band’s brand perception rose slightly to 4.9 out of 7 after participants interacted with the site. 37% of users giving them the highest ratings possible while 20% remained neutral.

Ease of Use

After the task we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy. Click on the image to enlarge.

Fitbit v MS Band ease of use

Fitbit’s users were mixed on how easy it was to find the companion app and their ratings ran the entire gamut of responses fairly evenly. This is why Fitbit’s average ease of use rating was in the neutral range with 4.4 out of 7.

Microsoft Band’s users felt the site was slightly less easy to use, ending with an average rating of 3.8 out of 7.

User Satisfaction

After the task we asked all the users to rate how satisfied they were with the site, with 1 = Very Unsatisfied, 4 = Neutral and 7 = Very Satisfied. Click on the image to enlarge.

Fitbit v MC Band ease of use 2

Despite having 17% of users rate their satisfaction as negative, Fitbit’s average satisfaction rating was 5.2 out of 7 with 53% of users giving them the highest ratings possible.

Microsoft Band’s average satisfaction rating was 4.5 out of 7, with ratings split fairly evenly along the spectrum.

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

Fitbit

Screen Shot 2016-01-08 at 4.24.47 PM

Microsoft Band

Screen Shot 2016-01-08 at 4.25.30 PM
If users chose Other they were given an open-ended question to describe their problem or frustration. The users who chose this for Microsoft Band said:

  • “There’s too much information to scroll through to get what I want”
  • “The images are too large, not enough useful content”
  • “The site is too long, so you need to scroll a lot to reach the bottom for more info”

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.

Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10. Click on the image to enlarge.

Fitbit v MS Band NPS

With slightly more than double the amount of promoters than Microsoft Band, Fitbit pulled ahead with a final NPS of -13% versus Microsoft Band’s -50%.

Conclusion

winner-fitbit

The winner this week is Fitbit for having a higher percentage of users successfully find the companion app, less overall problems and frustrations reported, as well as for having higher user satisfaction, ease of use, and NPS rankings.


Want to learn more about competitive
UX Benchmarking with UserZoom?

personalized demo