The Task-Based Benchmark Study

UserZoom ran a quick task-based benchmark study between Box and Microsoft’s OneDrive, two well-known file sharing and personal cloud based content management services, to compare the experience of finding how much storage a personal license gives you.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to Box and 30 went to OneDrive
  • The Task: Find how much storage you get with a free personal license
  • We validated the task by asking them how much storage space they would receive

Let’s Get Ready to Rumble!

Before participants visited the websites, we showed all 60 participants an image of Box‘s and OneDrive‘s homepage and asked them to choose which site they associated with several UX attributes:

BOTW

 

Brand Matrix

We’ve seen a trend in the UX Battles of the Week where people associate text heavy homepages with positive attributes like Organized, Trustworthy and Informational while also saying they find them Overwhelming, and by default, less Easy and less Welcoming. It’s up to each brand to listen to their customers and target audience to decide how they want their brand to appear at first glance – welcoming and easy but perhaps less trustworthy or organized and informational but also slightly overwhelming.

OneDrive’s picture laden homepage certainly won over this group of participants when it came to being Welcoming and Easy but ultimately ranked less favorably than Box in regards to seeming Informational and Trustworthy.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Given that participants had not yet visited the website, we also included the option for participants to say they’re not familiar with the brand.

Screen Shot 2015-08-25 at 1.32.37 PM

Both Box and OneDrive started with brand perceptions that were positive before participants interacted with their sites. Box started with an average rating of 5.2 out of 7. OneDrive started slightly higher, with an average rating of 5.7 out of 7. OneDrive had 40% of its participants say they were not familiar with their brand versus Box’s 27%.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive.

Brand Perception Post BOTH

Box saw an increase of 0.5 points after participants interacted with their website, ending with an average brand perception of 5.7 after a majority of participants who previously rated the site neutrally moved into the positive spectrum. Only 2 participants out of 30 rated Box negatively.

OneDrive saw a slight decrease, dropping to 5.0 from a starting average rating of 5.7. The number of participants who rated their site neutrally increased slightly and 3 participants out of 30 rated them negatively, ultimately lowering their score. What’s interesting is that OneDrive only had 3% of its participants give them the highest rating versus Box’s 27%.

Task Outcomes

In order to validate whether users were successful at finding the amount of online storage we asked them how much storage space a free personal license gave them after the task. If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or thought they had found the correct information but chose the incorrect answer, which we labeled as Error.

Screen Shot 2015-08-25 at 4.10.57 PM

Success

Participants were slightly more likely to successfully find the amount of free storage space on OneDrive’s site than on Box’s, only differing by a few participants when it came to the amount of successes and non successes.

Screen Shot 2015-08-25 at 4.30.21 PM

Despite having strikingly similar success rates, the websites had noticeable differences when it came to the time and clicks needed to find the information on storage space for personal licenses. On average it took participants on OneDrive’s site half a minute less time to find the information than on Box’s site, with less than half as many clicks and page views needed.

Box user session (with audio)

OneDrive user session (with audio)

Ease of Use

We asked users who successfully completed the task to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.

Ease of use both

Despite the differences in time on tasks, page views and clicks, successful participants rated both sites as being Very Easy to use with more than half of participants on each site giving them a 7 out of 7. Both sites tied with an average ease of use rating of 6.1 out of 7.

Non Success

Despite having similar amounts of non successful participants who either erred or abandoned, there was again a vast difference in the amount of time it took for them to do so.

Screen Shot 2015-08-26 at 12.32.25 PM

On Box’s site, people who thought they had found the correct information did so in around 30 seconds while participants who ended up abandoning the task continued to search for around 2 minutes – almost double the average time for successful participants on the site.

What’s interesting is that on OneDrive’s site this was reversed – participants were quick to abandon the task after only 23 seconds and 5 page views while users who erred searched for the same average amount of time as successful participants did.

Box user session (video only)

OneDrive user session (video only)

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues. Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10.

NPs both

In our seven years of experience, we have found that the average NPS differs by industry and that it’s not uncommon for brands to have negative Net Promoter Scores. What’s noticeable about these results are the differences in NPS despite the similarities in Success and Non Success rates.

Conclusion

Overall this group of participants preferred Box’s online experience to OneDrive’s, with Box’s participants ultimately being almost twice as likely to recommend the site than OneDrive’s users.

Why the discrepancy when both sites performed almost equally on the task? Take a closer look at what happened when we tasked participants with finding information on security with our infographic companion piece.