The Task-Based Benchmark Study

UserZoom ran a quick task-based benchmark study between Wealthfront and Betterment, two well-known automated investment services, to compare the experience of finding information on minimum account balances on their websites.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to Betterment and 30 went to Wealthfront
  • The Task: Find out what the minimum account balance requirement is
  • We validated the task by asking what the required minimum account balance was

Let’s Get Ready to Rumble!

Before participants visited the websites, we showed all 60 participants an image of Betterment’s and Wealthfront’s homepage and asked them to choose which site they associated with several UX attributes.

bettermenwealthfrontmonitors

Screen Shot 2015-09-21 at 2.12.09 PM

This week’s group of participants were split fairly evenly in regards to each categories ranking, with the notable exception of the Lively category. This continues trends we’ve noticed from previous Battle of the Weeks in which a homepage with little to no images is ranked considerably lower than a homepage with graphics that represent the product or service in terms of being Lively. Interestingly enough, the relationship between being considered “Lively” and “Overwhelming” has been more interconnected in the past. Perhaps most interesting of all is the dead split between companies in regards to seeming Trustworthy – an attribute any financial service strives for and both brands seem to achieve.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand. Click on the image to enlarge.

both pre brand

Both brands had a third of their participants say they were not familiar with the company, but where Wealthfront had an average rating of 5.9 out of 7 with no participants rating them negatively, Betterment had 17% of their participants rank them below a 4. This brought Betterment’s average rating down to a 4.5 out of 7 before participants interacted with their site.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Click on the image to enlarge.

Screen Shot 2015-09-22 at 10.28.27 AM

Betterment saw an increase from 4.5 to a 5.1 out of 7 after participants interacted with the site, with 40% of participants giving them the highest ratings possible. Their negative ratings dropped down from 17% of participants to 6%. Overall, Betterment’s ratings were spread fairly evenly from neutral to very positive.

Wealthfront saw a slight decrease from a 5.9 to a 5.4 out of 7 after participants interacted with their site. They still had 63% of their participants give them the highest ratings possible with only 6% of participants giving them a negative rating after their experience.

Task Outcomes

In order to validate whether users were successful at finding the minimum account balance requirements we asked them how what the balance requirement was after the task. If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or thought they had found the correct information but chose the incorrect answer, which we labeled as Error.

Screen Shot 2015-09-22 at 11.09.41 AM

Success

Participants were slightly more likely to successfully find the minimum account balance on Betterment’s site than on Wealthfront’s, but only by a difference of two participants.

Screen Shot 2015-09-22 at 12.14.02 PM

The real difference wasn’t in the amount of total participants who were successful, but in the time, page views and mouse clicks required on each site to find the information. Despite having slightly more successful participants, Betterment’s successful users spent on average almost 30 more seconds trying to find the information and clicked twice as much as Wealthfront’s successful participants.

Betterment user session (with audio)

Wealthfront user session (with audio)

Ease of Use

We asked users who successfully completed the task to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy. Click on the image to enlarge.

Screen Shot 2015-09-22 at 12.38.24 PM

Betterment’s successful participants rated the ease with which they were able to find the information at 5.1 out of 7. A total of 48% participants rated the site as being Easy to Very Easy to use while 20% of participants rated the site as being Moderately Difficult to Difficult.

Wealthfront, on the other hand, had an average rating of 6.3 out of 7 with 83% of their participants rating the site as Easy to Very Easy to use. Another telling detail is the fact that Wealthfront didn’t have a single participant that rated the site as difficult to use and only a single participant who felt neutral.

Non Success

Screen Shot 2015-09-22 at 1.02.51 PM

What’s interesting about these results is that they are a little counterintuitive based on what we saw from the successful participants – namely, that non successful users spent more time searching on Wealthfront’s site than on Betterment’s before either giving up or finding an incorrect answer. It’s also interesting to note that while a majority of non successful participants on Betterment’s site found an incorrect answer, a majority of the non successful participants on Wealthfront’s site abandoned the task entirely.

Betterment user session (video only)

Wealthfront user session (video only)

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues. Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10. Click on the image to enlarge.

Screen Shot 2015-09-22 at 1.22.52 PM

In our seven years of experience, we have found that the average NPS differs by industry and that it’s not uncommon for brands to have negative Net Promoter Scores. What’s noticeable about these results are the differences in NPS despite the similarity of Success rates.

Filtering the Data

When it comes to niche services, such as automated investment services, it is beneficial to take a look at how the results can change based on whether or not you’re using the general populace or a particular subset – in this case, investors.

We filtered our data to participants who said they actively invest in stocks and bonds and they handle their investments themselves. Here are the results:

Screen Shot 2015-09-22 at 2.17.32 PM

Screen Shot 2015-09-22 at 2.13.49 PM

Screen Shot 2015-09-22 at 2.15.37 PM

While looking at the subset of investors the result trends are the same as the overall trends we saw earlier. Betterment still had a higher success ratio while Wealthfront was rated easier to use and had a much higher NPS. When we filtered for the opposite spectrum of users, the participants who said they don’t do any kind of investing, the results also followed the same trend.

Conclusion

For this group of participants overall, including the investors subset as well as the general population, Wealthfront turned out to be the clear winner in terms of whether or not participants would recommend the site and service to friends, family and colleagues as well as in being rated an easier site to use.


Want to learn more about
UX Benchmarking with UserZoom?

xperience360-button