The Task-Based Benchmark Study

UserZoom ran a quick task-based benchmark study between RBC Royal Bank and M&T Bank, two well-known banking sites, to compare the experience of using online loan calculators.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to RBC and 30 went to M&T
  • The Task: Calculate the monthly loan payment amount for a 60 month, $25k new car auto loan at 0.5% interest
  • We validated the task by asking participants what the monthly loan payment amount would be

Let’s Get Ready to Rumble!

Screenshot Click Test

Before participants visited the websites we wanted to get a better understanding of their first impressions of the site, which is why we split the participants evenly and asked where they would click if they wanted to order personal checks from the homepage.

In this case we asked participants to only click once which is why the percentage of users and clicks is the same. Click on the images to enlarge.

RBC Royal Bank

RBC heatmap

As you can see, a majority of participants on RBC’s site clicked in two places: Checking or Order Checks. 40% of users clicked on Checking under Everyday Banking while 30% clicked on Order Checks. Both options lead users to the option of ordering checks for a success rate of 70%. The remaining participants clicked on Sign In, Bank Accounts and the Search Bar.

M&T Bank

M&T heatmap 2

Similarly, a majority of M&T’s participants clicked on Checking and Order Checks, which were split evenly at 40% each for a combined success rate of 80%. The remaining 20% clicked on Customer Service, Commercial, Open an Account and EZChoice Checking.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand. Click on the image to enlarge.

Screen Shot 2015-12-03 at 9.57.45 AM

RBC Royal Bank started with a brand perception rating of 5.1 out of 7, with 36% of participants giving them the highest ratings. 27% of participants said they were unfamiliar with the brand while 10% of participants reported a negative perception of the brand.

M&T Bank had a slightly higher starting brand perception of 5.7 out of 7 due to no negative ratings and 40% of participants giving them the highest ratings possible. Slightly more participants, 30%, said they were unfamiliar with M&T.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Click on the image to enlarge.

Post Brand Perception

RBC Royal Bank’s brand perception post-task stayed the same for an average rating of 5.1 out of 7 with 53% of participants giving them the highest ratings possible. M&T, however, saw their brand perception drop down to 5 out of 7 with 36% of participants giving them the highest ratings.

Task Outcomes

In order to validate whether users were successful at calculating the monthly auto loan payment, we asked participants what the monthly payment amount would come out to. If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or chose an incorrect payment amount, which we labeled as Error.

Screen Shot 2015-12-01 at 5.03.51 PM

Success

Both websites saw a fairly high success rate for the task but participants were 3% more likely to successfully calculate the monthly auto loan payment on RBC’s site than on M&T’s.

Screen Shot 2015-12-03 at 10.16.18 AM

RBC’s successful participants managed to calculate the monthly loan payments using the loan calculator in just under 2 minutes and after 4 page views. M&T Bank’s users, while taking slightly longer, were able to do so with less overall clicks.

RBC user session

M&T user session

Non Success

Screen Shot 2015-12-03 at 11.14.11 AM

RBC’s non successful users were slightly more likely to abandon the task than to make an attempt at calculating the monthly payments. What’s interesting is that the users who abandoned did so after the same average amount of time it took the participants who erred to use the calculator. Upon watching the videos it appears that users either balked at filling out the form, which accounts for the lower clicks and less overall time spent on the site versus the users who were successful, or used the wrong calculator as in the video below.

M&T Bank’s non successful users were split evenly between abandoning and erring. The average amount of time users who erred spent was similar to the amount of time that successful users spent but with less clicks while participants who abandoned did so in much less time. Similar to RBC, upon watching the videos it was evident non successful users were hesitant to fill out all the forms or used the wrong calculator.

RBC user session

M&T user session

Ease of Use

After the task we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy. Click on the image to enlarge.

Screen Shot 2015-12-03 at 11.19.48 AM

Both sites ended up with a neutral average rating in terms of ease of use, but RBC’s was slightly higher with a 4.9 out of 7 while M&T had an average rating of 4.2.

User Satisfaction

After the task we asked all the users to rate how satisfied they were with the site, with 1 = Very Unsatisfied, 4 = Neutral and 7 = Very Satisfied. Click on the image to enlarge.

Screen Shot 2015-12-03 at 11.55.12 AM

RBC’s users were overall more satisfied than M&T’s users, with RBC ending with an average of 5.2 out of 7 versus M&T’s 4.8. Most telling is the fact that 66% of RBC’s users gave them the highest ratings while M&T only had 34% of their participants give them the highest ratings.

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

RBC

Screen Shot 2015-12-01 at 5.08.15 PM

If users chose Other they were given an open-ended question to describe their problem or frustration. The user who chose this for RBC said:

  • “The site needs more color – there is way too much white.”

M&T

Screen Shot 2015-12-01 at 5.10.24 PM

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.

Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10. Click on the image to enlarge.

Screen Shot 2015-12-03 at 11.57.17 AM

As we saw through the entire battle, both websites were very similar in their average ratings from participants. But in the end, RBC Royal Bank had slightly more promoters and slightly less detractors for an average NPS of -37% versus M&T’s -47%.

Conclusion

The winner this week is RBC Royal Bank for having a higher percentage of users successfully calculate the monthly loan payments, as well as having higher user satisfaction, ease of use, and NPS rankings. M&T Bank still performed well, with users not far behind in each of the categories.


Want to learn more about
UX Benchmarking with UserZoom?

xperience360-button