The Task-Based UX Benchmark Study

UserZoom ran a quick task-based UX benchmark study between Base and Nimble, two well-known CRM solutions, to compare the experience of registering for a free trial, checking integrations and uploading contact lists.

  • We ran the unmoderated remote task-based benchmark study with 20 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 10 went to Base and 10 went to Nimble
  • The Online Tasks: Register for a Free Trial, Check if MailChimp can be integrated with the CRM, Upload a contact list CSV file
  • We validated the tasks via self-reported success/error/abandon, in conjunction with watching their session replay videos

conclusion anchor button

Let’s Get Ready to Rumble!

We showed all participants an image of Base’s and Nimble’s homepage and asked them to choose which site they associated with several UX attributes.

Base vs Nimble desktops

 

Base v Nimble brand attributes

We’ve seen this before when one homepage is minimal versus a homepage that requires more scrolling – generally, visitors first impressions are that the more minimalist homepage (Base) is seen as being Easy, Welcoming and Organized. The more scroll-heavy homepage (Nimble) is seen as being Lively, Informational but also Overwhelming.

Screenshot Click Test

We split the participants evenly and asked where they would click if they wanted to request a demo on the homepage.

Base

Base heatmap

80% of participants correctly clicked on Request A Demo. Of these, half clicked on Request A Demo in the header while the other half clicked in the middle of the page.

Nimble

Nimble heatmap

Participants were a little unsure on where to Request a Demo, but 50% of participants estimated it could be found under Try It Free. The remaining 50% were spread evenly around How It Works, Sign Up, Pricing, and More.

Online Tasks Outcomes

The very first thing we asked participants to do was register for a free trial, which all participants were able to do. Once registered, we asked them to complete two tasks in a logged-in environment: Check to see if MailChimp is an available integration, and Import a list of Contacts from a CSV file.

In order to validate whether users were successful we asked users to self-validate. If users reported they were successful and their session replay videos corroborated this they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or said they were unable to complete the task, which we labeled as Error.

Base vs Nimble success rates

Success

MailChimp Integration

Users were 10% more successful on Nimble’s site than on Base’s for the first task of checking for integrations with MailChimp.

On average, participants on Base’s site were able find information on the MailChimp and other integrations more than a minute faster than participants on Nimble’s site. Base’s users were also able to find the information with less clicks and less than half the page views required on Nimble’s site.

Base user session

Please keep in mind we blocked screen recording on any pages with personally identifiable information.

Nimble user session

Please keep in mind we blocked screen recording on any pages where personally identifiable information could be collected.

Import Contacts

We started participants on the same page they had ended on in the previous task. Users were 10% more successful on Base’s site than on Nimble’s for the second task of importing a list of contacts from a CSV file.

Nimble’s participants were able to complete the task in slightly less time than Base’s participants.

Base user session

Nimble user session

Non Success

MailChimp Integration

This was the task participants on both sides had the most difficulty with, as is evident by the amount of time and clicks that participants invested before having to abandon the task. Participants who erred thought they may have found the correct section but weren’t positive.

Base user session

Nimble user session

Import Contacts

Fewer participants on both sites had difficulties with this task. What’s interesting is that the participants who didn’t abandon the task on Nimble both said they weren’t sure whether their file had uploaded or not (see Problems & Frustrations section below).

Base user session

Nimble user session

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand.

Brand Perception Pre-Task

Both solutions started with relatively high amounts of participants say they were unfamiliar with the brand/service.

Brand Perception Post-Task

Base saw a slight drop in brand perception after participants interacted with the site. Nimble, however, saw a much more noticeable drop in brand perception after the online tasks.

Ease of Use

After the tasks we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.

Registering for a Free Trial

MailChimp Integration

Import Contacts

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

Base

Nimble

Participants who chose “Other” were given an open-ended comment box to explain. The participants who chose this for Nimble said:

  • “I couldn’t easily tell whether my .csv was uploading or completed, it didn’t tell me”
  • “Pop up windows when I clicked on things to help train me were very distracting and I couldn’t tell whether my contact file was uploaded”

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.

Nimble’s participants were slightly more likely to recommend the site than Base’s participants.

Conclusion

Both sites performed well and this battle was extremely close. The winner, however, in this week’s battle of CRM solutions is Nimble for having higher overall success rates on the tasks, fewer overall abandons, fewer problems reported, and a higher NPS.


Want to learn more about competitive
UX Benchmarking with UserZoom?

personalized demo