It’s Superbowl Sunday

You have all your friends over and everyone’s jammed around the TV. It’s 21-19 in overtime and the Falcons have 3 seconds left for a field goal on the Patriots 34 yard line. That’s when you hear the most dreaded words that can be uttered at a party:

“We’re out of food.”

There’s no way anyone’s leaving the game to run to the store, but thankfully, there’s Instacart to the rescue. If you’re not familiar with them, Instacart is a grocery delivery service that allows you place an order online and have the groceries delivered to your home or office.

The Task-Based UX Benchmark Study

We placed real people in the above scenario and ran a quick UX benchmark study on Instacart.com to see how they experienced navigational and core tasks, as well as how they rated their overall experience on the website.

  • We ran an unmoderated remote task-based benchmark study with 40 in-the-wild users on their own devices over the course of a single day
  • They completed several tasks while on the site: a quick survey, a screenshot click test, and a task-based test where we asked them to place ice, tortilla chips and dip in their cart and proceed to the checkout page
  • We also measured their brand perception and how they rated their overall experience

Who are the Players?

Before we had our participants begin their tasks we asked them if they had ever used Instacart.

Of the 15 who said they had, we wanted to know how they usually accessed Instacart.

Screenshot Click Test

To understand if visitors were able to easily find important information, we showed them an image of Instacart’s website. Based on that image, we asked them to complete a simple task: “Where would you click if you wanted to buy a gift card?”

20% of users correctly clicked on Account. The largest percentage of clicks (28%) was actually on the search bar. The remaining users were spread fairly evenly amongst Help, Home, Departments, Coupons and the Instacart logo.

Online Task-Based Test

We asked them to find and place a bag of ice, tortilla chips and dip in their cart and proceed to the checkout page. To evaluate the success of the task, validation through an URL was used so that only participants who reached the correct page would have the task be rated as success and then be redirected to further questions.

A Non-Success result meant that the participant decided to either abandon the task, which was marked as an Abandon, or they said they had found the page before reaching the correct URL which was marked as an Error.

So, how did participants on Instacart do?

Success

Overall 50% of participants were successful on the site. Of those who were successful, they were also quite speedy.

In the context we placed them in, being able to spend 1:11 minutes on average to get some Superbowl necessities is quite the bonus. Perfect amount of time to place an order during a commercial break (you can watch the best Superbowl commercials on YouTube, anyways.)

Instacart user session

Non Success

The non successful participants happened to break evenly into two groups – Abandons and Errors. We asked the participants who abandoned why they did so, and the primary reason they gave was because they felt uncomfortable having to register up-front. Which is unfortunate, because this means some people aren’t even experiencing the product before leaving the site. More than one participant that abandoned mentioned that they would have been much more likely to try out the service if they weren’t made to immediately create an account. Others abandoned after getting lost in the navigation, such as clicking on Become a Shopper.

After reviewing the Error videos, it was clear that users were falling into either one of two categories – they were either adding items to their cart and not clicking checkout or they happened to add 3 items that didn’t add to the $10 minimum order amount and said they were done when it wouldn’t let them checkout.

Ease of Use

After the tasks we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.

Given the varied experiences we saw above, it should come as little surprise that the ease of use ratings were also accordingly varied. Interestingly, the two largest groups were at the extremes of the likert scale. That being said, 61% of the tallies were in the positive end of the spectrum. The mean ease of use for Instacart was 4.5 out of 7, or just above neutral.

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

We gave the participants that selected “Other” an open ended comment box to explain. This is a representative sampling of what they said:

  • “A provider is not available in my Zip Code area”
  • “I hate signing up for stuff I just want to order and pay”
  • “I feel like there are a lot more options in store and everything was much more expensive than the store version”

Brand Perception

We asked participants to rate their perception of the brand before and after their experience with the site.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they weren’t familiar with the brand before the task.

Before they interacted with the site, the mean brand perception rating was a positive 5.5 out of 7.

It’s normal to see a slight dip in brand perception after participants interact with the site. In this case it was a minor dip, with a final mean brand perception rating of 5 out of 7.

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.

In our experience doing these UX Battles, we typically see this range of NPS from the general population for sites like this.

Conclusion

While there’s no way to know if our football scenario will play out on Sunday, one things for sure: if you do happen to need some tasty snacks, ice, or beverages to cheers (or drown sorrows in) while the games on – you won’t have to be the only person in the party missing out on the action to get them. Even if you’re only there for the commercials.