UX Battle: Superbowl Edition featuring Instacart
It’s Superbowl Sunday
You have all your friends over and everyone’s jammed around the TV. It’s 21-19 in overtime and the Falcons have 3 seconds left for a field goal on the Patriots 34 yard line. That’s when you hear the most dreaded words that can be uttered at a party:
“We’re out of food.”
There’s no way anyone’s leaving the game to run to the store, but thankfully, there’s Instacart to the rescue. If you’re not familiar with them, Instacart is a grocery delivery service that allows you place an order online and have the groceries delivered to your home or office.
The Task-Based UX Benchmark Study
We placed real people in the above scenario and ran a quick UX benchmark study on Instacart.com to see how they experienced navigational and core tasks, as well as how they rated their overall experience on the website.
- We ran an unmoderated remote task-based benchmark study with 40 in-the-wild users on their own devices over the course of a single day
- They completed several tasks while on the site: a quick survey, a screenshot click test, and a task-based test where we asked them to place ice, tortilla chips and dip in their cart and proceed to the checkout page
- We also measured their brand perception and how they rated their overall experience
Who are the Players?
Before we had our participants begin their tasks we asked them if they had ever used Instacart.
Of the 15 who said they had, we wanted to know how they usually accessed Instacart.
Screenshot Click Test
To understand if visitors were able to easily find important information, we showed them an image of Instacart’s website. Based on that image, we asked them to complete a simple task: “Where would you click if you wanted to buy a gift card?”
20% of users correctly clicked on Account. The largest percentage of clicks (28%) was actually on the search bar. The remaining users were spread fairly evenly amongst Help, Home, Departments, Coupons and the Instacart logo.
Online Task-Based Test
We asked them to find and place a bag of ice, tortilla chips and dip in their cart and proceed to the checkout page. To evaluate the success of the task, validation through an URL was used so that only participants who reached the correct page would have the task be rated as success and then be redirected to further questions.
A Non-Success result meant that the participant decided to either abandon the task, which was marked as an Abandon, or they said they had found the page before reaching the correct URL which was marked as an Error.
So, how did participants on Instacart do?
Overall 50% of participants were successful on the site. Of those who were successful, they were also quite speedy.
In the context we placed them in, being able to spend 1:11 minutes on average to get some Superbowl necessities is quite the bonus. Perfect amount of time to place an order during a commercial break (you can watch the best Superbowl commercials on YouTube, anyways.)
Instacart user session
The non successful participants happened to break evenly into two groups – Abandons and Errors. We asked the participants who abandoned why they did so, and the primary reason they gave was because they felt uncomfortable having to register up-front. Which is unfortunate, because this means some people aren’t even experiencing the product before leaving the site. More than one participant that abandoned mentioned that they would have been much more likely to try out the service if they weren’t made to immediately create an account. Others abandoned after getting lost in the navigation, such as clicking on Become a Shopper.
After reviewing the Error videos, it was clear that users were falling into either one of two categories – they were either adding items to their cart and not clicking checkout or they happened to add 3 items that didn’t add to the $10 minimum order amount and said they were done when it wouldn’t let them checkout.
Ease of Use
After the tasks we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.
Given the varied experiences we saw above, it should come as little surprise that the ease of use ratings were also accordingly varied. Interestingly, the two largest groups were at the extremes of the likert scale. That being said, 61% of the tallies were in the positive end of the spectrum. The mean ease of use for Instacart was 4.5 out of 7, or just above neutral.
Problems & Frustrations
We asked the users which, if any, of the following problems they encountered while on the site.
We gave the participants that selected “Other” an open ended comment box to explain. This is a representative sampling of what they said:
- “A provider is not available in my Zip Code area”
- “I hate signing up for stuff I just want to order and pay”
- “I feel like there are a lot more options in store and everything was much more expensive than the store version”
We asked participants to rate their perception of the brand before and after their experience with the site.
For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they weren’t familiar with the brand before the task.
Before they interacted with the site, the mean brand perception rating was a positive 5.5 out of 7.
Net Promoter Score (NPS)
After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.
In our experience doing these UX Battles, we typically see this range of NPS from the general population for sites like this.
While there’s no way to know if our football scenario will play out on Sunday, one things for sure: if you do happen to need some tasty snacks, ice, or beverages to cheers (or drown sorrows in) while the games on – you won’t have to be the only person in the party missing out on the action to get them. Even if you’re only there for the commercials.
Phil got his degree in creative writing, where they told him he most likely wouldn’t be able to use his degree for his career. He obviously won that round. When not working with UX researchers he can be found teaching martial arts and working on his fiction novels.