The Task-Based Benchmark Study

UserZoom ran a quick task-based benchmark study between Abercrombie & Fitch and American Eagle, two well-known fashion retailers, to compare the experience of finding the cost of a pair of jeans.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to Abercrombie & Fitch and 30 went to American Eagle
  • The Task: Find the cost of a pair of flex jeans
  • We validated the task by asking them what the cost is

Let’s Get Ready to Rumble!

Before participants visited the websites, we showed all 60 participants an image of Abercrombie & Fitch’s and American Eagle’s homepage and asked them to choose which site they associated with several UX attributes:

AbericanEagle

Screen Shot 2015-08-10 at 3.48.20 PM

Participants ranked Abercrombie & Fitch higher than American Eagle in almost all categories based on the homepage alone, particularly when it came to rating which site appeared Easy and Organized. American Eagle was rated significantly more Lively than Abercrombie but also more Overwhelming. The ratings are likely due to the fact that Abercrombie took a more minimalist approach to what appeared on their homepage while American Eagle had more sections and took longer to scroll through.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive.

Screen Shot 2015-08-10 at 5.11.54 PM

Both Abercrombie & Fitch and American Eagle’s brand perception started fairly high before participants interacted with their sites. Each brand had an average rating of right around 5 out of 7 or just above neutral. What’s interesting is that both sites were evenly-matched, with only 2 out of 30 participants rating each site negatively while 11 and 12 participants rated them 6 or higher, respectively.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive.

Screen Shot 2015-08-11 at 11.06.42 AM

Abercrombie & Fitch saw a slight decrease in their average brand perception rating after participants interacted with their site, dropping down 0.2 points from a 5.2 to a 5.0 average rating. American Eagle actually saw an increase of 0.3 points to end up with an average rating of 5.2 after participants interacted with their site. Both sites were again evenly-matched post-task, with each site having exactly 4 detractors out of 30 and 6 participants giving each site the highest rating possible.

Task Outcomes

In order to validate whether users were successful at finding the cost of the jeans we asked them what the cost was after the task. If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or thought they had found the correct information but chose the incorrect answer, which we labeled as Error.

Task Outcomes both

Success

Participants were more likely to successfully find the cost of the jeans on American Eagle’s site than on Abercrombie & Fitch.

Success both

Participants who were successful on both sites ended up taking the same average amount of clicks and page views to find the price. American Eagle, while having a higher overall success rating, took an average of 30 seconds longer to find the price on than it did on Abercrombie & Fitch’s site.

Abercrombie & Fitch user session (with audio)

American Eagle user session (with audio)

Ease of Use

We asked users who successfully completed the task to rate how easy or difficult it was to accomplish, with 1 = Difficult and 5 = Easy.

Ease both

What’s interesting is that while American Eagle had a majority of it’s participants successfully complete the task, with only 2 out of 30 people who weren’t successful, participants who were successful on Abercrombie & Fitch’s site actually rated the site slightly easier to use than American Eagle’s. Both sites ended up with an average ease of use rating of around 4 out of 5.

Non Success

Both sites saw all their non successful participants Error rather than Abandon, meaning users found incorrect information rather than giving up on the task. What sets the sites apart is the number of participants that were not successful at the task.

American Eagle only had 2 out of 30 participants Error while Abercrombie & Fitch saw 10 of it’s participants Error during the task.

Screen Shot 2015-08-11 at 3.44.27 PM

What’s interesting is that Abercrombie & Fitch’s non successful participants had the same average amount of time, clicks, and page views as its successful participants. American Eagle’s non successful participants spent almost a minute less searching than their successful participants, indicating in their post-task responses that they believed they had found the right price early on.

Abercrombie & Fitch user session (video only)

American Eagle user session (video only)

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues. Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10.

NPS both

In our seven years of experience, we have found that most brands have negative Net Promoter Scores and that the average NPS differs by industry. What’s noticeable about these results are how evenly matched both sites were in the end, with American Eagle only having one more Promoter and one more Passive than Abercrombie & Fitch despite the differences in task outcomes.

Conclusion

Overall this group of participants preferred American Eagle’s online experience to Abercrombie & Fitch’s, with American Eagle’s users being more likely to successfully complete the task as well as recommend the site to others. An interesting detail is that while more people were successful on American Eagle’s site, Abercrombie & Fitch’s participants were successful in less time and rated their site easier to use than American Eagle’s participants.