The Task-Based UX Benchmark Study

We ran a quick competitive UX benchmark study between two European fashion companies, H&M (Swedish) and Scotch & Soda (Dutch), to understand how users experienced navigational and core tasks and also how they rated their overall experience on these websites.

  • We ran an unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • Participants were equally divided between the companies’ UK sites so that 30 went to www2.hm.com/en_gb/index.html and 30 went to www.scotch-soda.com/gb/en/home
  • They completed several tasks while on the site: a screenshot click test, a tree test on their customer support menu and a task-based test, where we asked them to look for a boy’s shirt
  • We also measured their brand perception and how they rated their overall experience

conclusion anchor button

First impressions are lasting impressions

For starters we wanted to know the users’ first impression to both of the contenders’ sites. We showed all 60 participants images of the H&M and Scotch & Soda homepages and asked them to choose which site they associated with several UX attributes.

 

This first round was totally dominated by H&M. H&M ruled over every single of the UX terms, only being comparable to Scotch & Soda on the terms Lively and Overwhelming.

Without a doubt, Round 1 goes to H&M!

Screenshot Click Test

To understand if visitors were able to easily find important information, we showed them each an image of the assigned contender’s homepage. Based on that image, we asked them to complete a simple task: “Where would you click if you were looking to get email updates from the brand?”.

83% of the H&M participants managed to succeed on this test by clicking on either the top link for the Newsletter sign-up page (63%), or on the “Sign Up For Newsletter” field located on the bottom of the page (20%). The rest of the visitors clicked on other parts of the homepage: 3% on the “My H&M” and the remaining 14% on the hompage’s pictures.

The Scotch & Soda site only has one option to join their newsletter. This option is featured on the lower part of the homepage. 73% of the Scotch & Soda participants clicked there, while the rest of the brand’s participants clicked elsewhere, especially on images.

Round 2 goes to H&M!

Tree Test

Good information architecture is necessary for providing a good search experience. To evaluate the findability, labeling and organisation of the contending sites’ structures, we placed participants in a plausible scenario: “Let’s say you recently bought a wool jersey from this brand and want to know more on how to care for your new purchase”

Participants were shown the site’s customer support menu structure as a list of topics and asked to select the area where they think they would find information on how to take care of the garment. Once finished they would indicate the task as completed. If they reached the requested information, the task would be rated as Success.

Both sites have pages for product care instructions. H&M hosts the “Garment Care” page under “Products & Quality” while Scotch & Soda has a main category called “Product Care” dedicated to that purpose.

On the H&M site 70% of the participants chose the correct option, while the rest of the participants either chose “Sustainability” (20%), “Download App” (3%) or “H&M Search Guide” (7%).

On the Scotch & Soda site 83% of the participants chose the correct options, and the rest of the participants chose either “Payments” (3%), “Delivery” (7%) or “Ordering” (3%) under the “Your Order” category, or “Women”(3%) under the “Size Guide” option.

Round 3 goes to Scotch & Soda!

Online Task-Based Test

We asked participants to complete a basic task a visitor would usually do on either of these sites: Search and find a piece of clothing.

We made participants pretend they were shopping for a present for their 12-yr old nephew and showed them a picture of a boy’s denim shirt sold on each store. We asked participants to find the shirt and reach its details page. Both of the shirts were unique to the store and located under the Boys or Kids sections.

example

To evaluate the success of the task, validation through a URL was used so that only participants who reached the correct page would have the task rated as success and subsequently be redirected to further questions.

A Non-Success result meant that the participant decided to abandon the task.

So, how did the two challengers do?

Results show that slightly more participants were able to find the requested item on the Scotch & Soda site.

Most of the participants were able to find the requested shirts and only a few decided to abandon the task. When asked their reasons for leaving, this is what they said:

  • “It is very hard to find a specific shirt. There are too many shirts to choose from.” (H&M)

 

  • “I would not search it for myself and I have no idea where to go (to find it).” (Scotch & Soda)

 

Ease of Use

After the task we asked all the users who were able to complete the task successfully to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy. This is what they said:

Results show that more participants were able to complete the task on the Scotch & Soda site, as well as finding it easier to complete too.

Round 4 goes to Scotch & Soda!

Problems & Frustrations

After the study, we asked all participants if they encountered any problems or frustrations and their responses are here:

In general most of this week’s participants did not encounter any problems or frustrations. From those who did find problems while on the sites, they reported “too many clicks to find information” or “site was slow” were the most mentioned.

Brand Perception

Now, let’s talk about perceptions and find out what participants were really thinking about their experience. We asked participants to rate the perception they had of the challenging brands before and after their experience on sites.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. In the pre-test survey we also included the option for participants to say they’re not familiar with the brand.

Before participants interacted with the websites, H&M had a brand perception rating of 5.6 and Scotch & Soda a 5.5 out of 7. H&M had only 3% of users saying they were not familiar with the brand, while half of the Scotch & Soda participants said they were unfamiliar with the brand.

After the participants completed the different tasks, we asked if their brand perception changed. For H&M, brand perception increased 0.2 points while the Scotch & Soda brand perception slightly decreased 0.1 points.

Net Promoter Score (NPS)

As the final part of the study, once the participants have interacted with the sites, we asked everyone to rate how likely it was they would recommend the brands’ sites to friends, family or colleagues.

The results show that more people would be comfortable recommending H&M based on their experience with their homepage.

And the winner is…

This week both contenders performed well. The Screenshot click test, Tree test, and Navigational Task test were pretty close. Participants’ perceptions had a big influence on the final decision, and with a better first impression, NPS and brand perception, this week’s winner is H&M!