The Task-Based Benchmark Study

UserZoom ran a quick task-based benchmark study between LG and Sony, two well-known electronics manufacturers, to compare the experience of finding information about the latest televisions on their websites.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to LG and 30 went to Sony
  • The Task: Find the price of the 55”4k Ultra HD TV that’s between $1,000-$2,000
  • We validated the task by asking what the price of the TV was

 Let’s Get Ready to Rumble!

Before participants visited the websites, we showed all 60 participants an image of LG’s and Sony’s homepage and asked them to choose which site they associated with several UX attributes.

sonyldmonitors

Screen Shot 2015-11-02 at 12.29.15 PM

This week’s group of participants preferred the look of LG’s homepage over Sony’s due to the fact that Sony’s homepage had a longer scroll time to get to the bottom whereas LG has a more condensed homepage that took less time to navigate.

As we’ve seen in previous Battles of the Week, Sony’s scroll heavy and image laden homepage was initially seen as Lively but also Overwhelming. LG’s more condensed homepage garnered more votes for being Organized and Informational in particular. Despite these differences in initial opinions, both sites were fairly evenly split in regards to being seen as Welcoming.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand. Click on the image to enlarge.

Screen Shot 2015-11-02 at 1.32.02 PM

Both brands started off with a high brand perception rating before participants interacted with their sites, coming in evenly with a 5.7 out of 7. LG had 63% of their users give them the highest two ratings while Sony had slightly more, 66%, of users give them the highest ratings.

Neither brand had participants say they weren’t familiar with the brand.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Click on the image to enlarge.

Screen Shot 2015-11-02 at 1.52.56 PM

Both brands saw a slight decrease in brand perception after participants interacted with their sites. LG only saw a slight decrease down to 5.5 out of 7 with a majority of participants, 57%. still giving them the highest two ratings possible. Only 17% of participants said they now had a negative perception of the brand.

Sony saw a slightly more noticeable decrease in brand perception, dropping down to 5.1 out of 7. Despite having the same amount of participants give them the highest two ratings possible as LG, also 57%, Sony saw an increased amount of participants who rated their perception of the brand negatively (20%).

Task Outcomes

In order to validate whether users were successful at finding the pricing information, we asked participants what the price was for the 55”4k Ultra HD TV. If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or thought they had found the correct information but chose the incorrect answer, which we labeled as Error.

Screen Shot 2015-11-02 at 2.14.15 PM

Success

Participants were 17% more likely to successfully find the correct price of the television on Sony’s site than on LG’s.

Screen Shot 2015-11-02 at 2.17.49 PM

Despite having more people successfully find the price of the television on Sony’s site, LG’s successful participants were able to find the price in less time and with less page views than Sony’s participants.

LG user session (with audio)

Sony user session (with audio)

Non Success

Screen Shot 2015-11-02 at 2.49.51 PM

Participants who were not successful at finding the correct price on both sites were more likely to find an incorrect price than abandoning the task altogether. What’s interesting is that participants who erred (found the wrong price) on both sites ended up spending almost the exact amount of time searching the site and viewing the same number of pages – 2 minutes and 4 pages.

Participants who ended up abandoning the task ended up searching slightly longer on LG’s site than on Sony’s before giving up.

LG user session (video only)

Sony user session (video only)

Ease of Use

After the task we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy. Click on the image to enlarge.

Screen Shot 2015-11-02 at 3.03.07 PM

Sony’s users rated the ease of finding the pricing information on the site at 4.9 out of 7. Sony had 50% of their participants give them the two highest ease of use rankings while 17% gave them the lowest.

Despite having a 17% lower success rate overall, LG was rated slightly higher with an average ease of use score at 5.3 out of 7. LG had 64% of participants give them the highest two rankings while 17% gave them the lowest rankings.

User Satisfaction

After the task we asked all the users to rate how satisfied they were with the site, with 1 = Very Unsatisfied, 4 = Neutral and 7 = Very Satisfied. Click on the image to enlarge.

Screen Shot 2015-11-02 at 3.12.58 PM

Participants from both sites rated their satisfaction rate as being fairly high after their online experience. LG ended with an average satisfaction rating of 5.3 out of 7, with 50% of their users giving them the highest two ratings possible.

Sony ended with a slightly higher average satisfaction rating of 5.4 out of 7 with 67% of their users giving Sony the highest two ratings possible.

Visual Appeal

After the task we asked all the users to rate how visually appealing the site was, with 1 = Very Unappealing, 4 = Neutral and 7 = Very Appealing. Click on the image to enlarge.

Screen Shot 2015-11-02 at 4.33.02 PM

LG’s average visual appeal rating ended up at 5.7 out of 7, with 60% of users giving them the highest two ratings. Impressively, LG received zero negative ratings from their users.

Sony, despite having 7% of users rate them negatively, still ended with a slightly higher overall rating of 5.8 out of 7. 67% of Sony’s users gave them the highest two ratings for visual appeal.

Site Layout & Organization

After the task we asked all the users to rate how organized the site’s layout was, with 1 = Very Unorganized, 4 = Neutral and 7 = Very Organized. Click on the image to enlarge.

Screen Shot 2015-11-02 at 4.50.48 PM

Both sites tied for an average rating of 5.4 out of 7, but Sony received the top two highest ratings from 64% of their users versus LG’s 57%.

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

LG

Screen Shot 2015-11-02 at 5.05.30 PM

If users chose Other they were given an open-ended question to describe their problem or frustration. The two users who chose this on LG said:

  • “It wouldn’t direct me right to a store, so that I couldn’t comparison shop.”
  • “The header image loaded after I began my search, resulting in the page kicking down on me and having to scroll down to resume.”

Sony

Screen Shot 2015-11-02 at 5.15.28 PM

If users chose Other they were given an open-ended question to describe their problem or frustration. The two users who chose this on Sony said:

  • “There was too much going on, way too many pictures.”
  • “My search didn’t return any matches.”

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues. Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10. Click on the image to enlarge.

Screen Shot 2015-11-02 at 5.23.40 PM

In our seven years of experience, we have found that the average NPS differs by industry and that it’s not uncommon for brands to have negative Net Promoter Scores.

This week’s groups of participants were split fairly evenly about how they felt on both sites, but in the end, Sony managed to secure one more Promoter and one less Passive than LG for a winning NPS of -20%.

Conclusion

Sony is this week’s winner for having a higher percentage of users successfully find the pricing information, more users say they would recommend their site, and for having the highest percentage of users give them the top ratings in visual appeal, organization & layout, and satisfaction ratings.

It should be noted that LG performed admirably and wasn’t far behind in most of the rankings – even pulling ahead in regards to ease of use and with their successful users completing the task in less time than Sony.


Want to learn more about
UX Benchmarking with UserZoom?

xperience360-button