The Task-Based Benchmark Study

UserZoom ran a quick task-based benchmark study between eHarmony and Match.com, two well-known online dating sites, to compare the experience of finding information about gifting subscriptions to friends or family members.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to eHarmony and 30 went to Match.com
  • The Task: Find the page with information on how to gift a subscription
  • We had participants self-validate by asking whether they were able to accomplish the task

Let’s Get Ready to Rumble!

Before participants visited the websites, we showed all 60 participants an image of eHarmony’s and Match.com’s homepage and asked them to choose which site they associated with several UX attributes.

eharmony-matchcom-monitors-botw

Screen Shot 2015-11-20 at 11.07.54 AM

This week was another example of a condensed homepage (Match.com) competing against a scroll-heavy homepage (eHarmony) in regards to visitor first impressions. As is often the case, users saw the scroll heavy homepage as being Overwhelming while the minimalist homepage was seen as being much more Easy to use.

What’s different this week is that both websites were incredibly close in the Organized, Welcoming, Trustworthy and Helpful categories. Another difference from previous Battles of the Week is that the condensed homepage was viewed as being more Lively than the scroll heavy homepage.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand. Click on the image to enlarge.

Screen Shot 2015-11-20 at 11.42.15 AM

eHarmony had a very even range of respondents from the lowest to the highest possible ratings, ending with an average rating of 4.5 out of 7. Match.com had a slightly higher average rating of 5.4 out of 7 given that fewer participants had negative feelings associated with the brand. Neither company had any respondents say they were unfamiliar with the brand.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Click on the image to enlarge.

Screen Shot 2015-11-20 at 12.09.54 PM

After participants interacted with the website, eHarmony’s average brand perception rating increased slightly to 4.7 out of 7 with 40% of participants giving them the highest ratings possible.

Match.com saw a slight decrease in brand perception after they interacted with the site, dropping down to 5.1 out of 7 with 50% of participants giving them the highest ratings possible.

Task Outcomes

In order to validate whether users were successful at finding the gift subscription information we asked participants whether they were able to find the answer. If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or self reported they were not able to, which we labeled as Error.

Screen Shot 2015-11-20 at 12.54.23 PM

Success

Participants were 73% more likely to successfully find the gift subscription information on eHarmony’s site than on Match.com’s.

Screen Shot 2015-11-20 at 1.04.37 PM

Despite having vastly different success rates, the successful participants on both sites ended up spending similar amounts of time searching the sites. Both sites had the same average page views required to find the information as well but the main difference is in the clicks.

After reviewing the videos, it was clear that the reason for the clicks was due to the fact that participants were searching Match.com’s site and clicking repeatedly on the Gift Subscription link which was taking them to the page for redeeming a gift subscription. It wasn’t until participants manually searched for Gift Subscriptions in the search bar or on FAQ page that they found the information they were after.

eHarmony user session (with audio)

Match.com user session (with audio)

Non Success

Screen Shot 2015-11-20 at 1.47.08 PM

eHarmony’s non successful users were split evenly between Abandoning the task entirely and saying they weren’t able to find the information. Participants on Match.com were much less likely to abandon the task then to say they weren’t able to find the information.

While the distinction in this case is largely one of semantics, it’s still important to note that people who erred were people who felt like they had made some progress on the site but when asked if they were able to actually find the information they realized they had not. People who abandoned felt as if they weren’t making progress on the site, which was evident when we filtered the results.

eHarmony user session (video only)

Match.com user session (video only)

Ease of Use

After the task we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy. Click on the image to enlarge.

Screen Shot 2015-11-20 at 2.13.38 PM

eHarmony’s users ranked the overall ease of use of the site at 5 out of 7, with 70% of participants giving eHarmony a positive rating. Match.com’s overall ease of use ranking was slightly lower at 4.7 out 7. Despite having a majority of participants actually fail the task, Match.com still managed to have 60% of their users give them a positive ease of use rating likely due to the fact that many were able to quickly find the page for redeeming a gift subscription if not the page to obtain one.

User Satisfaction

After the task we asked all the users to rate how satisfied they were with the site, with 1 = Very Unsatisfied, 4 = Neutral and 7 = Very Satisfied. Click on the image to enlarge.

Screen Shot 2015-11-20 at 2.32.54 PM

Overall participants were more satisfied on eHarmony’s site than on Match.com’s, with an average user satisfaction rating of 5.1 out of 7 versus Match’s 4.4.

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

eHarmony

Screen Shot 2015-11-20 at 2.46.22 PM

If users chose Other they were given an open-ended question to describe their problem or frustration. The three users who chose this for eHarmony said:

  • “Had to search for the term ‘gift’ in order to find it”
  • “Too much text to scroll”
  • “There was too much information to scroll through”

Match.com

Screen Shot 2015-11-20 at 2.49.40 PM
If users chose Other they were given an open-ended question to describe their problem or frustration. The two users who chose this on Match.com said:

  • “It told me to sign up or log in. It wouldn’t let me in the site without it”
  • “It would be nice to be able to browse before having to enter information”

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues. Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10. Click on the image to enlarge.

Screen Shot 2015-11-20 at 2.53.43 PM

In our seven years of experience, we have found that the average NPS differs by industry and that it’s not uncommon for brands to have negative Net Promoter Scores.

This week’s groups of participants were more likely to recommend eHarmony’s site than Match.com’s site to friends or family.

Conclusion

The winner this week is eHarmony for having a higher percentage of users successfully find the gift subscription information, as well as having higher user satisfaction, ease of use, and NPS rankings. Match.com still performed well, with users not far behind in each of the categories.


Want to learn more about
UX Benchmarking with UserZoom?

xperience360-button