The Task-Based UX Benchmark Study

UserZoom ran a quick task-based UX benchmark study between Zillow and Redfin, two well-known real estate websites, to compare the experience of finding monthly mortgage information online.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to Zillow and 30 went to Redfin
  • The Online Task: You found your dream home and want to know how much the estimated monthly mortgage will be. From the homepage, navigate to where you would be able to find and note the estimated monthly mortgage for the property at 3633 Webster St in San Francisco, CA.
  • We validated the tasks via a validation question on the monthly mortgage amount

conclusion anchor button

Let’s Get Ready to Rumble!

We showed all participants an image of Zillow’s and Redfin’s homepage and asked them to choose which site they associated with several UX attributes.
Screen Shot 2016-07-22 at 5.07.40 PM

Screen Shot 2016-07-20 at 3.15.12 PM

This group of participants found Redfin’s homepage to be Organized, Easy and Welcoming but found Zillow’s homepage to be more Trustworthy, Informational, Helpful and Overwhelming.

Screenshot Click Test

We split the participants evenly and asked where they would click if they were looking for an affordability calculator.

Zillow

Zillow heatmap

24% of participants correctly clicked on Affordability Calculator under more valuable tools and info.

Redfin

Redfin heatmap

23% of participants correctly clicked on Tools. You can see that there was interest in the “how much you would save” slider.

Tree Test

We split the participants evenly and asked where they’d be able to find a real estate agent in their area.

Zillow

Screen Shot 2016-07-22 at 2.57.08 PM

Redfin

Screen Shot 2016-07-22 at 2.57.49 PMOnline Task Outcomes

We asked participants to find the estimated monthly mortgage for the property at 3633 Webster St in San Francisco, CA. In order to validate whether users were successful at the task we validated the task by asking them how much the monthly mortgage payment would be.

If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or chose an incorrect answer, which we labeled as Error.

Screen Shot 2016-07-22 at 3.59.08 PM

Success

Users on Zillow’s site were 16% more likely to successfully complete the task than Redfin’s users.

Screen Shot 2016-07-22 at 4.05.45 PM

On average, successful users on Zillow’s site were able to find the mortgage information in less time, pageviews and clicks than the successful users on Redfin.

Zillow user session

Redfin user session

Non Success

Screen Shot 2016-08-02 at 12.43.15 PM

Users were more likely to err than abandon on Zillow’s site, meaning they chose an incorrect answer rather than give up. When we watched the videos it was apparent that people were finding other prices that were on the page, such as rental price. Users on Redfin’s site were also more likely to err than abandon. It seemed that their users were unable to see where on the page the mortgage information was located as the icon was small and not being called out.

Zillow user session

Redfin user session

Ease of Use

After the tasks we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.

Screen Shot 2016-07-22 at 4.30.40 PM

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

Zillow

Screen Shot 2016-07-22 at 4.36.07 PM

Redfin

Screen Shot 2016-07-22 at 4.36.30 PM

Brand Perception

We asked participants to rate their perception of the brand before and after their experience with the site.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they weren’t familiar with the brand before the task.

Screen Shot 2016-07-22 at 4.45.24 PM

Both brands started with the same brand perception score, and both saw a slight decrease in their brand perception after participants interacted with the site. Zillow, however, ended with a slightly higher brand perception score.

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.

Screen Shot 2016-07-22 at 4.50.37 PM

This week’s group of participants were more likely to recommend Zillow’s site to friends, family and colleagues than Redfin’s site.

Conclusion

Screen Shot 2016-07-22 at 5.10.26 PM

The winner of this week’s usability battle is Zillow! Both sites performed well, but in the end, more participants were able to successfully complete the online task on Zillow’ site more efficiently and with less abandons. Zillow also saw a higher ease of use score and Net Promoter Score.


Want to learn more about competitive
UX Benchmarking with UserZoom?

personalized demo

128 Shares