The Task-Based UX Benchmark Study
UserZoom ran a quick task-based UX benchmark study between Zillow and Redfin, two well-known real estate websites, to compare the experience of finding monthly mortgage information online.
- We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
- We split participants equally between each website so that 30 went to Zillow and 30 went to Redfin
- The Online Task: You found your dream home and want to know how much the estimated monthly mortgage will be. From the homepage, navigate to where you would be able to find and note the estimated monthly mortgage for the property at 3633 Webster St in San Francisco, CA.
- We validated the tasks via a validation question on the monthly mortgage amount
Let’s Get Ready to Rumble!
This group of participants found Redfin’s homepage to be Organized, Easy and Welcoming but found Zillow’s homepage to be more Trustworthy, Informational, Helpful and Overwhelming.
Screenshot Click Test
We split the participants evenly and asked where they would click if they were looking for an affordability calculator.
24% of participants correctly clicked on Affordability Calculator under more valuable tools and info.
23% of participants correctly clicked on Tools. You can see that there was interest in the “how much you would save” slider.
We split the participants evenly and asked where they’d be able to find a real estate agent in their area.
We asked participants to find the estimated monthly mortgage for the property at 3633 Webster St in San Francisco, CA. In order to validate whether users were successful at the task we validated the task by asking them how much the monthly mortgage payment would be.
If users answered correctly they were labeled as Success.
Non Success meant that a user either Abandoned the task due to difficulties with the website or chose an incorrect answer, which we labeled as Error.
Users on Zillow’s site were 16% more likely to successfully complete the task than Redfin’s users.
On average, successful users on Zillow’s site were able to find the mortgage information in less time, pageviews and clicks than the successful users on Redfin.
Zillow user session
Redfin user session
Users were more likely to err than abandon on Zillow’s site, meaning they chose an incorrect answer rather than give up. When we watched the videos it was apparent that people were finding other prices that were on the page, such as rental price. Users on Redfin’s site were also more likely to err than abandon. It seemed that their users were unable to see where on the page the mortgage information was located as the icon was small and not being called out.
Zillow user session
Redfin user session
Ease of Use
After the tasks we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.
Problems & Frustrations
We asked the users which, if any, of the following problems they encountered while on the site.
We asked participants to rate their perception of the brand before and after their experience with the site.
For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they weren’t familiar with the brand before the task.
Both brands started with the same brand perception score, and both saw a slight decrease in their brand perception after participants interacted with the site. Zillow, however, ended with a slightly higher brand perception score.
Net Promoter Score (NPS)
After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.
This week’s group of participants were more likely to recommend Zillow’s site to friends, family and colleagues than Redfin’s site.
The winner of this week’s usability battle is Zillow! Both sites performed well, but in the end, more participants were able to successfully complete the online task on Zillow’ site more efficiently and with less abandons. Zillow also saw a higher ease of use score and Net Promoter Score.