The Task-Based Benchmark Study

UserZoom ran a quick task-based benchmark study between NRG and SolarCity, two well-known solar power providers, to compare the experience of finding information about leasing solar panels on their websites.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to NRG and 30 went to SolarCity
  • The Task: How many years would you have to lease to qualify for the $0 down option for home solar?
  • We validated the task by asking how many years they would have to lease in order to qualify for the $0 down option

Let’s Get Ready to Rumble!

Before participants visited the websites, we showed all 60 participants an image of NRG’s and SolarCity’s homepage and asked them to choose which site they associated with several UX attributes.

solarcitynrgbrands

Screen Shot 2015-10-20 at 3.06.43 PM

This week’s group of participants seemed to prefer NRG’s homepage across the board with only slight variations from category to category. As we’ve historically seen in previous Battle of the Weeks, Lively was closely connected with being Overwhelming. What’s interesting is that despite being seen as vastly different in almost all categories, both sites were very close in regards to being seen as Easy to use.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand. Click on the image to enlarge.

Screen Shot 2015-10-20 at 4.02.33 PM

Both brands had a relatively high percentage of participants start out not knowing anything about them – 37% for NRG and 27% for SolarCity. Aside from the participants who were unfamiliar with the brands, NRG had a high percentage of participants, 43%, give them the highest ratings while SolarCity only received 26%.

The other differentiators between the brand perception ratings was that SolarCity had 37% of participants say felt neutral versus NRG’s 7%, and the fact that SolarCity had 7% of participants say they had a negative perception of the brand while NRG had no negative ratings.

Brand Perception Post-Task

After participants interacted with the sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. Click on the image to enlarge.

Screen Shot 2015-10-20 at 4.27.54 PM

After participants interacted with NRG’s site 40% of participants gave them the highest ratings possible, down from 43%. What ended up dropping their overall brand perception down almost a full point was the fact that 20% of participants who interacted with their site rated them negatively.

SolarCity saw a much less drastic change in brand perception post-task from their participants, actually raising a little bit from 4.9 to 5 out of 7. Only 10% of participants gave them a negative score while the amount of participants who gave them the highest ratings rose to 30%.

Task Outcomes

In order to validate whether users were successful at finding the leasing information, we asked participants how many years they would have to lease in order to qualify for the $0 down option. If users answered correctly they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or thought they had found the correct information but chose the incorrect answer, which we labeled as Error.

Screen Shot 2015-10-20 at 4.48.40 PM

Success

Participants were more likely to successfully find the correct leasing information on SolarCity’s site than on NRG’s.

Screen Shot 2015-10-21 at 12.44.07 PM

On average it took successful participants slightly longer to find the leasing information on Solarcity’s site than on NRG’s site, and with slightly more clicks and page views required than on NRG’s site.

NRG user session (with audio)

SolarCity user session (with audio)

Ease of Use

We asked users who successfully completed the task to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy. Click on the image to enlarge.

Screen Shot 2015-10-21 at 11.33.53 AM

Despite having a 10% higher success rate overall, SolarCity’s successful users rated the ease of finding the leasing information on the site at 4 out of 7. SolarCity’s ratings were split fairly evenly, with 20% of users rating them the lowest and 20% rating them the highest.

NRG was rated slightly higher with an average ease of use score at 4.9 out of 7. NRG only had 6% of participants give them the lowest rankings while 36% gave them the highest rankings.

Non Success

Screen Shot 2015-10-21 at 12.04.04 PM

NRG’s non successful participants were split almost evenly between choosing an incorrect answer when it came to how long the lease was and abandoning the task, with slightly more participants abandoning over erring. For both sites, participants who gave up on the task spent just under 2 minutes searching but on NRG’s site they looked at twice as many pages on average.

SolarCity had a majority of their non successful participants err instead of abandon the task. What’s interesting is that participants who erred on SolarCity’s site ended up searching the site for almost twice as long as participants who erred on NRG’s site.

We asked participants on both sites who chose an incorrect answer how confident they were that they found the correct leasing information:

Screen Shot 2015-10-21 at 12.13.27 PM

NRG’s users weren’t so sure they had found the correct answer, split 50/50 on either side of neutral. SolarCity’s users, however, felt pretty confident that they had found the correct information, 43% going so far as to say they were extremely confident they had found the correct information.

NRG user session (video only)

SolarCity user session (video only)

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues. Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10. Click on the image to enlarge.

Screen Shot 2015-10-21 at 12.24.57 PM

In our seven years of experience, we have found that the average NPS differs by industry and that it’s not uncommon for brands to have negative Net Promoter Scores.

This week’s groups of participants were split fairly evenly about how they felt on both sites, but in the end, SolarCity managed to secure a few more Promoters and slightly less Detractors than NRG for a winning NPS of -30%.

Conclusion

For this group of participants, SolarCity turned out to be the winner in terms of having more users successfully find leasing information and whether or not participants would recommend the site and service to friends, family and colleagues. It should be noted though that NRG performed admirably and wasn’t too far behind in the aforementioned metrics, and actually had a slightly higher ease of use rating from their participants.


Want to learn more about
UX Benchmarking with UserZoom?

xperience360-button