The Task-Based UX Benchmark Study

UserZoom ran a quick task-based UX benchmark study between Kaplan and The Princeton Review, two well-known online education sites, to compare the experience of  purchasing a SAT prep course.

  • We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
  • We split participants equally between each website so that 30 went to Kaplan and 30 went to The Princeton Review
  • The Online Task: Add the self-paced SAT prep course to your shopping cart and proceed to checkout
  • We validated the task via a URL validation on the first page of the checkout

conclusion anchor button

Let’s Get Ready to Rumble!

We showed all 60 participants an image of Kaplan’s and The Princeton Review’s homepage and asked them to choose which site they associated with several UX attributes.

Kaplan and The Princeton Review homepages - UX Battle of the Week

Kaplan vs Princeton Review brand attributes - UX Battle of the Week

Between the two sites, users felt that Kaplan was more Organized, Easy, Welcoming, Helpful and Lively. The Princeton Review was seen as being more Informational and Trustworthy, but also was the clear pick for being seen as Overwhelming.

Screenshot Click Test

We split the participants evenly and asked where they would click if they wanted to register a new account.

Kaplan

Kaplan heatmap - UX Battle of the Week

60% of users correctly clicked on Sign-In in the header.

The Princeton Review

The Princeton Review heatmap - UX Battle of the Week

83% of participants correctly clicked on Register in the header.

Tree Test

We asked participants where they felt they would be able to find information about the majors colleges offer.

Kaplan

Kaplan tree test - UX Battle of the Week

Only 30% of users correctly chose Earn a Degree as the area where they’d be able to research majors at the colleges they’re interested in. The area that a majority (50%) of users chose instead, Our Programs, is actually referring to Kaplan’s partnership programs with specific universities.

The Princeton Review

The Princeton Review tree test - UX Battle of the Week

A vast majority of users, 93%, correctly chose Search Majors as the area where they’d be able to research majors at the colleges they’re interested in. Interestingly enough, the only other place users selected was under tutoring.

Online Task Outcomes

In order to validate whether users were successful at completing the process we validated the task via a URL validation on the checkout page. If users got to the correct page, they were labeled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or said they were done before reaching the correct page, which we labeled as Error.

Kaplan vs Princeton Review Task Success Ratios - UX Battle of the Week

Success

Users were 4% more successful on Kaplan’s site than on The Princeton Review’s.

Kaplan vs The Princeton Review efficiency ratios - UX Battle of the Week

Despite the similarity in overall success rates, there was a difference between the sites when we look at the average time on task and clicks. Overall, Kaplan’s users were able to complete the task in less time and with fewer clicks.

Kaplan user session

The Princeton Review user session

Non Success

Kaplan vs The Princeton Review non success rates - UX Battle of the Week

As you can see, while both sites had a similar number of non successful participants there was a vast difference between erring and abandoning between them. For Kaplan, the majority (80%) of non successful participants ended up clicking “I’m Finished” early when they were shown a cart overview page which resulted in an error.

The Princeton Review, however, had the bulk of their non successful participants (91%) abandon the site. When we reviewed the user session videos it became apparent that users were having a difficult time finding the location for SAT prep courses.

Kaplan user session

The Princeton Review user session

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand.

Brand Perception Pre-Task

Screen Shot 2016-03-23 at 12.07.44 PM

Before participants interacted with the site both Kaplan and The Princeton Review started out with fairly average brand perception scores. Around a third of participants on both sites reported they had no prior knowledge or experience with the brands.

Brand Perception Post-Task

Screen Shot 2016-03-23 at 12.13.57 PM

After participants interacted with the site Kaplan’s brand perception dropped down slightly to 5 out of 7 from a 5.2. The Princeton Review’s brand perception rose from a 4.9 previously to a 5.1 out of 7.

Ease of Use

After the task we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.

Kaplan vs The Princeton Review ease of use - UX Battle of the Week

Kaplan’s users felt that the task was fairly easy to accomplish, giving it an overall rating of 4.9 out of 7.

The Princeton Review’s users felt the task was somewhat difficult to accomplish on the site. 30% of users rated the task as “Very Difficult” to accomplish which brought down their ease of use score to a 3.8 out of 7.

Problems & Frustrations

We asked the users which, if any, of the following problems they encountered while on the site.

Kaplan

Kaplan user problems and frustrations - UX Battle of the Week

Participants who chose “Other” were given an open-ended follow up question to explain. This is what Kaplan’s users said:

  • “There were a lot of SAT testing options – not sure which one was the right one”
  • “The site was way too big, made it hard to find anything”

The Princeton Review

The Princeton Review user problems and frustrations - UX Battle of the Week

Participants who chose “Other” were given an open-ended follow up question to explain. This is what The Princeton Review’s user said:

  • “In general, the site just wasn’t very good.”

For more clarity we filtered the data to zoom in on this particular respondent and found they had abandoned the task, and subsequently clicked all the problems and frustrations other than the site was slow and an error was encountered.

Net Promoter Score (NPS)

After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.

Kaplan and The Princeton Review NPS - UX Battle of the Week

Despite the differences in user abandonment on the site and task success rates, The Princeton Review had a higher overall NPS from their participants.

Conclusion

Winner - Kaplan vs The Princeton Review - UX Battle of the Week

This week was extremely close and both sites performed well. However, despite The Princeton Review having better attitudinal and brand perception ratings from their users, the winner in this week’s usability battle between online education sites is Kaplan for having a slightly higher success rate in the task, far less user abandonments, and for users being able to complete the task more efficiently.


Want to learn more about competitive
UX Benchmarking with UserZoom?

personalized demo