The task-based mobile benchmark study.

UserZoom ran a quick task-based mobile benchmark study between Priceline and Kayak, two well-known travel search engines, to compare the mobile experience of obtaining hotel information.

– We ran the unmoderated remote task-based benchmark study with 52 in-the-wild users on their own mobile devices over the course of a single day

– We split participants equally between each website so that 26 went to Priceline and 26 went to Kayak

– The Task: Find the address of the Omni Hotel in San Francisco, CA

– We validated the task by asking them what the address is

Let’s Get Ready to Rumble!

Before participants visited the mobile sites, we showed all 52 participants an image of Priceline’s and Kayak’s homepage and asked them to choose which site they associated with several UX attributes:


Given that the mobile homepages are so similar, we gave participants the option to say they had no preference between the two. Participants still ranked Kayak higher than Priceline in almost all categories. The most notable exception is in the Overwhelming category, in which a vast majority of participants said they had no preference between the two; however, Priceline was ranked more overwhelming than Kayak amongst the participants who did have a preference.

Brand Perception

We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the mobile site.

Brand Perception Pre-Task

In order to get a feel for the brand before their online experience, we showed participants an image of the homepage.

For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive.

Screen Shot 2015-07-28 at 2.24.41 PM

Kayak’s brand perception started fairly high before participants interacted with the mobile site, with no negative ratings and over half of participants rating Kayak a 6 or above out of 7. Priceline also started fairly high, with a majority of participants rating it at just above neutral with a 5.

Brand Perception Post-Task

After participants interacted with the mobile sites we again asked them to rate their perception of the brand with the same rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive.

Screen Shot 2015-07-28 at 3.57.28 PM

Both brands saw a decrease in their average brand perception rating after participants interacted with their mobile sites but where Kayak’s average only dropped by 0.4 points, and actually saw an increase in participants who gave them the highest rating, Priceline saw a drastic decrease of 1.2 points with 14 out of 26 users rating the brand negatively versus Kayak’s 4 out of 26.

Task Outcomes

In order to validate whether users were successful at finding the address of the Omni Hotel, we asked them what the address was after the task. If users answered correctly they were labelled as Success.

Non Success meant that a user either Abandoned the task due to difficulties with the website or thought they had found the correct information but chose the incorrect answer, which we labelled as Error.

Screen Shot 2015-07-28 at 4.15.58 PM


Participants were almost twice as likely to successfully find the address for the Omni Hotel on Kayak’s site than on Priceline.

Screen Shot 2015-07-28 at 4.48.10 PM

On average it took Kayak’s successful users almost a minute less to find the information than it did on Priceline’s website with slightly less clicks needed.

Kayak user session (with audio)

Priceline user session (video only)

Ease of Use

We asked users who successfully completed the task to rate how easy or difficult it was to accomplish, with 1 = Difficult and 5 = Easy.

Ease of use both

What’s interesting is that while Kayak had a majority of it’s participants successfully complete the task, and in less time, participants who were successful on Priceline’s mobile site actually rated the site slightly easier to use than Kayak’s.

Non Success

Both sites saw more participants Abandon the task rather than Error, meaning more users gave up instead of finding incorrect information. What sets the sites apart is the number of participants that were not successful at the task.

Kayak only had 3 out of 26 Non Successful participants while Priceline saw over half of it’s participants, 14 out of 26, fail the task. Of the 14, only one person didn’t Abandon the task.

Screen Shot 2015-07-28 at 5.02.44 PM

Watching the user sessions that abandoned the task on the Priceline site, it is apparent that users were having technical difficulties using the mobile site. On Kayak, it seemed that the participants simply couldn’t find the Omni Hotel.

Priceline User Session with Audio

Kayak user session (video only)

Net Promoter Score (NPS)

After participants interacted with the mobile sites, we asked them to rate how likely it was they would recommend them to friends, family or colleagues. Users who rate this likelihood as low, 0-6 on the rating scale, are labelled as Detractors. Users that choose 7 or 8 are labelled as Passives and Promoters are users that rate the likelihood as 9 or 10.

NPS both

In our seven years of experience, we have found that most brands have negative Net Promoter Scores and that the average NPS differs by industry. What’s noticeable about these results are that Kayak’s users are much more likely to recommend the mobile site than Priceline, likely due to the technical difficulties users faced while on Priceline’s mobile site.


Overall this group of participants preferred Kayak’s mobile experience to Priceline’s, with Kayak’s users being much more likely to successfully complete the task and doing so in less time than on Priceline. Priceline unfortunately seemed to have technical issues while more than half of its participants were using the mobile site, thus forcing a high abandonment rate.

A takeaway for both mobile sites would be the ability to search and filter hotels by name as well as by price or ratings from the main search page. This would cut down on the need for users to scroll through the hotel lists, allowing users to spend less time searching and more time booking.

 Craving even more great content?