UX Battle of the Week: Pandora vs SiriusXM
The Task-Based UX Benchmark Study
UserZoom ran a quick task-based UX benchmark study between Pandora and SiriusXM, two well-known online music providers, to compare the experience of registering a new account, finding information about advertising online as well as purchasing gift cards from their site.
- We ran the unmoderated remote task-based benchmark study with 60 in-the-wild users on their own devices over the course of a single day
- We split participants equally between each website so that 30 went to Pandora and 30 went to SiriusXM
- The Online Task: Find the page where you would be able to purchase a gift card
- We validated the task via URL validation
Let’s Get Ready to Rumble!
We showed all 60 participants an image of Pandora’s and SiriusXM’s homepage and asked them to choose which site they associated with several UX attributes.
Pandora’s minimalist approach to their homepage won overwhelmingly in regards to being seen as Easy. SiriusXM’s homepage swept the remaining positive, and negative, categories – Informational in particular. Interestingly, both websites were tied when it came to being seen as Trustworthy.
Screenshot Click Test
We split the participants evenly and asked where they would click if they wanted to register a new account.
83% of participants successfully clicked on an area where they would be able to register a new account. The vast majority, 73%, clicked on Sign Up in the middle of the page while 10% clicked on Sign Up in the header.
17% of participants successfully clicked on an area where they would be able to register a new account. 10% clicked on My Account in the header while 7% clicked on the Get Started tab. Of the outside areas participants erroneously clicked, 37% chose Sign Up For a Trial.
We asked participants where they felt they would be able to find information about advertising their business to listeners.
80% of participants successfully chose Advertising, with 73% of participants getting it right on their first choice. Of the outside areas participants selected, a majority (13%) chose Businesses.
14% of participants successfully chose Advertise With Us, with 3 out of the 4 successful participants getting it right on their first choice. A vast portion of participants, 46%, felt that How Can We Help? would be the correct area.
Online Task Outcomes
In order to validate whether users were successful at finding the gift cards we validated the task via a URL validation. If users got to the correct page, they were labeled as Success.
Non Success meant that a user either Abandoned the task due to difficulties with the website or said they were done before reaching the correct page, which we labeled as Error.
Users were 20% more successful on Pandora’s site than on SiriusXM’s.
Not only were users more likely to be successful on Pandora’s site, they were also able to complete the task in far less time, less page views and less clicks.
Pandora user session
SiriusXM user session
Non successful users on both sites were far more likely to abandon the site than to make an error. For both sites, users tended to abandon after not being able to navigate to the correct section and not finding the right page after using the search function.
Pandora user session
SiriusXM user session
We split the participants equally between brands and asked them to rate their perception of the brand before and after their experience with the site.
For the rating scale: 1 = Very Negative, 4 = Neutral, and 7 = Very Positive. We also included the option for participants to say they’re not familiar with the brand.
Brand Perception Pre-Task
Both sites started off with positive ratings – 5.7 out of 7 for Pandora and 5.2 out of 7 for SiriusXM.
Brand Perception Post-Task
After participants interacted with Pandora, the overall brand perception dropped down a fraction of a point to 5.6 out of 7. 60% of participants gave them the highest scores possible.
SiriusXM saw a greater drop in their score, down to a 4.3 out of 7. 20% of users gave them the highest scores.
Ease of Use
After the task we asked all the users to rate how easy or difficult it was to accomplish, with 1 = Very Difficult, 4 = Neutral and 7 = Very Easy.
Pandora’s users felt the website was easy to use, with 80% of participants giving it the highest ease of use scores for an overall rating of 6 out of 7.
SiriusXM’s users felt the task was difficult to accomplish on the site, ending with an average ease of use rating of 2.8 out of 7. 10% of users gave SiriusXM’s site the highest ease of use scores possible.
Problems & Frustrations
We asked the users which, if any, of the following problems they encountered while on the site.
Net Promoter Score (NPS)
After participants interacted with the sites we asked them to rate how likely it was they would recommend them to friends, family or colleagues.
Pandora’s users more likely to recommend the site to friends and family than SiriusXM’s users.
The winner this week between the battle of online music providers is Pandora for having higher success rates in regards to registering a new account, finding out how to advertise to listeners, and our task on finding the page for gift cards. Pandora also had higher brand perception, ease of use, and net promoter scores.