What is more important to a UX researcher:

  1. The ability to test hundreds of users remotely and collect a lot of quantitative data, or
  2. Testing just a few users in a lab while seeing their actual online behavior?

Most researchers would agree that you need both to truly understand the actual experience that users had on your site.

Recording the users session by capturing a screen video gives researchers visibility into users’ actual online behavior while collecting quantitative UX data. With this feature, you now get both quantitative and qualitative insights, including task success and abandon rates, time on task, number of clicks, clickstreams or paths, click heatmaps, NPS, video clips of each user session, comments, and so much more!

We put together a quick pilot study to demonstrate the value of combining qualitative and quantitative results. For this pilot study we used Branders.com, world’s largest online seller of promotional items & corporate gifts. Please note that we recruited a small sample of non-target users. As such, the results of this study should not be regarded as conclusive.

Overview of the study

We asked 30 people to carry out two small tasks on Branders.com and answer some follow-up questions (please note that for a real study we would typically recruit at least 200 users. However for the purposes of this pilot, 30 users were sufficient). As they were attempting their tasks, UserZoom was tracking all those UX metrics while also recording their online behavior.

Task 1: Finding the price of an organic lip balm

For our first task, we asked participants to find the total price of ordering 650 organic lip balms (without shipping and tax). We also created an account on Branders.com and provided participants with the login and password in case if they needed that information.

Highlights of the findings

The success rate for this task was 23%, which suggests that participants had a hard time locating the total price for ordering 650 lip balms.  This also becomes evident when looking at the time it took users to complete the task. On an average, it took participants 3 minutes to complete the task- with 8 minutes being the maximum time and 1.2 minutes the minimum.

The low success rate can be attributed to the following issues: difficulty of navigation, forced registration/login, and confusion about the actual cost. Let’s go over these issues one by one.

Difficulty of Navigation

Here is some feedback that our participants gave about the difficulty of navigation:

“I found it difficult to find stuff based on the categories provided. Specifically, for both of the tasks, had there not been scrolling ads for those particular products, it would have taken a lot longer to find them. The lip balm in particular was difficult, as there was no obvious “beauty” category. I believe it was under “Health and …” or something like that.”

“Very hard to navigate around unless you had a specific search term”

“Not at all easy to use”

Indeed, below is one of the videos that captured a user having trouble finding the right category for lip balms. The first lip balm that the user clicked on was an ad but that was not the right product.

Forced registration/login

To get the quote for a certain product, you had to be either logged in into your account, or you had to register as a new user. Even if you were already a registered user but were not logged in into your account, you were forced to register as a new user once you clicked on “Get Quote” button. Based on the response, participants were not particularly excited about this.

In one of the post-task questions we asked participants what they liked the least about the Branders website. Here is a selection of the notes that participants gave about the forced registration/login in order to see the total price of a product.

“…that you have to log in to see the price”

“Couldn’t check what the bulk price was without logging in”

When reviewing the recorded user sessions we noticed that the vast majority of users tried to avoid filling out any forms. They proceeded to filling out a form only after they ran out of all other options.

In the video below, we see one of the users struggling with the “Get Quote” form. He tries to enter existing account information into a form intended for new unregistered users. Only after seeing the same error message for the third time was the participant able to click on the login now link and enter his account information. This could have been avoided had there been a separate option for registered and non-registered users.

Confusion about the actual cost

38% of the participants got to the correct final page with the price of the product but were not able to report the total price of ordering 650 organic lip balms before shipping and tax. They reported the total price as $454.99. However, they did not notice that that price didn’t include the $50 decoration fee. The total price of $504.99 was displayed on the very top of the page.  As a result, 61% of the people who got to the last page (with the correct pricing info) reported the wrong price because they were looking at the wrong section of the page.

Task 2: Finding the price of one Carabiner Aluminum Bottle

For our second task, we asked participants to find the per item price of green carabiner aluminum bottles if they were to make a bulk order of 900 pieces.

Highlights of the findings

The success rate for this task was 40% and the mean time on task was 1:35 minutes.
Out of those who didn’t complete the task successfully, 17% abandoned the task and 83% answered it incorrectly.

So let’s take a closer look at what caused participants to abandon the task.

When we asked participants as to why they abandoned the task or were not able to complete it, one of them responded that the site would not let him add anything to his cart. Two others said that they didn’t find the product.

When we look at the video below, we see that this participant abandoned the task after seeing an error message.

The next video shows a user searching for a “carabiner aluminium bottle”. His search didn’t yield any results. The search failed because the word “aluminium” was spelt without an “i” on the website. Both spellings are correct; it is just that “aluminum” is how we spell the word in the US and “aluminium” is the British spelling. One way to avoid such issues is to have an auto complete feature in place. Auto complete would work with the words that are present in the database. Yet another way to solve this is to have spelling correction based on the vocabulary present in the text of the website.

In the very last question of this study we asked participants how likely they would recommend Branders website to their friend, family or colleague on a scale from 0 to 10. This question is called Net Promoter and is used by many companies as a measure of customer loyalty. There are many factors that contribute to customer loyalty and website usability is definitely one of them. In fact, a study conducted by Measuring Usability suggests that improving usability will have a substantial impact on your customer loyalty.

Net Promoter Score is calculated by subtracting the percent of Detractors (0-6) from the percent of Promoters (9-10). Our study showed that the NPS for Branders was -43%. This means that there are 43% more users detracting than promoting Branders.com. This doesn’t sound that great, however you still need an industry benchmark to make sense of your Net Promoter Score.

Conclusion

Remote testing just got a whole lot more powerful! UX practitioners have been using remote unmoderated usability testing mostly for collecting valuable quantitative data, something that was not possible to do in a lab. Now,combining remote unmoderated usability testing with video session replay gives you the best of both worlds. With this approach companies can gather the necessary quantitative as well as qualitative data to better optimize online user experience on their site.