We’re excited to announce a new results feature – Confidence Intervals!

That’s right, in our upcoming Results Beta we’re making it easier to make informed decisions about your data with the addition of automated confidence intervals for navigation tasks.

The size of these confidence intervals will automatically adjust based on the sample size of your study, and within these values you have the flexibility to choose from the most commonly used confidence levels: 80%, 90%, 95%. Of course, if you’d prefer not to have them included, you also have the option to turn them off altogether.

As Kuldeep Kelkar explains in his article on what sample size do you really need for UX research:

  • The confidence level is the amount of uncertainty you can tolerate
  • The confidence interval is the amount of error that you can tolerate

Lower margins of error requires a larger sample. Anyone making a big decision would want at least 90% confidence level and be able to measure things at +- 10% margins.

If you’re comfortable with confidence intervals, you don’t need to read any further. Just sit back, relax, and enjoy.

If you’re not entirely confident what confidence intervals are or if you’re terrified of mathematical principles – well, you’re not on your own. And frankly, it’s cool. We can take care of that for you on the user testing platform.

However if you would like to know more about these concepts, here’s a handy guide in which I rope in the help of a genuine math expert with a PhD, our Senior UX Researcher Becky Wright, and furrow my brow in a vain attempt to understand what’s going on.

Before we even get to confidence intervals though, first we need to make sure you have a grasp of some other math stuff.

First of all…

What is the mean?

The mean is the average for a set of numbers. Let’s say you want to find the average height of all the Sesame Street characters… you’d line them all up in a row, measure them and you’d have various heights ranging from 2’ to 8’2”.

Big Bird throws this way out

Then you add up all the above heights, then divide by how many characters there are. So X divided by Y =  the mean height of Sesame Street characters.

As for the mean Sesame Street character, well, that’s Oscar the Grouch of course (wocka wocka).

What is a sample?

In the context of user research, a sample is a representative subset of a population.

Imagine you want to know what a whole country thinks. This is going to be difficult, expensive and frankly implausible. That’s why taking a sample is necessary.

This can either be entirely random from the entire population or you could be slightly more specific and break the population into subsets, where the people share common traits, and randomly pick from this group to get your sample.

What is standard deviation?

Math is Fun (yes, there’s actually a website called “Math is Fun”) describes standard deviation as a measure of how spread out a set of numbers are around the mean number.

A low standard deviation means that most of the numbers are close to the mean. A high standard deviation means the numbers are more spread out.

Standard deviation is useful because it provides a “standard” way of knowing what is normal, and what is extra large (Big Bird) or extra small (Elmo).

Got that? Great!

So this brings us neatly to… Confidence Intervals!

What are confidence intervals?

Confidence intervals are calculated from an estimate of how far away our sample mean is from the actual population mean.

Much like standard deviation, confidence intervals provide us with an upper and lower limit around our sample mean. However, within this interval we can be confident we have captured the population mean.

Why do we need confidence intervals?

When we run usability studies we are typically targeting a particular demographic, whether that is the general population, students in the UK, or women over 30 with at least one child as just a few examples.

Whatever our target audience, we can’t usually test all the people who are in this population due to the feasibility of accessing all these people, and the amount of time and money it would take.

Imagine we wanted to test two designs for a student bank account to see which design students preferred. In 2016 there were 2.28 million students studying at UK Universities alone – that would take a huge amount of time, cost and effort to ask all those students to rate the two designs.

So instead we’ll test 100 students with UserZoom. This sample size is a lot more manageable and cost effective and we hope our 100 UK students will be representative of the population of all UK students.

From this sample data we can get an idea of how well received the designs are, and it allows us to make a more informed decision about which design to implement on the website.

The problem with taking samples is sometimes our sample mean will be similar to the population mean, and sometimes we might collect a sample and the mean is actually quite different from the population mean. This is just due to something called sampling error.

The trouble is, we have no idea if our sample mean is a good or poor representation of the population mean. This is where confidence intervals come in to help!

The lower limit and upper limit around our sample mean tells us the range of values our true population mean is likely to lie within.

Why are confidence intervals useful in UX research?

When we run studies we want to be confident in the results from our sample. Confidence intervals show us the likely range of values of our population mean. When we calculate the mean we just have one estimate of our metric; confidence intervals give us richer data and show the likely values of the true population mean.

Sample size is one part of the equation used to calculate confidence intervals. If we increase our sample size (and kept everything else the same) we will see our confidence intervals reduce. When it comes to confidence intervals, the smaller the better! This is because we have a smaller range of values our population mean could lie within.

When time and money are tight in UX research we sometimes have to rely on smaller sample sizes. However, by calculating the confidence intervals around any data we collect, we have additional information about the likely values we are trying to estimate.

You can read more about sample sizes in our article What sample size do you really need for UX Research?

How do we compute confidence intervals?

We use the Adjusted Wald method to calculate confidence intervals.

Here’s a segment from Tom Tullis’ Measuring the User Experience to explain a little more:

“Sauro and Lewis (2005) demonstrated that the Adjusted Wald Method of calculating a confidence interval works well for many of the situations we encounter in usability testing. The basic idea behind the Adjusted Wald Method is that you need to adjust the observed proportion of task successes to take into account the small sample sizes commonly used in usability tests.” You can also click on the above article link to see the formula for calculating the Adjusted Wald confidence interval.

Please note, if other methods or further analysis is needed, all data can be downloaded in Excel or SPSS.

In conclusion…

Confidence intervals are there to help! They make your data analyses richer and give you more from the metrics you captured and help you to make more informed decisions about your research questions. And now, thanks to this latest update, confidence intervals are calculated for you automatically, enabling you to uncover richer, more nuanced insights at the click of a button.

To learn more about UserZoom’s all-in-one UX research platform, please visit www.UserZoom.com or get in touch!