What are confidence intervals and why do we need them in usability studies?
In our continuing effort to demystify the statistics in your user research studies, we’re looking at confidence intervals: why do we need them, what are they, and why they are useful?
Why do we need confidence intervals?
When we run usability studies we are typically targeting a particular demographic, whether that is the general population, students in the UK, or women over 30 with at least one child as just a few examples.
Whatever our target audience, we can’t usually test all the people who are in this population due to the feasibility of accessing all these people, and the amount of time and money it would take.
Taking our UK student example, imagine we wanted to test two designs for a student bank account to see which design students preferred. In 2015-2016 there were 2.28 million students studying at a UK Higher Education facilities – that would take a huge amount of time, cost and effort to ask all those students to rate how attractive they found the two designs. And that’s assuming we can get all of them to take part in our study!
If we did have access to all the students in the UK, and they all gave us their attractiveness ratings, the mean we could calculate for each design would be the actual population mean. The population mean is made up of scores from everyone who fits the demographic we want to test. It always exists, but is largely unknown, as we are very rarely able to test everyone in our population of interest.
What we are doing when we run our usability study, say using 100 students in the UK, is taking a sample of the population we are interested in.
This sample size is a lot more manageable and cost effective and we hope our 100 UK students will be representative of the population of all UK students.
From this sample data we can get an idea of how well received the designs are, and it allows us to make a more informed decision about which design to implement on the website.
The problem with taking samples is sometimes our sample mean will be similar to the population mean, and sometimes we might collect a sample and the mean is actually quite different from the population mean. This is just due to something called sampling error.
The trouble is, we have no idea if our sample mean is a good or poor representation of the population mean. This where confidence intervals come in to help!
Everything you need to integrate user research into product development!
Here’s an exclusive collection of content tackling the number one UX challenge – getting user research into the product development process.
This free-to-download bundle contains all the following resources:
- How to include user research in early product development
- Democratizing UX: how to spread user research education and insights throughout your organization
- How To Run Research In Agile Sprints By Democratizing It Across Teams [ebook]
- How to Run User Experience in Agile [on-demand webinar]
What are confidence intervals?
We won’t go into the details of sampling error and how we know how to calculate confidence intervals (phew!) but what exactly are they?
Confidence intervals are calculated from an estimate of how far away our sample mean is from the actual population mean; the amount of error (or discrepancy) from our sample mean to the population mean. Confidence intervals provide us with an upper and lower limit around our sample mean, and within this interval we can then be confident we have captured the population mean.
The lower limit and upper limit around our sample mean tells us the range of values our true population mean is likely to lie within.
You typically see 95% confidence intervals reported but what does this mean? If we calculate a 95% confidence interval, we can be 95% confident that our interval contains the population mean. If we ran our study again, we would be confident our new sample mean would fall somewhere in this interval. If we calculate a 80% confidence interval we could be 80% confident that our interval contains the population mean and so on.
What does this percentage associated with a confidence interval represent? It is how sure, or confident, we want to be that our interval contains the true population mean. If we are only 95% confident our confidence interval contains the population mean, we will 5% of the time have a confidence interval that won’t contain the population mean.
However, we will never know whether the confidence interval we have obtained is one of those 5%. This just something we have to come to terms with, that 5% of all the confidence intervals you look at over your lifetime will not contain the population mean.
If we want to be highly confident our interval captures the population mean we could calculate a 99% confidence interval. We would then be 99% confident that our population mean was captured within our confidence interval.
For example, looking at the same mean in Figure 1 below, we have calculated 95% and 99% confidence intervals, keeping the sample size and all other things constant. The 99% confidence interval has longer arms, in order to be more confident we have captured the population mean so we need to increase the width of our confidence intervals.
Why are confidence intervals useful in user research?
When we run studies we want to be confident in the results from our sample. Confidence intervals show us the likely range of values of our population mean. When we calculate the mean we just have one estimate of our metric; confidence intervals give us richer data and show the likely values of the true population mean.
Sample size is one part of the equation used to calculate confidence intervals. If we increase our sample size (and kept everything else the same) we will see our confidence intervals reduce. When it comes to confidence intervals, the smaller the better! This is because we have a smaller range of values our population mean could lie within.
When time and money are tight in user research, sometimes we do have to rely on smaller sample sizes. However, by calculating the confidence intervals around any data we collect, we have additional information about the likely values we are trying to estimate.
Confidence intervals, although they may not seem it, are there to help! They make your data analyses richer and give you more from the metrics you captured and help you to make more informed decisions about your research questions.
For an in-depth, practical guide to successfully planning and conducting Remote Moderated Research check out our free-to-download, comprehensive ebook:
Becky is a UX Researcher and at UserZoom she tailors and analyses projects to clients’ needs. She scopes projects, builds studies, analyses data and delivers insights to clients.
She holds a Masters in Research Methods and a PhD specialising in Decision Making. Before joining UserZoom she worked in a number of different research areas including medical decision making and risk taking. Becky also managed a lab at the Cambridge Judge Business School and lectured in psychology and statistics.