What’s the difference between formative and summative usability testing?
When it comes to usability testing, many are unaware that there are two types of evaluations that can be conducted during the product design and development lifecycle. These are formative and summative usability tests.
The main reason why usability tests are important is to find out the effectiveness of the design and to evaluate the ease-of-use of a product. Even though both types of testing are performed to understand and improve the usability of a design, they are performed at different stages of the development process.
So which tests do we use during the product design and development lifecycle? The only way to know is to understand the differences between the two.
Let’s take a look at what formative and summative usability testing means…
What is formative usability testing?
Formative usability testing is done early in the product development to help form the product’s shape and design.
Formative usability testing answers the why and how questions of the design’s usability. It answers why something is not working and involves iteratively evaluating a product during design and development.
The goal is to detect any issues and eliminate usability problems before a product is fully developed. Formative usability testing takes the role of a support tool for decision making during the beginning stages of the design process, which helps to discover insights and shape the product’s direction.
Formative usability testing tends to be more qualitative in nature. We can see how users actually experience the design and see where and why they get stuck, and hear what they say using the think-out-loud method.
It is crucial to observe and understand the users’ thought processes and their actions resulting from them. The data collected during formative usability testing is observational in nature, hence the name qualitative usability testing.
Everything you need to integrate user research into product development!
Here’s an exclusive collection of content tackling the number one UX challenge – getting user research into the product development process.
This free-to-download bundle contains all the following resources:
- How to include user research in early product development
- Democratizing UX: how to spread user research education and insights throughout your organization
- How To Run Research In Agile Sprints By Democratizing It Across Teams [ebook]
- How to Run User Experience in Agile [on-demand webinar]
Why conduct formative usability testing?
Formative usability testing heavily influences the design decisions and is considered to be an excellent tool to figure out which design features are useful and which are not.
It’s recommended to conduct at least two formative usability tests.
- The first in the early design concept phase with a wireframe prototype that has no working functionality. This approach allows for a validation of the workflows and the initial decisions around navigation, layout and terminology used.
- The second formative usability test should be conducted on a Minimum Viable Product (MVP) with some functionalities ready just before the development phase.
If we don’t conduct formative usability testing during the design and development lifecycle, we might end up designing something that will simply never be a usable solution.
Formative usability testing is carried out:
What is summative usability testing?
Summative usability testing is usually performed later in the product development process when a product is fully developed.
It is often conducted when a design is reasonably complete and involves evaluating the design against quantitative goals or competitor’s products.
It is an evaluation of a product with representative users and tasks designed to measure usability (defined as effectiveness, efficiency, and satisfaction) of the complete product.
The main purpose of summative testing is to evaluate a product through defined measures. It uses UX metrics of users’ success to assess whether the product meets defined usability success metrics.
So, summative usability tests are quantitative in nature, and they act as a final validation where usability issues have been identified and addressed.
For an in-depth guide to measuring UX and proving the value of your research, check out our free-to-download, 42 page guide to running both longitudinal and competitive benchmarking.
Why conduct summative usability testing?
Summative evaluation tells us how usable an interface or a product is. If we don’t do summative usability testing before a product is released, then we won’t find out if an interface or a product has any problems or not.
Summative usability testing is used to obtain measures to establish a usability benchmark or to compare results with usability requirements. The metrics recorded in summative usability testing reflect what actually happens during the test and not user perceptions or feelings.
The usability requirements should be task-based, including: task completion rate, time on task, error rates, and overall user satisfaction.
It’s about measuring effectiveness and ease of use. These metrics include: pass/fail of user tasks, the average time it takes to complete tasks, counting the number of clicks, users’ errors or system errors.
In summative testing, the volume and speed of tasks undertaken, and the pass/fail metrics are more important than the quality of the observation or user narrative.
Summative usability testing is carried out:
- At the end of a development stage before a product is released
- To validate the usability of a product- against usability metrics
- To establish a usability benchmark
- To compare against competitor products
- Test performed with fully functioning prototypes
- Conducted with 15-20 users
Both formative and summative are forms of evaluative research. With formative and summative testing during the design and development lifecycle, both qualitative and quantitative data are collected. These tests help to determine whether the design elements are working or not and how they can be improved.
Start formative testing as early and as often as possible in the design and concept phase for a user-centered design approach. Incorporating user feedback through formative and summative usability testing early and often can result in a better and more usable product.
If you’re looking for a single UX metric to clearly prove to stakeholders that your product is improving through user research, let us introduce you to the qxScore! Read all about it here:
Sabina joined UserZoom recently this year. Her background is in Human Factors Engineering and she specializes in research and strategy with a passion for storytelling and problem-solving. In the past, she has worked as a UX Researcher for companies like Motorola, Veritas Technologies, Echostar, and K12 Inc. She has been untangling the complexities of technology and human interaction since 2006. She transforms user research data into actionable insights that help to improve the ease of use of products. She is a firm believer in UX research and thinks that an excellent design starts with the user.