Accessibility testing: tips and best practices from industry leaders

Many companies want to conduct accessibility evaluations by running usability testing for people with disabilities. Accessibility should be a priority for all organizations. Sometimes, however, the challenge feels so steep that it’s hard to know where to begin.

Getting started with accessibility testing

Before running an accessibility test, you can make sure your site meets fundamental accessibility guidelines. For example: keyboard-only usage, text equivalents for images, and captioning for videos are pretty standard and don't require testers. Once this groundwork has been laid, you’re ready to move forward.

What type of disabilities should you include in your testing?

First you need to consider what disabilities to include in your usability testing.

The internet is inherently visual so you probably want to have some participants with visual disabilities–blind, low vision, color blind, etc.

And because websites require user interaction in the form of linking, scrolling, and forms, you'll want to include some people who have difficulty with fine motor control and/or those who cannot use a mouse or other input device. You’ll want to learn how they use your site, including all navigation and form entry using only their keyboard. Other disabilities you might want to consider are hearing impaired and people with cognitive impairments.

The broad categories of disabilities are:

• Vision (blindness, low vision, color blindness)
• Hearing (deaf or hard of hearing)
• Speech and voice-related
• Manual dexterity (unable to use a mouse)
• Mobility (wheelchair users)
• Cognitive (dyslexia and autism)

The examples we'll be discussing further down in the article are focused mainly on blind and low vision participants, but many lessons are applicable to accessibility testing as a whole.

Remote moderated accessibility testing

If you can conduct in-person moderated research with people with disabilities, that’s ideal. But this involves several logistical challenges. Keep in mind that navigating the physical world can be difficult for participants with disabilities. Also, many times it’s challenging to find 5 to 10 relevant participants in your location and get them to a lab.

So historically we have preferred to conduct remote moderated 1:1 sessions where participants are using their own computers and hence their own assistive technology. "In-the-wild" research could lead to additional insights which could be a added benefit running this type of research in a remote moderated fashion.

Key accessibility tips and best practices

Running a Remote Moderated Accessibility task-based study is similar to typical Remote Moderated studies, but with some additional considerations. Here are some tips and best practices that we use to ensure a successful project.

Smaller Sample Sizes: In this instance, we don’t recommend trying to recruit 50+ participants with a particular disability. It would likely be too costly and inefficient. Even with the help of a specialized panel partner, there simply may not be enough people who fit your criteria.

Two-Person Team: We found that having a two-person team helps increase efficiency and productivity. Typically we recommend having one person act as the moderator whose main role is to focus on the participant. The second person should act as tech support and should be someone who understands, oversees, and supports the technology used during the session.

Use Fewer Tasks: We recommend avoiding the kitchen sink approach here and to really focus in on a few key tasks. This is because people who rely on assistive technologies, such as a screen reader, often require more time to complete the tasks.

Carefully Define Your Audience: The user profiles or personas you create for testing people without disabilities also apply to people with disabilities. However, you need to give some consideration to the types of disabilities you need to test for (e.g., not all vision impaired users are blind).

Get Recruiting Help: Like we mentioned earlier, we highly recommend finding a recruiting partner that specializes in recruiting participants with disabilities. Trying to find participants through your regular channels can be more difficult than you might expect. You’ll also want to provide the recruiter with specific hardware and software requirements. For example, not all web conferencing features (e.g., screen sharing) support screen readers (i.e., JAWS).

Allocate Time for Two Meetings: This is an important one that you might not think about if you haven’t run an accessibility test before. Allocate budget and project timeline to include two separate meetings with the participants: 1) Tech Check and 2) Test Session. This will reduce the likelihood that significant technical issues will occur during the main test session.

Two examples from leaders in their fields

It’s great that many companies have been conducting audits through consultants and automated tools to understand issues that might prevent people with disabilities from navigating websites, mobile apps, and applications. But, as you will see from the two upcoming examples, we can learn a lot through one-on-one sessions and direct observations.

Recently, two of UserTesting's industry-leading customers came to us with a request to set up a test for users that are blind so that they could better understand–and serve–these customers. We learned quite a bit throughout the process and wanted to pass along this knowledge to others who are endeavoring to make their products more accessible.

Express Scripts

“It’s pretty amazing that it only took four weeks from presenting the results and recommendations to remedying the identified issues.”

Express Scripts provides integrated pharmacy benefit management services. Serving more than 83 million Americans, they make practicing pharmacy smarter so that better health is more affordable and accessible. Central to their goal is ensuring that design decisions are user-centric, and are leaders in healthcare when it comes to UX & design.

Express Scripts wanted to evaluate the accessibility and usability of their homepage, as well as the associated functionality of the website as a whole in regards to people with blindness or low visibility. On top of this, Express Scripts wanted to better understand the ease with which blind and low vision customers could navigate the site and understand the content on the page.

To do this they conducted remote moderated sessions with participants who were given several tasks to complete on the Express Scripts homepage. These participants provided think out loud feedback as they worked through the tasks.

The Study

Method: Remote Moderated 1:1 sessions with 7 blind participants
Platform: Data was collected using UserTesting platform
Study stimuli: Online Website (express-scripts.com)
Task objectives: Explore the homepage; Check order status; Refill a RX; View balance

Participants:
Express Scripts recruited 7 participants through Access-Works’ partnership with UserTesting. All the participants were blind and regularly use JAWS (a screen reader) to access digital content. All of them currently manage maintenance prescriptions so were a target audience.

The Results:
Due to the project Express Scripts was able to uncover accessibility issues for blind and low vision participants as well as general usability issues. The research team also identified what keyboard equivalents, mouse operations, and windows are supported for participants.

The team was also able to discover new use cases and scenarios, such as one participant managing both his and his wife’s prescriptions using their mobile app. Overall, the project educated the product team about the obstacles customers with disabilities face and built empathy that will be baked into the design process.

Fortune 500 Insurance Company

Over the years, this Fortune 500 company has made significant investments in improving overall User Experience & Design for all its digital properties, websites, and apps for consumers, businesses, insurance agents & financial advisors. They have long been considered a thought leader in the Design & UX space.

They wanted to evaluate their website for usability issues pertaining to those who have disabilities. The intent was to meet and exceed the 508 accessibility standards in order to be compliant with government requirements. Beyond this, the goal was to generate empathy for the degree of difficulty that blind and low vision customers face navigating any website using assistive technologies.

Their team had previously conducted a software test using Compliance Sheriff to identify errors in alt tags and was ready to move onto the next step – remote moderated testing with UserTesting.

The Study

Method: Remote Moderated 1:1 sessions with 9 blind and low vision participants
Platform: Data was collected using UserTesting platform
Study Stimuli: Online Website
Task Objectives: Find a video; Find a career within a geographic location; Customer support and feedback forms; How to file a travel claim; Search for mutual funds information

Participants:
They recruited 9 participants for this particular study – 7 blind users and 2 low vision users. One of the participants also has limited mobility. 

The Results:
With the company’s previous focus on alt tags, several tasks were relatively easier for blind users to navigate and find using keyboard navigation and JAWS software. During the study a few key issues were identified related to footer and site maps, as well as an issue blind and low vision participants had with long dropdown menus (country selection, for example), as well as a couple of mis-labeled alt tags and video tags. Furthermore, navigational challenges were identified so that the team could get to work making the website easier to navigate.

Thanks to the study, they were able to build more empathy for the challenges blind and low vision participants face, come away with a plan to better the user experience for these participants, as well as see their focus on accessibility paying off with real users.

Why accessibility testing is important

Most companies will have a portion of customers that will have some form of disability. For many of these customers, a common task like refilling a prescription on a healthcare website, finding a specific item on an e-commerce site, or even trying to open a checking account on a financial site can often be nearly impossible for users with blindness/low vision as well as other disabilities.

The good news is that many organizations conduct accessibility audits to identify issues that prevent customers with disabilities from completing typical tasks.

While UX teams often use In-lab Moderated Accessibility Tests to build empathy and awareness of these issues, and educate product teams on the challenges that this customer segment face, lab testing for users with disabilities can be very costly when you factor in all the necessary assistive technologies (i.e., screen readers), and the logistics for finding, recruiting, and getting them into the lab.

At UserTesting, we believe that products should be easy to use for everyone, and we heartily support the endeavors of our clients and organizations who take this issue seriously as well.