Accessibility Testing: Tips & Best Practices From Industry Leaders
Learn about best practices for accessibility testing from two leading companies in their industries.
Lots of companies want to conduct accessibility evaluations by conducting usability testing for people with disabilities. It’s the right thing to do, and companies genuinely want to be accessible to all. Sometimes, however, the challenge feels so steep that it’s hard to even know where to begin.
If you are new to accessibility and want to learn more, Usability.gov has good resources. W3C also has easy checks to review your website for accessibility. We at UserZoom are also on a mission to make it easier for enterprises that want to conduct accessibility research with real people with disabilities to do so. To learn more check out our partnership with Access-Works.
Where to begin?
Before running an accessibility test, you can make sure your site meets fundamental accessibility guidelines. For example: keyboard-only usage, text equivalents for images, and captioning for videos are pretty standard and not necessary to bring in any testers for. Once this groundwork has been laid, you’re ready to move forward.
Furthermore, be aware that one of the toughest parts of accessibility testing is knowing how to recruit participants. For UserZoom customers it’s easier because of our partnership with Access-Works. For anyone else feel free to directly contact Access-Works.
What type of disabilities should you include in your testing?
First you need to consider what disabilities to include in your usability testing.
The web is inherently visual so you probably want to have some participants with visual disabilities – blind, low vision, color blind, etc.
And because websites require user interaction in the form of linking, scrolling, and filling in forms, you will want to include some people who have difficulty with fine motor control and/or those who cannot use a mouse or other input device. You’ll want to learn how they use your site, including all navigation and form entry using only their keyboard.
Other disabilities you might want to consider are hearing impaired, especially if your site uses audio, and people with cognitive impairments that can tax perception and short-term memory.
The broad categories of disabilities are:
- Vision (blindness, low vision, color blindness)
- Hearing (deaf or hard of hearing)
- Speech and voice-related
- Manual dexterity (unable to use a mouse, e.g.)
- Mobility (wheelchair users, e.g.)
- Cognitive (dyslexia and autism, e.g.)
The examples we will be discussing further down in the article are focused mainly on blind and low vision participants, but many lessons are applicable to accessibility testing as a whole.
Remote Moderated Accessibility Testing
If you can conduct in-person moderated research with people with disabilities, that’s fabulous. But this involves several logistical challenges. Keep in mind that navigating the physical world can be difficult for participants with disabilities. Also, many times it’s challenging to find even 5 to 10 participants in location and get them to a lab.
So historically we have preferred to conduct remote moderated 1:1 sessions where participants are using their own computers and hence their own assistive technology. In-the-wild research could lead to additional insights which could be a added benefit running this type of research in a remote moderated fashion.
Tip: Check out our introduction to remote moderated article and our webinars about remote moderated usability testing to learn more!
Key tips & Best Practices
Running a Remote Moderated Accessibility task-based study is similar to typical Remote Moderated studies, but with some additional considerations. Here are some tips and best practices that we use to ensure a successful project.
Smaller Sample Sizes: In this instance, we don’t recommend trying to recruit 50+ participants with a particular disability. For one, that would be incredible costly and inefficient. Second, even with the help of a specialized panel partner, there simply may not be that many people that fit your criteria.
Two-Person Team: We found that having a two-person team helps increase efficiency and productivity. Typically we recommend having one person act as the moderator whose main role is to focus on the participant. The second person should act as tech support and should be someone who understands, oversees, and supports the technology used during the session.
Use Fewer Tasks: We recommend avoiding the kitchen sink approach here and to really focus in on a few key tasks. This is because people who rely on assistive technologies, such as a screen reader, often require more time to complete the tasks.
Carefully Define Your Audience: The user profiles or personas you create for testing people without disabilities also apply to people with disabilities. However, you need to give some consideration to the types of disabilities you need to test for (e.g., not all vision impaired users are blind).
Get Recruiting Help: Like we mentioned earlier, we highly recommend finding a recruiting partner that specializes in recruiting participants with disabilities. Trying to find participants through your regular channels can be timely and inefficient. You’ll also want to provide the recruiter with specific hardware and software requirements. For example, not all web conferencing features (e.g., screen sharing) support screen readers (i.e., JAWS).
Allocate Time For Two Meetings: This is an important one that you might not think about if you haven’t run an accessibility test before. Allocate budget and project timeline to include two separate meetings with the participants: 1) Tech Check and 2) Test Session. This will reduce the likelihood that significant technical issues will occur during the main test session.
Two Examples From Leaders in their Fields
It’s great that many companies have been conducting audits through consultants and automated tools to understand issues that might prevent people with disabilities from navigating websites, mobile apps, and applications. But, as you will see from the two upcoming examples, we can learn a lot through one on one sessions and direct observations.
For example – if you have never seen how a blind user navigates your website through assistive technology you are missing out, plain and simple.
Recently, two of UserZoom’s industry-leading customers came to us with a request to set up a test for users that are blind so that they could better understand, and better serve, these customers. We learned quite a bit throughout the process and wanted to pass along this knowledge to others who are endeavoring to make their products more accessible.
“It’s pretty amazing that it only took four weeks from presenting the results and recommendations to remedying the identified issues.”
Express Scripts provides integrated pharmacy benefit management services. Serving more than 83 million Americans, they make practicing pharmacy smarter so that better health is more affordable and accessible. Central to their goal is ensuring that design decisions are user-centric, and are leaders in healthcare when it comes to UX & design.
Express Scripts wanted to evaluate the accessibility and usability of their homepage, as well as the associated functionality of the website as a whole in regards to people with blindness or low visibility. On top of this, Express Scripts wanted to better understand the ease with which blind and low vision customers could navigate the site and understand the content on the page.
To do this they conducted remote moderated sessions with participants who were given several tasks to complete on the Express Scripts homepage. These participants provided think out loud feedback as they worked through the tasks.
Method: Remote Moderated 1:1 sessions with 7 blind participants
Platform: Data was collected using UserZoom platform
Study stimuli: Online Website (express-scripts.com)
Task objectives: Explore the homepage; Check order status; Refill a RX; View balance
Express Scripts recruited 7 participants through Access-Works’ partnership with UserZoom. All the participants were blind and regularly use JAWS (a screen reader) to access digital content. All of them currently manage maintenance prescriptions so were a target audience.
Due to the project Express Scripts was able to uncover accessibility issues for blind and low vision participants as well as general usability issues. The research team also identified what keyboard equivalents, mouse operations, and windows are supported for participants.
The team was also able to discover new use cases and scenarios, such as one participant managing both his and his wife’s prescriptions using their mobile app. Overall, the project educated the product team about the obstacles customers with disabilities face and built empathy that will be baked into the design process.
Fortune 500 Insurance Company
Over the years, this Fortune 500 company has made significant investments in improving overall User Experience & Design for all its digital properties, websites, and apps for consumers, businesses, insurance agents & financial advisors. They have long been considered a thought leader in the Design & UX space.
They wanted to evaluate their website for usability issues pertaining to those who have disabilities. The intent was to meet and exceed the 508 accessibility standards in order to be compliant with government requirements. Beyond this, the goal was to generate empathy for the degree of difficulty that blind and low vision customers face navigating any website using assistive technologies.
Their team had previously conducted a software test using Compliance Sheriff to identify errors in alt tags and was ready to move onto the next step – remote moderated testing with UserZoom.
Method: Remote Moderated 1:1 sessions with 9 blind and low vision participants
Platform: Data was collected using UserZoom platform
Study Stimuli: Online Website
Task Objectives: Find a video; Find a career within a geographic location; Customer support and feedback forms; How to file a travel claim; Search for mutual funds information
They recruited 9 participants for this particular study – 7 blind users and 2 low vision users. One of the participants also has limited mobility. All 9 participants were recruited through IntelliZoom partner Access-Works. Access-Works helped technical set up in addition to finding and scheduling participants.
With the company’s previous focus on alt tags, several tasks were relatively easier for blind users to navigate and find using keyboard navigation and JAWS software. During the study a few key issues were identified related to footer and site maps, as well as an issue blind and low vision participants had with long dropdown menus (country selection, for example), as well as a couple of mis-labeled alt tags and video tags. Furthermore, navigational challenges were identified so that the team could get to work making the website easier to navigate.
Thanks to the study, they were able to build more empathy for the challenges blind and low vision participants face, come away with a plan to better the user experience for these participants, as well as see their focus on accessibility paying off with real users.
Why Accessibility Testing is Important
Most companies will have a portion of customers that will have some form of disability. For many of these customers a common task like refilling a prescription on a healthcare website, finding a specific item on an ecommerce site, or even trying to open a checking account on a financial site can often be nearly impossible for users with blindness/low vision as well as other disabilities.
The good news is that many organizations conduct accessibility audits to identify issues that prevent customers with disabilities from completing typical tasks.
While UX teams often use In-lab Moderated Accessibility Tests to build empathy and awareness of these issues, and educate product teams on the challenges that this customer segment face, lab testing for users with disabilities can be very costly when you factor in all the necessary assistive technologies (i.e., screen readers), and the logistics for finding, recruiting, and getting them into the lab.
Some forward-thinking UX teams, such as the teams at Express Scripts and the Fortune 500 Insurance company, have turned to Remote Moderated Accessibility Testing as a more convenient, cost-effective, more flexible option for both the organization and participants. Participants use their own computers and assistive technologies, and UX teams can gather evidence (video clips, attitudinal data) about accessibility issues from users in their own words.
Here at UserZoom we believe that products should be easy to use for everyone, and we heartily support the endeavors of our clients and organizations who take this issue seriously as well.
If you are a UserZoom customer talk to your Customer Success Manager and we will walk you through all the best practices and how to’s! We are also happy to conduct usability testing with people with disabilities or help you conduct this type of research.
Kuldeep Kelkar — VP of UX Consulting & Professional Services, UserZoom
Kuldeep Kelkar has more than 15 years of UX research, design and engineering experience and joins UserZoom from TATA Consultancy Services (TCS), where he was Head of User Experience. Prior to that, he led UX Design and User Research at PayPal for nearly a decade. He also has led UX efforts at KLA-Tencor and Icon Medialab.
Kuldeep is a renowned author and speaker, and he brings a BE in Industrial Engineering from Nagpur University and an Masters Degree in Human Computer Interaction from the Industrial Engineering department at Clemson University.
John Romadka — Sr. User Experience Researcher at UserZoom
John is a User Experience Researcher with 10+ years of practical experience in a broad range of quantitative and qualitative user research, analysis, and design for software and web products. John is focused on helping companies learn how to make their web and mobile products easier to use.
Thank you for signing up!
You should receive an email from us shortly