Accessibility audit: functional testing or UX research? (user experience)

When discussing accessibility audits with our clients or within the team, we inevitably compare them with our long-established services. Not without reason: an accessibility audit combines characteristics of both functional testing and UX testing. Which service is closer to an accessibility audit: functional testing or UX research? In short: both, and neither. But let’s start at the beginning. (user experience)

What is functional testing?

The purpose of functional testing is to compare the implemented system against the plans and identify defects – in other words, points where the system deviates from the specification or from the way the rest of the system works. The outcome of testing is a bug list containing well-documented, reproducible issues. The list generally records objectively existing defects that need fixing – although there are, of course, cases where it is debatable whether a given element is a ‘bug’ or a ‘feature’.

What is an accessibility audit?

The purpose of an accessibility audit is to identify accessibility issues in a digital interface and ensure that all users – including people with disabilities – can use it without barriers. The audit checks compliance with the requirements of the WCAG standard. Its outcome is also a bug list, which likewise contains – for the most part – objective issues. (Web Content Accessibility Guidelines)

Functional testing vs accessibility audits

Similarities

• Approach – Both processes assess how a system works against predefined requirements.
• Objective, measurable results – Both accessibility audits and functional testing focus on objectively existing issues that can be reproduced and fixed.
• Documenting errors and issues – Both result in a bug list, and resolving it brings the system closer to the expected behaviour.

Differences

• Focus – Functional testing focuses on how the system works, whereas an accessibility audit examines whether the system is accessible to all users, including those using assistive technologies.
• Specificity of requirements – The requirements defined in the accessibility standard, WCAG (Web Content Accessibility Guidelines), are phrased fairly broadly. In contrast, functional-testing test cases are based strictly on the specification of the software in question.
• Source of requirements – The requirements used in accessibility audits were developed by the World Wide Web Consortium (W3C). They are internationally accepted, have effectively become a standard, and every accessibility audit assesses compliance with these same requirements. Functional testing, on the other hand, uses bespoke test scenarios for the specific system: a test scenario used for one system cannot be applied to another, as they differ from system to system.

UX testing vs accessibility audits (user experience)

The purpose of UX testing – also known as usability testing or UX research – is to understand users’ needs, behaviours and problems, so that a more intuitive, efficient and enjoyable digital experience can be created on that basis. It examines how users interact with a product or service, where they get stuck, and what would help them use and enjoy a particular digital interface smoothly. (user experience)

Now let’s compare UX research and accessibility audits. (user experience)

Similarities

• Goal – Both examine how usable a system is for a particular target group and aim to make it more usable.
• Outcome – In both cases, testing results in a bug list, although their formats differ (see the differences).
• Recommendations for a better experience – Both methodologies aim not only to identify problems, but also to provide recommendations for resolving them.

Differences

• Specificity of requirements – WCAG requirements are phrased relatively broadly, at least compared with the test scenarios used in functional testing. However, the requirements considered in UX research are even less specific. For example, concepts such as understandability and learnability leave considerable room for interpretation. (Web Content Accessibility Guidelines) (user experience)
• Objectivity – Although both result in a bug list, issues identified through UX research are less objective. UX researchers often conclude that ‘most users’ understood something in one way or another. By contrast, accessibility audits are more objective: the issues they identify are less a matter of opinion – though accessibility audits also have subjective elements. (user experience)
• Target group – UX research always examines the behaviour of a defined target group or groups; in other words, a central question in UX research is who the participants are. By contrast, an accessibility audit is not target-group-specific: it assesses the interface against a universal set of usability criteria. (user experience)
• Methodology – UX research always involves test participants, whereas an audit is carried out by an expert auditor. Accordingly, it would theoretically be possible to conduct UX audits and accessibility research too, but in our experience there is currently less client demand for these. (user experience)

Summary

Accessibility audits share many similarities with both methodologies presented here. In their focus on users, they are closer to UX research; in their specificity and pursuit of objectivity, they are closer to functional testing. This dual nature makes accessibility audits unique, as they must meet both technical requirements and user-experience considerations. While functional testing focuses on whether a system ‘works’, and UX research on whether people ‘enjoy using it’, an accessibility audit sits between the two by examining whether ‘everyone can use it’. (user experience)