QA

Accessibility testing: what automation catches and what it never will

"I ran axe and got zero errors, so the site is accessible." It's the most misleading sentence in accessibility. Automated tools are great and you should use them — but according to the tools' own studies, they catch something around one third of accessibility problems. The other two thirds require a keyboard, a screen reader and human judgment. Understanding this split is what separates "passed the scanner" from "a blind person can use it".

Why accessibility is a QA topic

Accessibility (abbreviated a11y — "a", 11 letters, "y") means making sure people with disabilities — visual, motor, auditory, cognitive — can use the product. Besides being the right thing to do and widening your audience, it is a legal requirement in many contexts (in Brazil, the LBI; abroad, the ADA and the European EN 301 549). For QA, it's just one more quality dimension: a button the screen reader doesn't announce is as broken as one that returns a 500 error — except the bug is invisible to anyone testing by looking at the screen.

WCAG and levels A, AA, AAA

The reference standard is WCAG (Web Content Accessibility Guidelines). It is organized around four principles — the app must be Perceivable, Operable, Understandable and Robust (the POUR mnemonic) — and three conformance levels. The industry's practical target is AA.

LevelWhat it covers
AThe minimum: without it, entire groups are left out (e.g. image alt text, keyboard navigation)
AAThe realistic target and the one most demanded legally (e.g. 4.5:1 contrast, visible focus, form labels)
AAAThe ideal, not always achievable for all content (e.g. 7:1 contrast)

The automated layer: fast, cheap and limited

Tools like axe-core (and its wrappers: the axe DevTools extension, Lighthouse, Pa11y) scan the DOM and catch what's verifiable by rule: images without alt, insufficient contrast, form fields without a label, broken heading order, invalid ARIA attributes. It's cheap and should become part of the pipeline — a test that fails the build when someone introduces an a11y regression:

// axe-core inside an E2E test (Playwright/Cypress)
const resultados = await new AxeBuilder({ page }).analyze()
expect(resultados.violations).toEqual([])   // fails the build on any violation

The limit is conceptual: the machine checks presence, not meaning. It confirms the image has alt — not that the alt text describes the image. It confirms a label exists — not that it makes sense. alt="imagem123.png" passes the scanner and is useless to someone using a screen reader.

What only manual testing catches

This is where most real problems live — and no tool replaces it. Two manual tests give a huge return:

1. Keyboard-only navigation. Put the mouse away and go through the entire page with Tab, Shift+Tab, Enter and Esc. The questions: can you reach and trigger everything? Is the focus always visible? Does the order make sense? Does a modal trap focus while open and return it on close? A broken keyboard is the most common a11y bug and the easiest to find.

2. Screen reader. Turn one on (NVDA on Windows, VoiceOver on Mac/iOS, TalkBack on Android) and navigate with your eyes closed. Is each element announced with its name, role and state? Does an icon button say "close button" or just "button"? Is a form error announced when it appears? It's uncomfortable at first — and that's exactly the point.

The question that sums up manual testing isn't "does the attribute exist?", but "can a person who depends on this complete the task?". No scanner answers that question.

How to combine it all in a process

The model that works is layered, from cheapest to most expensive — the same logic as the test pyramid applied to a11y:

  ┌─────────────────────────────────────────────┐
  │  Automated (axe in CI)    → catches ~1/3, always │  cheap, every PR
  ├─────────────────────────────────────────────┤
  │  Manual: keyboard + screen reader            │  per feature
  ├─────────────────────────────────────────────┤
  │  Testing with real users with disabilities   │  the final verdict
  └─────────────────────────────────────────────┘

Let the machine block the obvious regressions on every PR; reserve human effort (keyboard and screen reader) for critical flows on each feature; and, when possible, validate with people who actually use assistive technology — it's the only test that measures real experience, not conformance on paper.

Conclusion

Testing accessibility isn't running a tool and filing the green report. Automation (axe in the pipeline) is the cheap base that catches a third of the problems and prevents regressions; keyboard and screen reader catch most of the rest; and testing with real users gives the verdict. Aim for WCAG AA, treat a button that's mute to the screen reader as the bug it is, and remember that "passed the scanner" and "is usable" are different claims.

Read also

Back to the blog