Building reusable accessibility testing tools: what should we test and how should we demonstrate reliability? #6988

One useful reliability fixture would be a complete keyboard task, not just a component at rest

Agreed, components are not used in isolation, we'll need to test in realistic scenarios, which includes completing user flows via keyboard navigation.

I would publish the tested USWDS/browser/assistive-technology versions

This could be a report we implement. However the git history/release versioning and test files themselves should be the source of truth as far as what has/hasn't been tested at any particular point in time.

Then run the same known-pass and deliberately broken variants repeatedly to report misses, incorrect warnings, and inconsistent results separately.

That would be more of a test of the tooling we develop. For example the tool we develop for testing that screen-reader output is observed will have tests to verify that it's passing and failing when it should.

If a tool cannot observe the screen-reader output, that check should remain “not assessed” rather than counted as a pass.

Right. We should be tracking whats not being tested. One of the challenges here is to identify what we should be testing for any particular component and what scenarios should be tested. One thing I'm currently working on is an accessibility checklist that defines the minimal expected checks that need to be in place per component and scenario. This will help us identify what needs to be tested so we can try automating it or have visibility of it so that we know what our automation gaps are and what will need to be manually tested. I'll be making the accessibility checklist public in about a week, any community feedback would be appreciated.

Thanks for sharing this. I noticed your original comparison focused on conventional scanners. We’re interested in testing whether AI-assisted tools can reliably address some of those gaps.

Have you come across Aloud? Its approach to capturing screen-reader output and producing inspectable evidence is relevant to what we’re exploring.

Our goal is faster, actionable feedback on PRs, repeatable regression tests, and clearer evidence of what was tested for each release. Any additional coverage would need validation, with manual assessment where automation remains insufficient.

Thanks for sharing this. I noticed your original comparison focused on conventional scanners. We’re interested in testing whether AI-assisted tools can reliably address some of those gaps.

The first bullet asked about automation without specifying "AI" (which your second bullet did). I can't tell from this statement if you're acknowledging or dismissing the info I shared. Which I suppose doesn't matter, but if the latter I feel I answered the question as posed.

Have you come across Aloud? Its approach to capturing screen-reader output and producing inspectable evidence is relevant to what we’re exploring.

I have not. I see that you are one of the people behind it, so I know you'll have more insight than my quick review of the README. But some immediate concerns looking at it:

• Screen reader text output is not a proxy for actual use (let alone announcement) as I outline in a 2020 post. I wrote that partly as a response to JAWS Inspect.
• Aloud appears to only evaluate screens as if "read-all" was run, since I see nothing outlining interactions nor how it tests for state.
• The VO output appears to be computed, not a true feed, meaning there will be parity risks.
• I see nothing about Braille support.
• It treats iOS as the only mobile iDevice platform, when iPadOS is at least worth a confirming test pass (assuming the apps will never be installable on watchOS, tvOS, visionOS, sandwichOS).
• This cannot be used by many orgs' existing QA nor accessibility testers as it requires two dev machines to run (macOS and a simulator).
• The experimental web support appears to only be Chromium (so Firefox is a gap) and runs NVDA as an option (which even in a Windows desktop is not an ideal browser / SR pairing and definitely cannot recreate Android AAPIs), and clearly would be useless to test iDevice web content (wrong engine, wrong OS, wrong everything). It's very much mixing and matching things that aren't mixed / matched. I would probably disallow teams from using it because of the risk of false positives and negatives.
• I don't understand the OpenACR output (508 on its own doesn't seem enough?). I would presume the tool is meant to produce output for devs and testers, in which case I'd look for WCAG-EM-friendly output. Otherwise the tool is limiting itself to a very US-centric audience, and a gov-centric one at that.

All those notes aside, Aloud does not appear to be an "AI"-assisted tool (which is what your response seems to want). This appears to be largely API handling, string manipulation, regex, and tests using existing deterministic automated rules engines (which I addressed in my prior answer). In short, a series of "conventional scanners" (to use your words) in a trenchcoat. Which can make it harder to identify scanning gaps.

That said, given your goal of faster testing I think this shows promise. Its output, as I understand it, cannot be used on its own to confirm conformance, but it can support manual testers by flagging some of the more obvious issues in native apps. I look forward to seeing how it matures.

Our goal here is to improve accessibility, not reduce costs.

any poor accessible component may have to be remediated many times over.

This is why we’re building tools to automate more testing where we can demonstrate reliability, while retaining manual testing where needed. We expect this to help us catch accessibility issues earlier and deliver USWDS improvements faster, with stronger evidence supporting each release.

Agreed. Usability testing with people with disabilities can reveal barriers that conformance checks miss. We’re aiming to strengthen automated testing while being clear about what it can establish and where testing with people remains necessary.

Yes I think we should maintain those! In addition I'd love to connect those to our tests. For the checks we manage to automate the testing of, you'd be able to see the actual test and results that verified each requirement.

I believe this topic needs more research before we begin to build. I agree that we should have evidence that agentic tools are more accurate and cost-effective than commercially available tools that we are already using.

We cannot automate the full range of testing needed: Accessibility means that humans can access and use the website, and understand the content (this includes Plain Language). Much of what makes something accessible has to do with senses a machine does not have, and some WCAG criteria are too subjective for machines (for instance, what is a good caption for an image given what the author is trying to say)?

That said, there is potentially room for agentic tools to fill in some gaps around where our current tools miss errors. This would let us catch some problems earlier and save time and money by fixing them before we build on top of those errors.