QualState: Finding Website States for Accessibility Evaluation
Filipe Rosa Martins, Letícia Seixas Pereira, Carlos Duarte · 2024 · Proceedings of the 21st International Web for All Conference (W4A) · doi:10.1145/3677846.3677851
Summary
This paper introduces QualState, a web crawling tool designed to solve a fundamental limitation of automated accessibility evaluation: the inability to test dynamic content in Single Page Applications (SPAs). SPAs like Facebook, Google Maps, and Gmail change their content in response to user interactions without reloading the page, meaning that traditional accessibility evaluation tools — which load and assess a single DOM snapshot — miss a significant portion of the page's content and potential accessibility barriers. QualState uses the Puppeteer framework to control a browser via Chrome DevTools protocol, systematically exploring different states of a SPA by identifying and triggering interactive elements. The tool discovers events on page elements through two phases: first using DomDebugger.getEventListeners to find explicitly attached events, then filtering for elements like links and submit buttons that browsers process natively. After each action, QualState compares the resulting DOM state against previously discovered states using hash comparison, saving only unique states. The tool supports both automatic crawling and semi-automatic exploration through user-provided configurations including login credentials, form data, cookie dismissal instructions, and directed paths through the application. QualState was integrated with QualWeb, an established automated accessibility evaluation tool, extending its reporting format to include per-state assessments for each URL.
Key findings
Evaluation across three SPAs — Plant22 (media elements), PLACM (accessibility data visualizations), and the Portuguese Accessibility Observatory (tabular data) — demonstrated that QualState consistently increases the number of applicable accessibility tests as more states are explored. On Plant22, evaluating 10 states instead of 1 increased passed tests from 11 to 126 and failed tests from 3 to 35, while inapplicable tests dropped from 61 to 571 out of 820 total (showing many more elements became testable). The PLACM SPA showed similar patterns, with passed tests rising from 19 to 210 across 10 states. Importantly, the tool found new accessibility failures that were invisible in the initial state — for example, images with alt text that only appeared after user interaction could now be evaluated for appropriateness. Performance testing showed that increasing both crawl time and concurrent processes yielded more discovered states: for the Portuguese Accessibility Observatory, running 10 processes for 3 minutes found 150 states compared to just 6 with 1 process for 1 minute. The modular architecture allows QualState to be integrated with other evaluation tools beyond QualWeb.
Relevance
This research addresses a growing blind spot in accessibility testing. As the web increasingly moves toward SPA architectures built with frameworks like React, Angular, and Vue, the gap between what automated tools can evaluate and what users actually experience widens. For accessibility practitioners, QualState represents a practical step toward more comprehensive automated evaluation — it can be configured to handle authentication, dismiss cookie banners, and fill forms, making it applicable to real-world testing scenarios. The tool is particularly relevant for large-scale accessibility monitoring, where manually interacting with every state of every page is impractical. However, QualState currently relies on user configuration for complex interactions, and its effectiveness depends on the SPA's structure. Practitioners should view it as a complement to manual testing rather than a replacement, but one that significantly expands the coverage of automated evaluation in dynamic web environments.
Tags: automated testing · web accessibility · single page applications · web crawling · accessibility evaluation · dynamic content · DOM
Standards referenced: WCAG · ACT-R · EN 301 549 · Section 508