Semantic Web enabled web accessibility evaluation tools
Shadi Abou-Zahra · 2005 · Proceedings of the 2005 International Cross-Disciplinary Workshop on Web Accessibility (W4A) · doi:10.1145/1061811.1061830
Summary
This paper by Shadi Abou-Zahra of the W3C presents the Evaluation and Report Language (EARL) as a Semantic Web-based solution to fundamental problems in web accessibility evaluation tooling. Abou-Zahra identifies that the accessibility evaluation tool market of 2005 provided little consistency in reliability and performance across tools, with some tools potentially confusing or misleading inexperienced users. EARL addresses this by providing a platform-independent, vendor-neutral data format for expressing test results using the Resource Description Framework (RDF). The EARL data model consists of four core classes: Assertor (who or what conducted the test), Subject (what was tested — typically a web page but extensible to any object), Testcase (the criteria being tested against, such as WCAG checkpoints), and Result (outcomes including pass/fail status, confidence indicators, and failure reasons). Because EARL is built on RDF Schema, it inherits compatibility with the broader Semantic Web ecosystem including ontology languages (OWL, DAML), query languages (SPARQL, RDQL), and existing vocabularies (Dublin Core, FOAF). The RDF inheritance model allows domain-specific extensions while maintaining backward compatibility.
Key findings
Abou-Zahra identifies eight concrete use cases for EARL that go well beyond simple report generation. Combining reports allows results from multiple tools with different strengths (e.g., one with better color contrast analysis, another with better text analysis) to be merged into a single repository, maximizing evaluation coverage. Comparing test results enables reviewers to prefer certain tools for specific WCAG checkpoints, reducing both false negatives and false positives. Benchmarking tools against known-correct results (like WCAG test suites) can reduce guideline misinterpretation and improve cross-tool consistency. Data mining enables sorting and prioritizing results by severity, repair cost, or other policies. Customized reporting allows the same test data to generate detailed technical views for developers or executive summaries for managers. Integration into authoring tools, browsers, and search engines extends EARL's utility — browsers could use accessibility results to re-render content (e.g., linearizing tables or suppressing motion), and search engines could filter results by accessibility. Two key research challenges are identified: pointing to web content reliably (since content changes constantly and XPointers only work for XML), and reasoning about checkpoint-level assertions from multiple proprietary sub-tests.
Relevance
This paper captures a vision for interoperable accessibility evaluation that remains partially unrealized but increasingly relevant. The problem Abou-Zahra identified — inconsistency across accessibility testing tools and the inability to combine their results — persists today, as different automated tools still produce different results for the same page. While EARL itself did not achieve mainstream adoption, its core ideas influenced subsequent developments: modern accessibility testing APIs (like axe-core) use structured JSON output that serves similar interoperability purposes, and CI/CD pipeline integration of accessibility testing reflects the authoring tool integration EARL envisioned. The vision of browser-integrated accessibility evaluation has partially materialized through browser DevTools accessibility panels. For practitioners, the paper's framework for thinking about evaluation tool capabilities — that no single tool catches everything, and combining results reduces errors — remains essential guidance for setting up effective accessibility testing workflows. The paper also highlights the enduring challenge of automated testing: pointing to dynamic web content and reasoning about complex, multi-element accessibility requirements.
Tags: automated testing · accessibility testing · semantic web · EARL · evaluation tools · quality assurance · interoperability · W3C
Standards referenced: WCAG · EARL · RDF · OWL · SPARQL