Monitoring Accessibility: Large Scale Evaluations at a Geo Political Level
Silvia Mirri, Ludovico Antonio Muratori, Paola Salomoni · 2011 · The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2049536.2049566
Summary
This paper presents AMA (Accessibility Monitoring Application), a system designed to monitor web accessibility across large collections of URLs from a geo-political perspective. Developed as part of the VaMoLà project — a collaboration between the Emilia-Romagna Region and the University of Bologna — AMA addresses the need for public institutions to systematically track the accessibility compliance of websites under their jurisdiction. Unlike existing tools that evaluate individual websites, AMA provides macro-level analysis by linking web resources to the specific institutions responsible for them and the geographical areas they serve. The system periodically evaluates URLs using an external accessibility validator called AVA (derived from AChecker) that checks against multiple guideline sets including WCAG 1.0, WCAG 2.0, Section 508, and the Italian Stanca Act. A custom colour contrast validator (Co2) was developed to properly analyse XHTML and CSS code, accounting for stylesheet cascading characteristics and inheritance rules that existing contrast analysers ignore. Results are presented through both geo-referenced maps (using Google Maps mashups with colour-coded accessibility levels by country/region) and detailed tabular data with temporal comparison capabilities.
Key findings
An instance of AMA monitoring 154 main European public institution websites (approximately 5 per country including parliaments, governments, and prime ministries) revealed significant geographical variation in accessibility levels. The Netherlands had the highest percentage of error-free websites (66.67%) while Ireland had the lowest average errors per page (2.00) according to WCAG 2.0 Level A. The system introduced a novel metric called the Barriers Impact Factor (BIF), computed by summing the product of error count and weight for each assistive technology/disability category affected. Barriers are grouped into seven impact categories: screen reader/blindness, screen magnifier/low vision, colour blindness, input device independence/movement impairments, deafness, cognitive disabilities, and photosensitive epilepsy. Weights are assigned by conformance level (Level A errors = 3, Level AA = 2, Level AAA = 1, manual controls = 0). The BIF on screen reader/blindness was consistently highest, as these barriers are most frequently and automatically detectable. The Emilia-Romagna Region was actively using AMA to monitor 376 websites (about 4,000 pages) across its regional, provincial, and municipal institutions.
Relevance
This research addresses a critical gap in accessibility governance: while regulations mandate compliance, institutions rarely have the tools to systematically monitor and enforce it across their jurisdictions. AMA demonstrates that automated, periodic, geo-referenced accessibility monitoring is both technically feasible and operationally useful for public administrations. The Barriers Impact Factor metric is a practical contribution — by weighting errors based on which assistive technologies and disabilities they affect, it produces more meaningful accessibility scores than simple error counts. For accessibility practitioners and policymakers, the geo-political visualisation approach enables strategic resource allocation, directing remediation efforts and training investments where they are most needed. The system's ability to compare results across time periods supports accountability and progress tracking. The custom CSS contrast checker that accounts for cascading inheritance is also noteworthy, as colour contrast analysis remains a weak point in most automated tools even today.
Tags: web accessibility · automated testing · accessibility evaluation · policy · compliance · governance · metrics · data visualization
Standards referenced: WCAG 1.0 · WCAG 2.0 · Section 508 · Stanca Act