Metrics for accessibility on the Vamolà project
Silvia Mirri, Ludovico Antonio Muratori, Marco Roccetti, Paola Salomoni · 2009 · Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibililty (W4A) · doi:10.1145/1535654.1535688
Summary
This paper describes the Vamolà project (Italian acronym for "accessibility validator and monitor"), a collaboration between the University of Bologna and the Emilia Romagna Region that integrates an accessibility validator with a monitoring system designed to track accessibility compliance over time across Italian public sector websites. The system is built to enforce Italy's Stanca Act (Law 4/2004), which mandates 22 specific accessibility requirements for public institutions. The validator was developed in cooperation with the AChecker team at ATRC (University of Toronto) and is implemented in three forms: a web application for URL/file evaluation, a REST web service for programmatic access, and an Apache module that automatically generates accessibility reports for every requested page. The monitor periodically queries the validator and computes quantitative accessibility measures using configurable metrics, presenting results through geographic visualizations (SVG maps) organized by Italian administrative divisions (communes, provinces, regions). The paper provides a thorough review of existing accessibility metrics — Sullivan and Matson's failure rate, Parmanto's WAB (Web Accessibility Barrier) score, Bühler's disability-group-specific metric, and Vigo's comprehensive WAQM metric — and describes how Vamolà extends these approaches with customizable, multi-dimensional measurement.
Key findings
The Vamolà validator classifies messages into four types of decreasing certainty: automatically detected errors (e.g., missing alt attributes), potential errors requiring manual verification (e.g., checking alt text accuracy), heuristically detected potential errors (e.g., parsing alt values for strings like "image," "photo," or file extensions that suggest fake alternatives), and errors indirectly related to requirements. This four-tier classification provides finer granularity than the typical three-tier (pass/fail/warning) approach used by most accessibility checkers. The monitoring system introduces a novel concept of "accessibility space" where each metric represents a dimension, and changes over time are represented as distance vectors — enabling visualization of both the magnitude and direction of accessibility change. The system supports disability-specific measurement by clustering requirements by the disability types they address, enabling separate "sight-accessibility" or "motor-accessibility" scores. Crucially, metric parameters are designed to be customizable by expert communities, who can assign different weights to different aspects through questionnaires, creating crowd-sourced metric configurations. The prototype was tested against approximately 800 web pages from Italian public institutions with weekly monitoring intervals.
Relevance
Vamolà represents one of the most comprehensive attempts to build a national-scale accessibility monitoring infrastructure tied to specific legal requirements. The integration of validation and longitudinal monitoring addresses a persistent gap: most accessibility evaluation is point-in-time, while Vamolà tracks accessibility dynamics — whether sites are improving or degrading over time. For accessibility practitioners and policymakers, the geographic visualization of accessibility across administrative boundaries provides a powerful tool for identifying regional disparities and targeting intervention. The four-tier error classification, particularly the heuristic detection of fake alt text (checking for strings like "image.jpg" or "photo"), anticipated pattern-based checking approaches now common in modern tools like axe-core. The concept of customizable, community-defined metrics is forward-thinking — accessibility measurement is inherently value-laden (what matters most depends on context), and allowing expert communities to configure metric weights acknowledges this. The collaboration with the AChecker team demonstrates how open-source accessibility tools can be adapted to national legal frameworks, a model relevant as more countries adopt accessibility legislation.
Tags: accessibility metrics · accessibility monitoring · automated testing · Stanca Act · accessibility evaluation · Italian accessibility law · WAB score
Standards referenced: WCAG 1.0 · WCAG 2.0 · Stanca Act · Section 508