The Impact of Accessibility Assessment in Macro Scale Universal Usability Studies of the Web
Rui Lopes, Luís Carriço · 2008 · Proceedings of the 2008 International Cross-Disciplinary Conference on Web Accessibility (W4A 2008) · doi:10.1145/1368044.1368048
Summary
This paper introduces Web Interaction Environments (WIEs), a modelling framework for studying universal usability of the web at large scale. The authors argue that existing approaches to accessibility evaluation treat audiences as homogeneous groups, failing to capture the synergies and differences between distinct user populations. A WIE is defined as a particular audience's group of intrinsic characteristics upon which tailored evaluation procedures can be applied. The framework organizes characteristics into four domains: Users (diversity characteristics like disabilities, age, culture), Devices (input/output modalities, browser capabilities), Usage Situations (environmental factors like lighting, noise, connectivity), and User Intentions (information seeking, transactions, communication). The authors created an OWL ontology with over 130 concepts to provide a formal vocabulary for describing WIEs, drawing partially from the WHO's International Classification of Functioning, Disability and Health (ICF) while extending it to cover web-specific concepts like color blindness types that the ICF lacks. Using this framework, they expressed the implicit audience model within WCAG 1.0 and conducted a large-scale accessibility study of Wikipedia, crawling 7,791 webpages from 100 randomly selected starting pages and following all hyperlinks one level deep.
Key findings
The Wikipedia study revealed that only about 10% of pages fully complied with the 14 automated WCAG 1.0 checkpoints tested, with an average accessibility quality score of 84.6%. A critical finding was the difference between internal and external pages: Wikipedia's internal pages averaged 89.79% accessibility quality compared to 81.83% for externally linked pages. The ratio of fully compliant pages was 2:1 (internal to external). Notably, no parse errors were found in Wikipedia's internal pages, while external pages had a 13.75% error rate. This demonstrates that Wikipedia's template mechanism provides a "baseline quality" — users can create content without accessibility expertise because templates enforce well-structured markup. However, linking to external sites introduces an uncontrolled accessibility degradation, since users have no awareness that their hyperlinks may lead to less accessible content. The authors also identified a fundamental limitation of WCAG: guidelines are "black-boxed," treating all audiences as a single group with no mapping between specific checkpoints and the audience characteristics they serve, making it impossible to assess accessibility quality for particular user groups.
Relevance
This paper makes an important theoretical and practical contribution to how we think about accessibility evaluation at scale. The WIE framework anticipates modern concerns about the gap between guideline compliance and actual user experience — the idea that passing WCAG checks does not necessarily mean a site works well for any specific disability group. For practitioners managing large content platforms (CMSs, wikis, knowledge bases), the finding about template-driven accessibility is directly actionable: well-designed templates can enforce baseline accessibility even when content authors lack accessibility knowledge, but outbound links remain a vulnerability. The criticism of WCAG as "black-boxed" remains relevant today, as organizations still struggle to prioritize which guidelines matter most for their specific user populations. The framework's approach to formally modelling audience characteristics could inform more targeted, user-centered accessibility testing strategies.
Tags: universal usability · audience modelling · web accessibility · WCAG evaluation · Wikipedia · large-scale study · ontology · accessibility metrics
Standards referenced: WCAG 1.0 · UWEM · ICF