User tests demonstration: real experiences in measuring web accessibility needs for people with disabilities and the elderly
Jesus Hernandez Galán, J. Ángel Martinez Usero, M. Jesús Varela Méndez · 2009 · Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibililty (W4A) · doi:10.1145/1535654.1535677
Summary
This workshop paper from the ONCE Foundation, CIDAT, and Technosite in Spain describes a practical demonstration of user testing with people with disabilities and elderly participants as part of web accessibility evaluation. Set within the European policy context of the 2005 eAccessibility Communication, the 2006 Riga Declaration, and the i2010 e-Inclusion initiative, the paper argues that despite substantial policy efforts, the majority of European websites — both public and private — still fail to comply with basic accessibility guidelines. The MeAC (Measuring Progress of eAccessibility in Europe) study confirmed that people with disabilities continue to face many barriers to everyday ICT and web services. The authors present a three-phase evaluation methodology: first, analyzing websites against WCAG 2.0 and the European UWEM methodology (complemented by ONCE Foundation's own MEWA methodology); second, designing procedures based on user test findings; and third, establishing ongoing monitoring and review strategies. The assessment extends beyond web pages to include non-web documents such as Flash, PDF, Word, RTF, ODF, and multimedia content.
Key findings
The methodology combines three evaluation methods: automated testing (using the TAW tool), expert testing, and real user testing with controlled observation. For page selection, the approach samples representative pages (home, sitemap, contact, search), technically representative pages (containing forms, data tables, JavaScript, frames, image maps), topically representative pages, and randomly selected pages. The user testing involves a diverse panel of seven user profiles: a blind person, a partially-sighted person, a deaf person, a person with limited hand movement, a person with cognitive disability, an older person, and a person without disabilities. During controlled testing, users perform predefined tasks while an expert observer documents difficulties, users provide real-time impressions, and post-session satisfaction questionnaires capture accessibility and usability ratings. The paper emphasizes that including disabled and elderly users in the evaluation process provides knowledge about non-compliance that automated and expert methods alone cannot capture, giving evaluators direct evidence of how real users experience accessibility barriers.
Relevance
This paper documents an important practical methodology for integrating real user testing into accessibility evaluation — an area where many organizations still rely solely on automated tools and expert review. The ONCE Foundation's approach of including seven distinct user profiles covering visual, hearing, motor, cognitive, and age-related disabilities provides a model for comprehensive user testing that goes beyond the common practice of testing only with screen reader users. The European policy context described — with its emphasis on harmonized evaluation methodologies and the challenge of transitioning from WCAG 1.0 to 2.0 — mirrors challenges that organizations worldwide faced during that period and continue to face with each standards update. For practitioners, the structured page sampling strategy (representative, technical, topical, and random) offers a practical framework for selecting test pages that balances coverage with feasibility. The inclusion of non-web document formats (PDF, Word, Flash) in the evaluation scope acknowledges that real-world accessibility extends beyond HTML web pages — a concern that has only grown more pressing as organizations publish content across multiple formats.
Tags: user testing · accessibility evaluation · aging · European accessibility policy · UWEM · usability testing · disability
Standards referenced: WCAG 1.0 · WCAG 2.0 · UWEM