A4U - An Approach to Evaluation Considering Accessibility and Usability Guidelines
Leandro Agostini do Amaral, Renata Pontin de Mattos Fortes, Thiago Jabur Bittar · 2018 · Proceedings of the 24th Brazilian Symposium on Multimedia and the Web (WebMedia 2018) · doi:10.1145/3243082.3264666
Summary
This short paper presents A4U (Ambiente de Analise de Avaliacoes de Acessibilidade e Usabilidade), a web-based environment that integrates accessibility and usability evaluation into a single workflow. The authors argue that the relationship between usability and accessibility is insufficiently explored in evaluation practice, with most tools addressing only one dimension. A4U was built as a PHP/MySQL web application using the MVC pattern, with YAML for responsive CSS and jQuery for JavaScript. The system incorporates two sets of guidelines: WCAG 2.0 success criteria for accessibility and the U.S. Department of Health and Human Services (HHS) guidelines for usability, chosen because they are validated and represent one of the most comprehensive usability guideline sets available. A4U differentiates itself from existing tools (TAW, WaaT, AChecker, Hera) by supporting manual evaluation alongside automated results, allowing import of EARL-formatted reports from other tools, storing evaluator decisions and comments, and crucially, presenting accessibility and usability criteria side by side. The evaluator can register a web page, run an automated analysis via the integrated AChecker tool, import additional EARL reports from other tools, then perform manual inspection on criteria that automated tools cannot fully verify, while simultaneously reviewing usability guidelines in a separate tab.
Key findings
A case study was conducted evaluating the Sao Paulo state government portal (saopaulo.sp.gov.br), chosen due to Brazilian law (Decreto Lei 5.296 of 2004) requiring accessibility on government websites. Ten participants took part: three web evaluation researchers (two professors and a doctoral student from the Federal University of Goias and a master's student from USP) and seven software developers from a company in Sao Carlos. After using A4U, participants completed a Likert-scale questionnaire. The study revealed that inexperienced developers had significant difficulty understanding the descriptions of WCAG and HHS guidelines, finding them too technical or abstract. The main cost of the integrated approach was the high number of criteria to check, leading to excessive evaluation time. However, participants rated the experience of considering accessibility and usability together in the same environment as positive. At least one participant reported perceiving a conflict between accessibility and usability decisions during evaluation, suggesting the integrated view successfully surfaced tensions between the two quality dimensions that separate tools would miss.
Relevance
This paper addresses a persistent gap in web evaluation practice: the artificial separation between accessibility and usability testing. While accessibility practitioners focus on WCAG compliance and usability professionals run separate heuristic evaluations, the two dimensions are deeply intertwined — an accessible interface can be unusable, and a highly usable interface may be inaccessible. A4U's approach of presenting both sets of criteria in a unified environment, while modest in scope, points toward how evaluation tools should evolve. The finding that developers struggled with guideline descriptions highlights an ongoing challenge in the field: WCAG success criteria remain difficult for non-specialists to interpret and apply. The Brazilian legal context (mandatory government website accessibility) provides a useful comparison point for practitioners working under similar mandates. A limitation is the paper's brevity and the small study size, making it more of a proof-of-concept than a validated tool.
Tags: accessibility testing · usability testing · integrated evaluation · WCAG · HHS guidelines · web evaluation tools · semi-automatic testing
Standards referenced: WCAG 2.0 · HHS Usability Guidelines · EARL · WAI-ARIA