Evaluation of Assistive Technologies from the perspective of Usability, User Experience and Accessibility: a Systematic Mapping Study
Tatiany Xavier de Godoi, Guilherme Corredato Guerino, Natasha Malveira C. Valentim · 2021 · Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (IHC) · doi:10.1145/3472301.3484323
Summary
This systematic mapping study (SMS) investigates how assistive technologies (ATs) are evaluated from the perspectives of usability, user experience (UX), and accessibility — three interrelated but distinct software quality factors. The authors searched Scopus, ACM Digital Library, and IEEEXplore using a structured PICOC-based search strategy, retrieving 368 papers and selecting 112 after two filtering stages. The study was motivated by the well-documented problem of AT abandonment: users discard ATs due to handling difficulties, high cognitive load, need for follow-up service, and poor device quality. The researchers argue that evaluating usability, UX, and accessibility together — rather than in isolation — would help identify more failures and better meet users' requirements. They identified 33 distinct evaluation instruments across the 112 papers, cataloguing eight key instruments in detail: QUEST 2.0 (Quebec User Evaluation of Satisfaction with Assistive Technology), ATD-PA (Assistive Technology Device Predisposition Assessment), NASA-TLX, VAS (Visual Analog Scale), PYTHEIA, PIADS (Psychosocial Impact of Assistive Devices Scale), SUS (System Usability Scale), and PDQ-39 (Parkinson's Disease Questionnaire). The study classified each paper by evaluation type (usability, UX, or accessibility), evaluation method, disability type addressed, user age range, AT category, platform, tool support, evaluation environment, and analysis type.
Key findings
The most striking finding was that none of the 112 studies evaluated ATs from all three perspectives — usability, UX, and accessibility — together. Only 1.92% (N=4) considered even two perspectives (usability and accessibility). Usability evaluation dominated at 84.62% of instruments, with UX at only 7.21% and accessibility at just 6.25%. The most common evaluation method was inquiry (questionnaires and interviews) at 62.80%, followed by usability testing at 16.91%, simulation at 4.83%, and inspection at 2.42%. Physical disability was the most frequently addressed (45.11%), followed by visual impairment (18.80%), hearing impairment (6.77%), and multiple disabilities (4.51%) — with notably few evaluation instruments designed for deaf users. Most evaluations involved adults (33.14%), with children significantly underrepresented (5.71%). The predominant AT categories were computer accessibility features (29.60%) and mobility aids (16%). Embedded platforms accounted for 42.19% of ATs, reflecting the trend toward smart objects in daily life. Only 10.71% of papers had tooling support for their evaluations, and 89.29% had none, suggesting significant opportunity for developing evaluation support tools. Most evaluations occurred in hospitals (17.19%), homes (17.19%), or labs (17.19%), with industry settings at just 1.56%.
Relevance
This study exposes a fundamental gap in how the assistive technology field evaluates its own products: the three quality factors most relevant to AT adoption — usability, UX, and accessibility — are almost never assessed together. This siloed approach means that an AT could score well on usability metrics while being inaccessible to some of its intended users, or could be technically accessible but provide such a poor user experience that it gets abandoned. The finding that most evaluation instruments are generic rather than AT-specific is concerning, as the unique contexts of disability — varying physical capabilities, cognitive loads, and interaction patterns — demand specialized evaluation approaches. For practitioners, the catalog of eight well-described evaluation instruments provides a practical toolkit for AT assessment. The near-absence of evaluation instruments for deaf users highlights an underserved area. The study's call for integrated evaluation frameworks that address usability, UX, and accessibility simultaneously points toward a more holistic approach to AT quality that could help reduce the persistent problem of technology abandonment.
Tags: assistive technology · usability · user experience · accessibility evaluation · systematic review · research methodology · AT abandonment · evaluation instruments · quality assurance
Standards referenced: ISO 9241-210