Empowering Accessible Gesture Input Design with Gesture-A11Y
Mihail Terenti, Laura-Bianca Bilius, Ovidiu-Ciprian Ungurean, Radu-Daniel Vatavu · 2025 · Proceedings of the 22nd International Web for All Conference (W4A) · doi:10.1145/3744257.3744267
Summary
This paper from Stefan cel Mare University of Suceava (Romania) introduces Gesture-A11Y, the first open-access web-based tool and database for accessible gesture input design, containing over 22,000 gesture records collected from 122 users with visual and/or motor impairments. The tool addresses a critical gap: open gesture datasets are notoriously scarce in HCI, and those representing users with disabilities are even rarer, hindering both replication of research and practical development of accessible gesture interactions. Since prevalent mobile devices rely primarily on gesture input, gesture accessibility directly determines overall mobile accessibility and can impact employment. The database covers four gesture modalities: touchscreen stroke gestures (2D paths on smartphones and wearables), mid-air motion gestures (3D movements captured as linear accelerations), on-body gestures (touch on the torso or limbs), and on-wheelchair gestures (interactions with armrests or wheelchair surfaces). Records include computational representations (x,y coordinate series for touchscreen; x,y,z acceleration series for motion), user-defined gesture-to-function mappings, and self-reported preference ratings (ease of use, recall, goodness of fit, social acceptability). User demographics include specific motor symptoms (tremor, spasm, rapid fatigue, difficulty gripping), visual acuity levels (degrees of myopia/hyperopia), medical diagnoses, and standardized WHODAS 2.0 disability scores.
Key findings
The database contains 4,662 stroke gestures from 35 people with motor impairments and 3,313 from 27 people with visual impairments on smartphones; 7,290 stroke gestures from 14 people with motor impairments on wearables (smartwatch, smart ring, smart glasses); 3,809 mid-air motion gestures from 14 participants with motor impairments; 231 user-defined gesture-function mappings with 924 associated ratings from 11 wheelchair users; and 2,625 wearable device preference ratings from 21 people with motor impairments. The paper demonstrates notable differences in gesture articulation between users with and without impairments — for example, touchscreen gestures for a five-point star by a user with congenital nystagmus and high myopia show dramatically higher variation in geometry and structure compared to a user without impairments, requiring fundamentally different recognition algorithms. Baseline analysis of the visual impairment stroke dataset shows average production times of 3.8s (SD=3.9) and path lengths of 24.6cm (SD=14.5), with Nearest-Neighbor recognition accuracy reaching 92.8% with sufficient training data. Practical use cases include identifying suitable gestures for accessible smart TV interfaces and evaluating gesture recognition algorithms against disability-representative data.
Relevance
Gesture-A11Y fills a fundamental infrastructure gap in accessible interaction design. Without representative gesture data from users with disabilities, developers default to designing for typical motor and visual abilities, producing interfaces that may be technically functional but practically unusable for significant populations. The tool enables evidence-based gesture design by letting practitioners search by specific impairment symptoms (e.g., "difficulty gripping" + "rapid fatigue") and immediately see which gestures users with those characteristics can perform, prefer, and find socially acceptable. For accessibility practitioners, the key insight is that gesture recognition algorithms trained on data from users without impairments may fail for users with disabilities due to higher geometric and temporal variation in gesture articulation — a problem that can only be addressed with representative training data. The vision of integrating Gesture-A11Y with design tools like Figma plugins to automatically suggest accessible gestures when new UI elements are added could transform how accessible mobile and wearable interfaces are designed. The open-source, open-access licensing supports democratization of accessible gesture design including DIY assistive technology.
Tags: gesture interaction · motor accessibility · visual impairment · mobile accessibility · wearable technology · open data · input methods · wheelchair accessibility
Standards referenced: WHODAS 2.0