Gesture-A11Y: A Large-Scale Hub for Accessible Gesture Input
Mihail Terenti, Laura-Bianca Bilius, Ovidiu-Ciprian Ungurean, Radu-Daniel Vatavu · 2025 · Proceedings of the 22nd International Web for All Conference (W4A 2025) · doi:10.1145/3744257.3744280
Summary
This paper introduces Gesture-A11Y, a large-scale, open-source, web-based hub that serves as a searchable database of gesture input data collected from users with visual or motor disabilities. The tool is the culmination of an eight-year research effort following a four-stage roadmap: reviewing the state of the art to identify gaps in open gesture datasets for users with impairments, conducting gesture collection studies with users who have conditions such as Spinal Cord Injury, Multiple Sclerosis, Cerebral Palsy, Parkinson's, and Traumatic Brain Injury, curating and analyzing the collected gesture data, and finally designing and developing the web-based platform. The database contains over 22,000 records spanning four gesture types: touch strokes (screen coordinates with timestamps), motion gestures (accelerometer data from finger, wrist, and head movements), on-body gestures (hand gestures performed on the body for remote control), and on-wheelchair gestures. Data was collected across multiple devices including mobile phones, smartwatches, glasses, and rings, in both private and public settings. The platform was built with Vue.js on the frontend, Golang for backend operations, and Pandas for data processing, with search filters based on keyword, gesture type, impairment, and device type.
Key findings
Gesture-A11Y indexes 22,854 gesture records from a total of 133 users across seven distinct datasets. The largest dataset contains 7,290 touchscreen stroke gesture records from 14 users with motor impairments performing gestures on smartwatches, glasses, and rings. Another significant dataset includes 3,313 touchscreen stroke gesture records from 27 users with visual impairments. The tool enables practitioners to identify gesture sets aligned with specific impairments (such as spasm or tremor), conduct comparative analyses across gesture and device types, and train AI models on the data for generative interaction design in accessibility. The database includes diverse data types: numerical screen coordinates, accelerometer readings, textual descriptions with manually extracted attributes, and self-reported ratings on perceived ease of execution, social acceptability, and goodness of fit. The authors highlight the tool's potential for accessible employment, arguing that since mobile device interactions rely heavily on gesture input, improving gesture accessibility directly contributes to inclusive access to digital resources and services.
Relevance
Gesture-A11Y addresses a critical gap in accessible interaction design: the lack of open, comprehensive gesture data from users with disabilities. Practitioners designing gesture-based interfaces have historically lacked empirical data on how users with various motor or visual impairments actually perform gestures, leading to designs based on assumptions rather than evidence. This tool enables evidence-based gesture set design by allowing designers to search for gestures that match specific ability profiles and device contexts. For organizations developing mobile or wearable applications, the database provides a practical resource for validating whether proposed gesture interactions are feasible and preferred by users with disabilities. The open-source, open-data approach also supports reproducibility in accessibility research and democratizes access to resources that were previously siloed across individual research groups.
Tags: gesture input · motor disabilities · visual disabilities · open data · accessible interaction design · touchscreen gestures · wearable devices