How Do People with Limited Movement Personalize Upper-Body Gestures? Considerations for the Design of Personalized and Accessible Gesture Interfaces
Momona Yamagami, Alexandra A Portnova-Fahreeva, Junhan Kong, Jacob O. Wobbrock, Jennifer Mankoff · 2023 · Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '23) · doi:10.1145/3597638.3608430
Summary
This study investigates how people with upper-body motor impairments design personalized gestures for common device functions like move, select, zoom-in, open, and delete. The researchers recruited 25 participants with conditions including spinal cord injury (13), muscular dystrophy (3), peripheral neuropathy (3), essential tremor (2), and other disabilities (4), and asked each to create their own gesture for 10 standard device functions. To capture the full range of movement, participants wore inertial measurement unit (IMU) and electromyography (EMG) sensors distributed across their shoulders, upper arms, forearms, and fingers. The study combined qualitative video analysis of gesture design choices with quantitative analysis of sensor differentiability, muscle activation heterogeneity, and gesture agreement scores. The methodology built on the user-defined gesture elicitation paradigm but deliberately focused on characterizing individual personalized gesture sets rather than deriving a unified gesture vocabulary, recognizing that the diversity of abilities among participants would make a one-size-fits-all approach inappropriate. Participants were prompted with device-agnostic animations to avoid biasing gestures toward any particular form factor.
Key findings
The results demonstrate that personalized gestures are highly ability-specific, with striking variation even among participants with the same disability type. Gesture agreement between participants was very low (0.11 to 0.37 across functions), confirming that a unified accessible gesture set would be impractical. Eight percent of gestures used the head, neck, or shoulders rather than hands and fingers, showing that gesture systems must track the whole upper body. Fifty-one percent of gestures were performed with hands resting on or barely lifting off the armrest, challenging the common assumption that mid-air gestures require raising arms to chest level. Ten percent of gestures were isometric — participants activated muscles without visible movement — meaning motion-based sensors like cameras would miss them entirely, while EMG sensors could detect them. EMG analysis revealed that even participants with the same disability activated very different muscle groups, with no consistent pattern within disability types. Both IMU and EMG sensors showed statistically significant differentiability between personalized gestures (p < 0.0001), suggesting wearable sensor fusion is a viable path for personalized gesture recognition. The study proposes adding isotonic and isometric gestures as subcategories to existing gesture taxonomies.
Relevance
This research has direct implications for the design of gesture-based interfaces in smartphones, smart homes, augmented and virtual reality, and wearable devices. The three core design recommendations — track the whole upper body, use sensing mechanisms agnostic to body position and orientation, and incorporate sensors that detect muscle activation without movement — provide a practical framework for making gesture systems accessible. For accessibility practitioners, the key insight is that personalization is not optional but essential: standardized gesture sets inherently exclude users whose abilities do not match the assumed movement patterns. The finding that participants' gestures evolved during the study, moving from mimicking touchscreen interactions to more ability-appropriate movements, suggests that gesture elicitation processes should allow time for users to explore beyond familiar paradigms. The work also highlights the potential of EMG sensors as a complement to IMU and camera-based tracking for capturing the full range of human gestural expression.
Tags: motor impairments · gesture input · personalization · wearable sensors · electromyography · spinal cord injury · muscular dystrophy · assistive technology · user-defined gestures