Breaking Barriers with Assistive Macros
André Rodrigues · 2015 · ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility · doi:10.1145/2700648.2811322
Summary
This paper introduces Assistive Macros, an Android accessibility service that enables users with motor impairments to perform multi-step interactions with a single selection. The system addresses two critical problems: the extreme time consumption of sequential scanning navigation, and the complete inaccessibility of app features that require specific gestures with no alternative input method. For switch access users who rely on automatic scanning to navigate interfaces, even simple tasks can require scanning through dozens of elements. The problem worsens when interfaces present less relevant content first, or when buttons lack proper accessibility labels. Some app features (like drawer menus) can only be accessed through swipe gestures—completely blocking users who cannot perform these movements. Assistive Macros offers two macro creation approaches. Manual creation allows users or caregivers to record action sequences, supporting both description-based actions (which identify elements by their accessibility labels and position in the interface hierarchy) and gesture-based actions (which record raw touch coordinates and timing). Description-based macros can adapt to dynamic interfaces, while gesture-based macros enable otherwise impossible interactions like swipe gestures. Automatic creation monitors user interactions and applies the Rabin-Karp string-matching algorithm to detect repeated sequences, prompting users to save frequent patterns as macros.
Key findings
A preliminary case study demonstrated the system's impact with "John," a user with motor, visual, and speech impairments who operates his smartphone using a single-button switch. To play his favorite radio station, John had to scan through eleven options and make four selections in the best case—any mistake could double this time. The play/stop buttons lacked accessibility descriptions, making it impossible for John to know where they appeared in the scanning sequence. After a five-minute explanation, John's parents were able to record macros that reduced this entire process to selecting a single shortcut from the home screen. His mother expressed enthusiasm about creating personalized macros to help her son "live a more independent life." The system works across all applications without requiring developer integration—unlike existing tools like Macrodroid or Tasker that depend on apps implementing specific APIs. Macros can chain unlimited actions, combine description- and gesture-based steps, and cross application boundaries. Macros can be triggered from home screen shortcuts or automatically when launching associated apps.
Relevance
This research addresses a fundamental inefficiency in switch-based mobile interaction that significantly impacts quality of life. For users who can only operate a single switch, every additional scan cycle represents real time and effort—reducing a four-step process to one selection can transform an impossible daily task into a trivial one. The caregiver involvement model is particularly valuable. Family members and friends can record macros tailored to the user's specific needs and routines, democratizing the creation of accessibility solutions. This recognizes that accessibility is often a collaborative effort between users and their support networks. The automatic macro detection feature points toward adaptive accessibility—systems that learn from user behavior and proactively suggest improvements. The proposed future work on crowd-sourced macros could create an ecosystem where accessibility workarounds are shared across the community, providing immediate support when users install new apps. For practitioners, this demonstrates that accessibility solutions can work around poorly designed apps rather than waiting for developers to fix them.
Tags: switch access · motor impairment · mobile accessibility · Android · macros · scanning · assistive technology · automation · caregivers · multi-impairment