Making Mobile Augmented Reality Applications Accessible
Jaylin Herskovitz, Jason Wu, Samuel White, Amy Pavel, Gabriel Reyes, Anhong Guo, Jeffrey P. Bigham · 2020 · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2020) · doi:10.1145/3373625.3417006
Summary
This paper tackles a fundamental accessibility gap in augmented reality: AR applications are overwhelmingly visual, mixing virtual 3D objects with the real world in ways that existing screen readers and accessibility services cannot interpret. The researchers analyzed 105 existing iOS AR apps from Apple's App Store across five categories (Entertainment 39%, Education 31%, Retail 16%, Utility 9%, Other 5%) and identified a taxonomy of five core "building-block" tasks common across AR applications: (1) observing AR content — perceiving virtual objects and their relationships to each other and the physical world; (2) establishing physical/virtual correspondence — scanning environments to detect surfaces (80% of apps), scanning specific objects (10.5%), or marking locations (11.5%); (3) creating virtual content — placing objects at points (62%), drawing (10.5%), or marking vertices (4%); (4) transforming virtual content — moving (50.5%), rotating (51.5%), resizing (45%), or deleting (16%) objects; and (5) activating virtual content — selecting objects (35%) or triggering effects like animations and sounds (48.5%). For the three most common task categories, the team built five prototype accessible alternatives on iOS using ARKit, plus two complete app prototypes (a furniture retail app and a solar system educational app) that combined the building blocks into realistic experiences.
Key findings
A user study with 10 blind participants (8 blind, 2 low vision, ages 30-79, all VoiceOver users) demonstrated that the prototypes made AR interactions possible for blind users across all five tasks. Key design solutions included: a "freeze" feature that captures a stable view so users can explore without holding the phone steady; transforming 3D AR objects into 2D VoiceOver-accessible elements by exposing SceneKit bounding boxes to the accessibility view hierarchy; verbal notifications during environment scanning announcing surface types and total area; camera-based placement tying virtual objects to the phone's physical position; guided placement offering step-by-step location choices (floor/table, center/edge/corner); camera-based and guided search methods for finding virtual objects with distance and direction feedback. For the furniture app, participants spent an average of 5.8 minutes and generally found it easy to use after initial adjustment. For object search, guided search (4.1 min average) was preferred over camera search (5.2 min) for unknown objects, consistent with prior VizLens findings. Important insights emerged about how blind users perceive "virtual" objects — some participants who had prior vision drew meaningful parallels to mental maps, while congenitally blind participants found spatial concepts harder to grasp. Participants wanted richer contextual descriptions including awareness of physical objects, safety information, and spatial relationships.
Relevance
This paper is essential reading for anyone building AR applications or developing accessibility standards for immersive technologies. As AR becomes ubiquitous in retail, education, and entertainment, the accessibility gap will only widen without proactive intervention. The taxonomy of AR tasks provides a concrete framework developers can use to audit their own apps for accessibility barriers. The prototype solutions demonstrate that AR can be made non-visually usable with current technology — the "freeze" feature and 3D-to-2D accessibility bridge are immediately implementable patterns. The participatory insight that blind users need different interaction methods depending on context (guided vs. exploratory) and that spatial understanding varies significantly based on vision history challenges one-size-fits-all approaches. A key limitation is that prototypes only addressed VoiceOver users on iOS and did not explore Switch Control or other access methods. The authors note the accessibility community is often in a position of "playing catch up" with new technologies and advocate for building accessibility into AR platforms from the start.
Tags: augmented reality · visual accessibility · blindness and low vision · mobile accessibility · screen readers · VoiceOver · inclusive design · emerging technology
Standards referenced: WCAG