Leveraging Augmented Reality to Create Apps for People with Visual Disabilities: A Case Study in Indoor Navigation
Chris Yoon, Ryan Louie, Jeremy Ryan, MinhKhang Vu, Hyegi Bang, William Derksen, Paul Ruvolo · 2019 · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2019) · doi:10.1145/3308561.3353788
Summary
This paper presents Clew, an iOS app that leverages Apple’s ARKit augmented reality framework to help people who are blind or visually impaired (B/VI) navigate indoor environments — a setting where GPS is unreliable (accuracy drops to ~5m+ indoors). Clew uses visual-inertial odometry (VIO), which fuses camera-based visual tracking with gyroscope and accelerometer data, to estimate the phone’s 3D position at 60Hz. The app has two core functions: recording a route by laying down virtual "breadcrumbs" as the user walks, and later navigating that route in either forward or reverse direction with automatic guidance via speech ("continue straight for 10 feet"), haptic feedback, or spatial audio pointing toward the next waypoint. Routes can be saved and reloaded using ARKit’s visual relocalization feature, which matches current camera imagery against a stored 3D map of visual landmarks. When visual alignment fails (e.g., lighting changes), a physical alignment backup uses phone landmarks placed against flat surfaces like doors. The app was developed through a user-centered design process with four B/VI co-designers over weekly sessions spanning months, and three members of the research team themselves had visual impairments. Clew was released on the iOS App Store and downloaded over 5,000 times from 50 countries, with an average of 60 daily users at the time of publication.
Key findings
Benchmarking tests showed ARKit’s position tracking was accurate to under 1 metre of mean error even on routes up to 63m long, including those with staircases — sufficient for practical indoor navigation guidance. Large-scale analysis of 5,789 routes from real users revealed several important findings. The baseline positive rating was 68%, but multiple factors significantly influenced user experience: phone angle was critical — users holding the phone at more than 72 degrees from horizontal (i.e., too flat) experienced statistically significant drops in satisfaction, because VIO algorithms work best when the camera tracks visual features at varied depths; tracking errors ("insufficient visual features" or "excessive motion") had a steep negative impact, with four or more errors dramatically reducing positive ratings; and routes between 9-15 metres had the highest satisfaction, with longer routes (>33m) performing worse. A key co-design insight emerged from working with George, a congenitally blind user who could not maintain phone-to-body alignment — he could rotate his body to face the correct direction but not independently align his phone. This led to developing a body-direction offset algorithm that provided guidance relative to the user’s body rather than phone orientation. The paper also found that low-vision users were a significant user group despite the app initially being designed only for blind users, and that B/VI users deliberately avoided iOS updates due to accessibility-breaking bugs.
Relevance
Clew demonstrates that mainstream smartphone AR technology — available at no cost on any modern iPhone — can meaningfully address indoor O&M challenges without requiring environmental modifications like Bluetooth beacons, RFID tags, or special signage. This is a significant advantage for scalability: any indoor space becomes navigable without infrastructure investment. The design guidelines offered are valuable for any researcher building AR-powered assistive apps: provide feedback to maintain vertical phone orientation, design for multiple levels of vision (not just blindness), consider differing spatial processing abilities (especially between congenitally and adventitiously blind users), invest in internationalization early, and support older iOS versions since accessibility users avoid upgrades that may break VoiceOver. The finding that only 30% of working-age Americans who are B/VI are employed, but that better O&M skills correlate with higher employment, underscores the real-world stakes of indoor navigation technology. The paper is also notable for its distributed co-design methodology — releasing the app globally and treating all users as potential co-designers through feedback and usage data analysis.
Tags: augmented reality · indoor navigation · orientation and mobility · blind · low vision · visual inertial odometry · ARKit · mobile accessibility · wayfinding · haptic feedback