AIGuide: Augmented Reality Hand Guidance in a Visual Prosthetic
Sooyeon Lee, Chien Wen Yuan, So-yeon Yoon, John M. Carroll · 2022 · ACM Transactions on Accessible Computing · doi:10.1145/3508501
Summary
AIGuide is an iOS smartphone application designed as a "visual prosthetic" to help people with visual impairments locate and physically reach objects in their environment. The app addresses what the authors call the "last meter problem"—while existing object detection apps like Seeing AI can identify objects, they don't guide users' hands to actually pick them up. AIGuide uses Apple's ARKit framework to detect objects through the phone's camera and track their 3D position in real-time. The app provides multimodal feedback (speech for direction, sound/beeps for distance, and haptic vibrations) to guide the user's hand toward the target object. The guidance process has four phases: selection (choosing what to find), localization (scanning for the object), guidance (hand movement toward object), and confirmation (verifying the correct object is found). A key technical innovation is that AIGuide maintains object position even when the target leaves the camera frame, allowing more flexible phone holding. The app is self-contained—requiring no internet connection or external hardware—making it portable and usable anywhere.
Key findings
The researchers conducted a remote user study with 10 participants who are blind (9 totally blind, 1 legally blind), testing three feedback modes: sound-only, haptic-only, and combined sound+haptic. Task completion times averaged 20-28 seconds across conditions. Sound-only feedback showed the fastest performance (though not statistically significant), while combined sound+haptic was most preferred by participants—a classic performance-versus-preference gap. Video analysis revealed practical challenges: participants struggled with one-handed operation while walking, needed to hold the phone carefully to avoid blocking the camera, and found the confirmation phase (shaking the phone) unintuitive. Eight of ten participants found the distance and direction information highly useful. Participants expressed enthusiasm about generalizing AIGuide beyond grocery items to clothing, medication, lost keys, and misplaced household objects. They strongly preferred smartphone-based solutions over specialized devices, citing portability and avoiding stigmatization as key factors.
Relevance
This research advances assistive technology by bridging the gap between object recognition and physical interaction—a critical missing link in current smartphone accessibility apps. The study offers practical design guidance for hand-guidance interfaces: use multimodal feedback with different modalities for different information types (sound for distance, speech for direction), design for one-handed operation, and consider wearable accessories for haptic feedback. The finding that users preferred combined feedback despite better performance with sound-only underscores that accessibility features must account for user comfort and confidence, not just raw efficiency. The participants' strong preference for mainstream smartphone technology over specialized devices reinforces the importance of building accessibility features into everyday devices. Future work could incorporate machine learning for generic object detection and conversational interfaces.
Tags: augmented reality · visual impairment · object detection · hand guidance · multimodal feedback · haptic feedback · smartphone accessibility · assistive technology
Standards referenced: VoiceOver