Gesture-Based Interactive Audio Guide on Tactile Reliefs
Andreas Reichinger, Anton Fuhrmann, Stefan Maierhofer, Werner Purgathofer · 2016 · ASSETS '16: Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/2982142.2982176
Summary
This paper presents a gesture-controlled interactive audio guide (IAG) that operates directly on tactile relief surfaces using a low-cost depth camera (Intel RealSense F200). Tactile reliefs—2.5D "height field" representations where each point has a specific elevation—offer significant advantages over flat tactile diagrams for blind users, as depth, 3D shape, and surface textures are directly perceivable. However, reliefs alone cannot convey textual information like labels, context, or relationships between elements. The system was prototyped using a tactile relief of Gustav Klimt's "The Kiss" (1908/09), with 20 labeled regions containing audio descriptions (averaging 20 seconds for region names, 50-60 seconds for detailed descriptions). The depth camera, mounted above the relief, tracks hands and detects specific gestures without requiring embedded sensors or stickers on the relief itself. This approach avoids the "Midas touch" problem—where any contact triggers unwanted audio—by requiring distinct intentional gestures. Two gesture categories were implemented: on-object gestures (pointing with a single finger to trigger location-specific audio) and off-object gestures (holding a closed fist above the relief to stop playback, or extending 1-5 fingers to select background information chapters). The system supports hierarchical exploration—users first hear basic region descriptions, after which sub-regions become available for more detailed information.
Key findings
The system was evaluated in two sessions with 20 visually impaired participants total (6 fully blind, 4 with minimal sight, 3 with some residual sight; ages 11-72). Results were strongly positive across multiple dimensions: Users rated the technology as highly meaningful for museums (average 9.5/10), with the IAG helping them better understand the painting (average 9.5/10). General impression of the relief itself averaged 9.2/10, and satisfaction with description texts averaged 9.3/10. Participants praised the system as "super," "perfect," and noted they "felt guided" while maintaining independence to explore at their own pace. A key design validation: users rated the importance of "audio only played when wanted" at 9.5/10, confirming the value of avoiding unwanted audio interruptions during natural tactile exploration. The pointing gesture was learnable (8.8/10 for ease of performing gestures), though some users found it uncomfortable or unnatural—particularly elderly participants whose hands were less flexible. Technical limitations emerged: the silhouette-based hand detection required relatively flat hand positions parallel to the camera, and detection was less reliable at relief edges where hands extended beyond the scanning region. Some users accidentally triggered off-object commands by lifting hands while talking to the examiner. Participants suggested alternative interaction modes including hardware buttons, voice commands, and playback controls (pause, repeat, speed adjustment).
Relevance
This research demonstrates a practical approach to making visual art accessible in museums without modifying the tactile objects themselves. Unlike systems with embedded sensors or attached stickers, the depth camera approach allows the same hardware to work with multiple interchangeable reliefs—important for museums with rotating exhibitions or limited installation budgets. The low-cost sensor hardware (consumer depth cameras now integrated into laptops and mobile devices) makes home and educational deployment feasible. Users expressed interest in using such systems for photo exploration, geography education, object annotation, and educational institutions like zoos. Six of 13 participants said they would purchase the system (~200 EUR) without hesitation. For practitioners, the key design principles are: (1) distinguish intentional interaction from natural exploration to avoid the Midas touch problem; (2) support both quick overview and detailed on-demand information through hierarchical content; (3) allow undisturbed exploration without constant audio; and (4) consider physical constraints—gestures requiring specific hand positions may exclude users with limited dexterity or flexibility. The combination of tactile exploration and interactive audio creates a more complete experience than either modality alone.
Tags: tactile graphics · tactile relief · blind accessibility · depth camera · gesture recognition · audio guide · museum accessibility · finger tracking · interactive systems