Delivering Sign Language in a Live Planetarium Show Using Head-Mounted Displays and Infrared Light
Michael D. Jones, M. Jeannette Lawler · 2019 · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2019) · doi:10.1145/3308561.3353809
Summary
This paper presents a system for delivering real-time American Sign Language (ASL) narration during live planetarium shows using head-mounted displays (HMDs) and infrared light. Planetariums pose unique accessibility challenges for deaf and hard-of-hearing (DHH) audiences: the room is dark, visual content is projected across the dome above, and a sign language interpreter on the floor forces viewers to constantly shift their gaze between the dome and the signer. Projecting the signer onto the dome creates light pollution that washes out dim stars. Written captions do not replace ASL for many DHH viewers, particularly those who are not fluent readers or lack astronomy vocabulary in written English. The system captures the interpreter using an IR-sensitive camera (Canon XA-10 with its IR cut filter removed) illuminated by 850nm IR floodlights — invisible to the naked eye, eliminating light pollution. The video is wirelessly transmitted via RTSP at 480x360 pixels, 30fps, to Epson Moverio BT-200 see-through binocular HMDs worn by viewers. This allows each viewer to see ASL wherever they look on the dome. The design team included three hearing and three DHH members. Technical details were carefully considered: the backdrop used IR-absorbing black velvet (since some black fabrics reflect IR), LED status indicators were taped over to prevent light pollution, and the system achieved 2-5 second latency with a 300ms buffer.
Key findings
Three iterative studies with 29 DHH students (ages 12-18) from schools for the deaf revealed that display brightness relative to the planetarium dome projection is the critical factor determining whether HMDs help or hinder learning. In Study 1 (n=7), the HMD was too bright — participants could not see dim stars, and recalled fewer constellation names with the HMD (20 total) than with the floor signer (37 total), a ratio of 0.54. Red filters were added to the HMD light path for Study 2 (n=15), but recall was still lower with HMDs (31 vs 45, ratio 0.69), with mixed feedback on brightness (5 said too dark, 5 just right, 3 too bright). In Study 3 (n=7), conducted at a planetarium with a brighter projection system, the ratio flipped to 1.75 — participants recalled 14 constellation names with HMDs versus only 8 with the floor signer, suggesting that properly balanced brightness can improve learning. Qualitative findings showed no consensus on ideal signer position within the HMD (4 preferred center, 6 preferred periphery). Five participants found the HMD easier for simultaneous viewing of signs and dome content. Seven disliked the blurry or red-tinted appearance. Four found the glasses physically uncomfortable.
Relevance
This research addresses an underexplored area of accessibility: making informal science learning environments like planetariums inclusive for DHH audiences. The iterative study design is exemplary — each study's findings directly informed the next iteration, leading to a clear understanding that relative brightness between the HMD and environment is the make-or-break variable. This insight applies beyond planetariums to any dark-environment or variable-lighting scenario where sign language or captions must be delivered via HMDs (theaters, concerts, lecture halls with dimmed lights). The IR lighting solution is clever and practical, using commodity security equipment to create an invisible light source. For accessibility practitioners working on venue accessibility, this work demonstrates that simply providing a sign language interpreter is insufficient if the environment prevents effective viewing. The small sample sizes (7 in the critical third study, reduced from 17 due to technical failures) limit the statistical power of the conclusions, but the directional results across three studies are compelling. The broader implication is that wearable display technology can decouple sign language viewing from physical signer placement, potentially transforming access in any setting where sightline conflicts exist.
Tags: sign language · deaf and hard of hearing · head-mounted display · augmented reality · informal learning · museum accessibility · STEM education · ASL