Teaching ASL Signs using Signing Avatars and Immersive Learning in Virtual Reality
Lorna C. Quandt, Jason Lamberton, Athena S. Willis, Jianye Wang, Kaitlyn Weeks, Emily Kubicek, Melissa Malzkuhn · 2020 · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2020) · doi:10.1145/3373625.3418042
Summary
This demonstration paper presents SAIL (Signing Avatars & Immersive Learning), the first ASL instructional system built for immersive virtual reality. Developed at Gallaudet University by a majority-Deaf team, SAIL uses a signing avatar — a computer-animated virtual human created from motion capture recordings of a native Deaf signer — to teach introductory ASL vocabulary in a 3D virtual environment. The system is built on Unity 3D and accessed via an Oculus Rift S headset, with a LEAP Motion sensor mounted on the front of the goggles to track the user's hand movements. This allows learners to see digital representations of their own hands from a first-person perspective as they imitate signs demonstrated by the Teacher avatar. Motion capture was performed using a 16-camera Vicon system with 123 body markers, plus a custom-built Faceware rig that allowed the signer to produce signs near the head and face — a technical innovation addressing a common limitation of standard motion capture setups. The avatar was designed to appear humanoid but not uncannily realistic, presenting as a knowledgeable, kind, and professional ASL teacher with emphasis on clearly visible hands and eyes. The initial version teaches 30 ASL signs covering food, everyday items, and basic actions, focusing on signs produced in the neutral "signing space" in front of the body due to LEAP sensor limitations with body-anchored signs.
Key findings
A preliminary usability study with six hearing, English-speaking adults (ages 25-34) with little or no prior ASL knowledge yielded universally positive results — all six participants said they would be interested in using VR to learn ASL. Feedback centred on two areas: timing and corrective feedback. Ten comments addressed the pace of signing, with most recommending more time between signs, leading the team to reduce the avatar's signing speed to 65% of the original. Four participants specifically requested corrective feedback features such as error identification and more interactive back-and-forth with the instructor, noting that the lack of error monitoring made the experience "less natural/believable." Technical limitations included the inability to represent body-anchored signs (like CAT, which touches the cheek) and two-handed signs where hands overlap (like TRAFFIC), which are significant portions of the ASL lexicon. The embodied learning approach — where physically producing signs engages the sensorimotor system — is grounded in research showing that hands-on experience with content increases brain engagement and improves learning outcomes.
Relevance
SAIL represents an important intersection of virtual reality, sign language accessibility, and educational technology. For accessibility practitioners, the project highlights the potential of immersive technologies to address a critical gap: remote ASL instruction for hearing parents of deaf children, who need early exposure to sign language for optimal language development. The system's development by a majority-Deaf team at Gallaudet University — with input from Deaf educators, ASL users, and Deaf team leadership — exemplifies best practices in participatory design with the Deaf community. The technical challenges documented, particularly around representing the full ASL lexicon in VR, point to important limitations that developers must address when building sign language technologies. The emphasis on motion capture from native signers rather than algorithmically generated signs reflects the community's preference for authentic, natural language representation. Future iterations plan to add corrective feedback systems, which could make VR a viable platform for self-directed sign language learning at scale.
Tags: sign language · American Sign Language · virtual reality · signing avatars · motion capture · embodied learning · Deaf accessibility · language learning · gesture tracking