AuslanSpell: An Interactive Technology for Improving Auslan Fingerspelling Comprehension
Kalin Stefanov, Andre Ky Pham, Antony Smith Loose, Lucy M Robertson-Bell, Louisa Jane Vaughan Willoughby · 2026 · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26) · doi:10.1145/3772318.3791563
Summary
AuslanSpell is an interactive learning tool that converts arbitrary English text into 3D motion-captured animations of Australian Sign Language (Auslan) fingerspelling, targeting hearing learners who struggle with the hardest part of sign-language learning: reading fingerspelling back. Auslan uses a two-handed manual alphabet, so the authors could not reuse ASL resources, and the existing Auslan Signbank offered only static images. The team collected the first large-scale high-quality 3D Auslan fingerspelling dataset, capturing 8 Deaf signing models across 93 carefully chosen words using Manus Prime 3 mocap gloves for handshape and a Vicon system for upper-body motion. Animations were cleaned, retargeted in Blender via the Rokoko plugin, and validated by Deaf signers. The application was built through a co-design partnership with 5 experienced Deaf Auslan teachers plus representatives from Deaf Australia and major Auslan teaching providers, framed explicitly as not a 'disability dongle' but as technology for hearing learners endorsed by the Deaf community. Core features include arbitrary text input, smooth letter-to-letter transitions via a custom key-interval animation engine with finger-collision avoidance, rotatable 3D view, variable transition duration, adjustable speed (0.25x-1.5x), left/right-handed models, and simple vs realistic hand meshes. The app ships free on iOS, iPadOS, macOS, web, and via Auslan Signbank, with open source on GitHub.
Key findings
In a 45-minute user study with 33 novice signers (Monash Linguistics students), participants scored well above chance on beginner fingerspelling readback tasks after only 15 minutes with the app. On a 10-item multiple-choice test they averaged 8.2/10 (SD 1.7), with 9 participants scoring perfectly. On the harder 10-item free-text transcription test they averaged 3.4/10, and error analysis showed a strong salience pattern: first letters were correctly identified by 80.6% of responses and last letters by 73.9%, but middle letters by only 34.2% - a statistically significant drop (p<.001). Errors were not random; learners confused letters with similar handshapes (M vs N, noisy vs nosey, complete vs compete). Word length did not correlate with error rate (r=-0.11). System Usability Scale scores averaged 81.2 (median 84) indicating high usability. Feature ratings showed adjustable speed and rotatable 3D view as standout features; swapping between simple and realistic hands was less valued. Participants found the slowest speed (0.25x) appropriate but 42% rated the fastest setting (1.5x, 2.23 letters/second) as too fast, consistent with fluent ASL rates exceeding 6 letters/second.
Relevance
For accessibility practitioners, AuslanSpell is a model of how to build signing-avatar technology responsibly: Deaf educators led the design, signing models were fluent and compensated, animation output was validated against citation forms rather than naturalistic variation that novices cannot yet interpret, and the tool targets a real training gap (interpreter shortage in Australia) rather than attempting to replace human signers. The technical decisions - motion capture over rule-based or diffusion-model generation, explicit key-interval constraints to preserve articulation accuracy, collision-avoidance for finger transitions - are practical guidance for anyone building sign-language learning tools. Limitations: the study tested only novices (and only for readback, not production), the signing corpus is limited to 93 words worth of transitions, and the current animation range is citation-form rather than naturalistic, which will matter more as learners advance. The open dataset and open-source web package lower the barrier for extending the approach to other sign languages.
Tags: Auslan · fingerspelling · sign language · signing avatar · language learning · participatory design · educational technology · deaf accessibility