TESSA, a System to Aid Communication with Deaf People
Stephen Cox, Michael Lincoln, Judy Tryggvason, Melanie Nakisa, Mark Wells, Marcus Tutt, Sanja Abbott · 2002 · Proceedings of the Fifth International ACM Conference on Assistive Technologies (Assets 02) · doi:10.1145/638249.638287
Summary
This paper describes TESSA (Text and Sign Support Assistant), an experimental system developed collaboratively by the University of East Anglia, the Royal National Institute for Deaf People, and Televirtual Ltd. to facilitate communication between deaf customers and Post Office clerks. The system translates a clerk's spoken English into British Sign Language (BSL) displayed by a three-dimensional signing avatar. Rather than attempting general-purpose speech-to-sign translation — which would require solving extremely difficult problems in speech recognition, language translation, and sign synthesis simultaneously — TESSA exploits the highly constrained, formulaic nature of Post Office transactions. The clerk's speech is recognised using a constrained grammar network that maps to a finite set of pre-defined phrases, which are then displayed as pre-recorded sign sequences. The avatar is driven by motion capture data recorded from deaf signers wearing sensor gloves and a head-mounted tracking system, with data captured at 30-60 Hz and interpolated into continuous motion streams. The avatar is a full 3D model rendered at 50 frames per second, whose position, pose, size, and even identity can be customised by the user. The system was developed with collaboration from Consignia (the UK Post Office) and formed part of the EU-funded ViSiCAST project.
Key findings
Evaluation with six pre-lingually profoundly deaf participants and three Post Office clerks revealed mixed but encouraging results. For sign quality, average accuracy of identification was 61% for complete phrases and 81% for individual sign units, with considerable variation across participants (42-70% for phrases). Acceptability ratings averaged 2.2 on a 3-point scale, with 30% of phrases rated highly acceptable. Critically, transactions with TESSA took nearly twice as long as without (112 seconds vs 57 seconds, p<0.001), and acceptability ratings were slightly lower with TESSA (1.9 vs 2.6 on a 3-point scale). However, these results must be contextualised: the evaluation used transactions that were already relatively easy without TESSA, clerks had only about an hour of practice, and the participants were all fairly good communicators with reasonable written English skills. Deaf participants provided valuable feedback identifying specific improvements needed: clearer handshapes, better fingerspelling, reduced delay between speech recognition and signing, and improved avatar facial expressions and lip patterns. All deaf participants said avatars would be most useful for complex communication, and all clerks said they would prefer TESSA available as an option when communication became difficult.
Relevance
TESSA represents an important early attempt at real-world deployment of sign language technology, moving beyond laboratory demonstrations to test a system in a realistic service context. The paper's honest reporting of both successes and limitations provides valuable lessons for current sign language technology development. The finding that constrained-domain approaches can produce working systems — even when general-purpose translation remains unsolved — is a pragmatic insight still relevant today. For practitioners, the evaluation highlights that communication technology for deaf users must be assessed not just on technical accuracy but on real-world transaction outcomes, user acceptance, and whether it genuinely improves upon existing communication strategies like writing notes or lip reading. The work also demonstrates the importance of involving deaf users throughout development and evaluation, as their feedback identified issues (facial expressions, signing style, regional variation) that technical metrics alone would miss.
Tags: deaf accessibility · sign language · British Sign Language · sign language avatar · speech recognition · motion capture · translation systems · communication accessibility