Manual Evaluation of Synthesised Sign Language Avatars
Robert Smith, Brian Nolan · 2013 · Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2513383.2513420
Summary
This poster paper evaluates whether adding emotional facial expressions to sign language avatars improves their comprehension and naturalness for Irish Sign Language (ISL) users. Sign languages convey emotion and prosody primarily through non-manual features (NMFs) — facial expressions, head movements, and body posture — which carry up to 70% of a sign's meaning. The researchers used the JASigning framework, developed through the EU-funded ViSiCAST and eSIGN projects, which synthesises sign language avatars from HamNoSys phonetic notation converted to SiGML (Signing Gesture Mark-up Language) markup. They augmented the baseline avatar system by creating new facial morphs representing Ekman's seven universal emotions (happiness, sadness, anger, disgust, contempt, fear, and surprise) using the ARPtoolkit, manually adding emotional facial expression markup to each SiGML file. The evaluation compared augmented avatar utterances against baseline versions, focusing on comprehension and naturalness of facial configuration.
Key findings
A manual evaluation was conducted with 15 native ISL users over two days at the Deaf Village of Ireland (DVI), with participants demographically balanced. The study examined the Deaf community's appetite for sign language avatars, the effect of adding emotional facial expressions on comprehension, and whether a human-like avatar offered advantages over a caricature-like one. The evaluation used content from the well-established Signs of Ireland (SOI) corpus as test material. The context of the evaluation is significant: the average reading age of Deaf school leavers is comparable to that of an 8-9 year old hearing child, creating a strong need for communication materials in sign language format. However, producing sign language video content is costly and limited, making synthesised avatars a potentially cost-effective alternative. The research specifically tests whether making avatars more emotionally expressive — and therefore more human-like — improves their usability for the Deaf community.
Relevance
This research addresses a fundamental challenge in sign language accessibility technology: the gap between the expressiveness of human signers and the often flat, emotionless output of sign language avatars. Since non-manual features carry such a large proportion of meaning in sign language, avatars that lack facial expressions are inherently limited in their communicative effectiveness. The work has practical implications for web accessibility, where sign language avatars could provide Deaf users with content in their preferred language without the expense of recording human interpreters for every page. For accessibility practitioners, this study underscores that sign language accessibility is not just about hand movements — facial expressions are integral to the grammar and meaning of signed languages, and any avatar system that neglects them will fail to serve Deaf users effectively. The study is limited by its poster format and small scope, but it contributes to the important ongoing effort to make sign language avatars more natural and usable.
Tags: sign language · sign language avatar · Irish Sign Language · deaf · facial expressions · non-manual features · HamNoSys · SiGML · accessibility