South African Sign Language Machine Translation Project
Lynette van Zijl · 2006 · Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '06) · doi:10.1145/1168987.1169031
Summary
This paper describes the South African Sign Language Machine Translation (SASL-MT) project at Stellenbosch University, aimed at increasing the South African Deaf community's access to information by automatically translating English text into South African Sign Language (SASL). The Deaf community in South Africa is marginalised due to a scarcity of sign language interpreters, a lack of published information about SASL, and low literacy rates particularly in disadvantaged communities. The project faced the fundamental challenge that almost no published linguistic data on SASL existed when it began, requiring the team to create linguistic resources from scratch. Data was gathered following standard guidelines, with video recorded by SASL interpreters and transcribed by hand, producing an annotated word list of approximately 800 words and a bilingual English-SASL phrase book — which became the only freely available electronic SASL data source in the country and proved to be in high demand among hearing people learning SASL. The SASL-MT system consists of three components: a linguistics and toolset component, a machine translation component, and an output component using a signing avatar.
Key findings
The machine translation component uses a tree adjoining grammar (TAG) parser for English analysis, reusing grammar definitions from the TEAM project for American Sign Language translation. SASL grammar trees and rule-based transfer rules from English to SASL trees were constructed by hand from a prototype set of sentences. The system handles pronoun resolution to correctly place objects in the signing space — for example, in "Harry eats a chocolate," both "Harry" and later "he" must reference the same position. The team identified co-referential noun phrase resolution as a key challenge for correct spatial placement. For non-manual signs (facial expressions and body posture that are grammatically essential in sign languages), the project used a novel approach based on text-to-speech prosody algorithms — mapping concepts like intonation, pitch, and accent to their sign language analogues of frowning expressions, pitch changes, and emphatic stress. The signing avatar component was developed as a pluggable, re-usable, stand-alone generic avatar based on the H-Anim standard, extended with facial expression capabilities. Animations were developed separately in Java, defined relative to the avatar's size (a shoulder joint moves maximally one-third of the upper arm length), allowing any reasonable avatar to be coupled to the animation code.
Relevance
This project highlights critical accessibility and language justice issues for Deaf communities in the developing world. While sign language translation research has primarily focused on American Sign Language (ASL) and a few European sign languages, SASL and many other national sign languages lack the basic linguistic resources needed for computational work. The project's contribution of creating the first freely available electronic SASL data — and the high demand for it from hearing learners — demonstrates that building linguistic infrastructure is itself a significant accessibility intervention. The approach of adapting prosody-based text-to-speech algorithms for generating non-manual signs is a creative cross-disciplinary insight, recognising the functional parallels between intonation in spoken language and facial expression in signed language. The work also illustrates the broader challenge of machine translation for sign languages: unlike text-to-text translation, sign language output requires generating spatially-organised, time-dependent, multi-channel animations encompassing hand shape, movement, location, facial expression, and body posture simultaneously.
Tags: sign language · machine translation · deaf · South Africa · natural language processing · signing avatar · accessibility · linguistic resources