Brazilian Sign Language Multimedia Hangman Game: A Prototype of an Educational and Inclusive Application
Renata Cristina Barros Madeo · 2011 · The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2049536.2049623
Summary
This doctoral consortium paper presents a prototype educational game — a hangman game that uses Brazilian Sign Language (BSL/Libras) as its input method. Instead of typing letters on a keyboard, players perform fingerspelling signs from the Libras manual alphabet in front of a webcam, and a computer vision module recognizes which letter they are signing. The game serves two educational purposes: encouraging both deaf and hearing children to learn Libras, and helping deaf children improve their Portuguese vocabulary by associating written words with their sign language equivalents. The system architecture consists of a Java-based graphical user interface and a computer vision module with two phases: video processing (extracting hand silhouette features from the first frame and tracking centroid sequences across all frames) and recognition (analysing hand configuration, movement, and orientation as sign language primitives, then integrating these three channels to identify the letter).
Key findings
The computer vision recognition module achieved 84.3% accuracy in identifying Libras manual alphabet letters from webcam video input. The recognition approach is based on decomposing signs into three fundamental primitives of sign language: hand configuration (shape), movement, and orientation. The GUI allows users to record videos, submit them for processing, view the recognition result, and accept or reject it if the system misidentified the intended sign. For educational reinforcement, users can view the sign in Libras or see an illustrative image as a hint while guessing the word. After guessing correctly, the game associates the Portuguese word with its Libras sign to build bilingual vocabulary. The inclusive design means both deaf and hearing children can play together using the same interface, promoting social inclusion rather than segregating deaf users into separate tools.
Relevance
This work demonstrates how sign language recognition technology can be applied to create genuinely inclusive educational applications — tools designed for use by both deaf and hearing users rather than one group adapting to the other's interface. The approach of using game mechanics to motivate sign language learning is particularly relevant as interest in sign language education grows globally. For accessibility practitioners, the project illustrates important design principles: using sign language as the primary input modality rather than as an afterthought, building bilingual bridges between signed and written languages, and creating shared experiences that bring deaf and hearing users together. The computer vision approach using sign language primitives (configuration, movement, orientation) reflects established linguistic models of sign language structure and could potentially be extended to other sign languages beyond Libras.
Tags: sign language · sign language recognition · deaf accessibility · educational technology · game accessibility · gesture recognition · computer vision · inclusive design