CoSignPlay: A Collaborative Approach to Learning Non-Manual Signs in ASL for Hearing Families with Deaf Children
Xuanyu Liu, Hsin-Le Cheng, Guillaume Chastel, Margaret Chastel, Zhen Bai · 2025 · ASSETS 2025: 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746382
Summary
This paper introduces CoSignPlay, a novel collaborative learning approach for teaching non-manual signs (NMS) in American Sign Language to hearing families with deaf and hard-of-hearing (DHH) children. Over 90% of DHH children are born to hearing families, and without accessible sign language environments, these children face risks of language deprivation, cognitive delays, and socio-emotional isolation — a phenomenon known as "Dinner Table Syndrome." NMS — facial expressions, head movements, and body posture that serve lexical, morphological, and syntactic functions in ASL — are particularly challenging for hearing learners due to both cognitive demands (processing visual language simultaneously across multiple channels) and cultural barriers (discomfort with exaggerated facial expressions). Inspired by "group narrative," a collaborative storytelling activity in Deaf communities where one person handles NMS from the front while another performs manual signs from behind, CoSignPlay allows two players to jointly control a 3D avatar in a game context. The researchers built a working prototype using MediaPipe for face and hand tracking mapped to a Unity avatar, and conducted a design and technology probe study with six hearing parents of young DHH children, six ASL instructors, and two speech-language pathologists. The game is set in a restaurant scenario where players role-play as an ASL interpreter helping a Deaf chef, incorporating four basic NMS types: Yes/No questions, Negation, Degree, and Lexical.
Key findings
Participants identified five main channels for learning NMS: classroom-based learning, home visits and early education programs, technology-based tools, cultural immersion with Deaf communities, and collaborative learning experiences. Key challenges included the cognitive difficulty of shifting from sequential spoken language to simultaneous visual language, the need to process MS and NMS simultaneously, lack of grammar knowledge about NMS functions, failure to recognize NMS importance, and cultural embarrassment around exaggerated facial expressions. CoSignPlay received positive feedback across three benefit categories: socio-cultural benefits (family bonding, reduced embarrassment through playful avatar interaction, alignment with Deaf culture activities), cognitive benefits (reduced cognitive load by offloading MS and NMS to separate players, peer feedback opportunities), and practical benefits (self-learning promotion, emphasizing NMS importance). Key limitations included concerns about compromising the holistic nature of ASL by separating MS and NMS, synchronization difficulties between players, and questions about avatar movement naturalness. Participants suggested role-switching between MS and NMS, progressive difficulty levels, both collaborative and competitive modes, avatar customization, and minimizing text-based instructions for young children.
Relevance
This research addresses a critical gap in sign language education technology by focusing specifically on NMS — an essential but under-supported component of ASL that is often neglected in existing learning tools. The collaborative approach draws directly from Deaf cultural practices (group narrative), modeling how accessibility research can ground technology design in community traditions rather than imposing external frameworks. For practitioners, the findings highlight that NMS learning requires attention to both cognitive scaffolding and cultural sensitivity, particularly around facial expression use across different cultural backgrounds and gender norms. The family-centered design implications are significant for early intervention programs serving DHH children, suggesting that playful collaborative technology can bridge the communication gap in hearing families while building connections to Deaf culture. The design guidelines for avatar-based learning — including synchronization support, customization options, and age-appropriate interfaces — provide a practical foundation for developing inclusive educational technologies.
Tags: American Sign Language · non-manual signs · deaf and hard of hearing · collaborative learning · avatar · sign language education · family-centered learning · gamification · Deaf culture