ArtfulSign: A Closed-Loop, Semantics-Grounded Mobile System for Learning Chinese Sign Language
Yuan Zhao, Yueran Wang, Chenglong Tan, Siyang Tong, Dengfeng Yao, Wei Zhen · 2026 · Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA '26) · doi:10.1145/3772363.3799131
Summary
ArtfulSign is a mobile iOS application that reframes Chinese Sign Language (CSL) learning as an embodied skill practiced through short closed-loop interaction cycles rather than passive video consumption. The authors argue that most existing sign language learning tools - video dictionaries, illustrated phrase collections, browsing-based tutorials - operate open-loop: they present demonstrations but never observe the learner's actual production, leaving error detection to the learner or an external instructor. ArtfulSign closes that loop by pairing on-device, real-time camera-based sign recognition with meaning-oriented explanations and lightweight situated micro-contexts illustrating deaf cultural practice. The interaction is structured as a four-stage cycle (prepare, perform, feedback, retry-or-pass) repeated per vocabulary item in roughly one to two minute sessions. A central design move is semantic grounding: rather than treat each sign as an arbitrary gesture to memorise, the system pairs every vocabulary item with a short explanation and illustration showing the iconic or metaphorical motivation linking the sign's form to its meaning - for example, the sign for 'baby' is presented alongside an image of cradling a baby. Four design principles are articulated: closed-loop practice structure, immediate and interpretable feedback, meaning-oriented scaffolding, and lightweight situated context. The system runs fully on-device, requires no account and no network, and is presented as an interactive CHI demo rather than as a study with measured learning outcomes.
Key findings
This is an interaction-design and demo paper rather than an empirical evaluation, so the contributions are design-level rather than statistical. The authors articulate and instantiate a closed-loop, practice-centred interaction pattern for sign language learning, and show that on-device, camera-based recognition is now feasible on consumer mobile hardware in a way that supports real-time correctness feedback without server round-trips or accounts. They demonstrate that semantic grounding - explicitly linking sign form to meaning through iconic illustration - can be integrated directly into the practice loop rather than treated as separate reference material, and that brief situated micro-contexts can convey deaf pragmatic and cultural knowledge (such as how to politely get a deaf person's attention in public) without overwhelming the core word-level loop. The authors openly note three limitations: vocabulary coverage is intentionally narrow to support focused demo interactions; recognition reliability degrades under poor lighting, occlusion, or unstable framing; and the system is currently visual-only, with no non-visual access path for blind learners. Privacy is addressed by keeping camera input on-device with no transmission.
Relevance
For accessibility practitioners working on deaf education, sign language tooling, or embodied-skill training, this paper is useful less for measured outcomes and more as a clear articulation of why open-loop video tutorials are inadequate for motor learning and what a closed-loop alternative looks like in practice. The framing of sign language acquisition as embodied skill learning rather than symbolic vocabulary acquisition has direct implications for product design in this space. The semantic grounding move is particularly transferable: pairing recognition feedback with iconic motivation explanations is a pattern applicable to ASL, BSL, and other sign languages, and to other embodied learning contexts. The paper also serves as a useful reminder that on-device recognition addresses real privacy concerns for camera-based learning. Notable gaps for future work include validation of learning outcomes against video-only baselines, accessibility of the tool itself for deafblind learners, and how the closed-loop pattern generalises beyond isolated vocabulary to phrases, syntax, and conversational signing.
Tags: Chinese Sign Language · sign language learning · embodied learning · closed-loop interaction · semantic grounding · mobile accessibility · on-device recognition · deaf accessibility · iconicity · practice-based learning