A Systematic Review of User Studies as a Basis for the Design of Systems for Automatic Sign Language Processing
Soraia Prietch, J. Alfredo Sánchez, Josefina Guerrero · 2022 · ACM Transactions on Accessible Computing · doi:10.1145/3563395
Summary
This systematic literature review examines how user studies have been conducted in the design of Automatic Sign Language Processing (ASLP) systems. The authors searched seven repositories plus Google Scholar, identifying 2,486 papers and ultimately selecting 37 primary studies published between 2015 and 2020 that reported user study findings. The review covers three categories of ASLP: automatic sign language generation (ASLG, 13 papers), automatic sign language recognition (ASLR, 13 papers), and automatic sign language translation (ASLT, 11 papers). The analysis was guided by four research questions: What are the goals and research methods used? In which phases of the interaction design lifecycle is user involvement reported? How are cultural and collaborative aspects addressed? What lessons can be learned? The authors employed qualitative metasummary techniques to synthesize findings across diverse research contexts spanning 11 countries and 21 different sign languages. The review is framed by the principle that Deaf persons—whether or not they use sign language—have historically been underrepresented in technology design decisions. The authors emphasize that conducting user studies with marginalized populations is essential for guaranteeing their right to participate in choices made "for, with, and by them." This framing positions the systematic review not merely as a methodological survey but as an examination of whether ASLP research practices align with inclusive design principles.
Key findings
Across the 37 studies, 747 participants were involved, with an average of 21 per study. Deaf or hard of hearing persons were the largest participant group (358 participants in 16 papers), though 10 papers did not specify participant type. Numerical scale questionnaires were the most common research instrument (25 papers), followed by prototyping (15 papers), experimental design (12 papers), and usability testing (12 papers). A striking finding concerns user involvement timing: potential users are predominantly included in the evaluation phase (Phase 4 of the interaction design lifecycle), with informative or consultative involvement. Only two papers reported participatory design approaches where users actually co-designed solutions. The pattern is "design for" users rather than "design with" or "design by" them. This means Deaf community members rarely shape ASLP system requirements or conceptual designs—they are brought in primarily to validate what researchers have already built. Cultural aspects received minimal attention: only one study explicitly incorporated cultural theory (Co-Cultural Theory) into system design and research methodology. Regarding collaboration, twice as many hearing professionals participated as Deaf ones. While 13 papers reported D/HH collaborators as sign language experts, consultants, or recruiters, none reported having sign language users as collaborators throughout the entire study process. Additionally, 20 of 37 studies created their own sign language datasets due to the lack of standardized, publicly available resources—a major barrier to progress in the field.
Relevance
This review provides a critical assessment of inclusion practices in ASLP research that has implications well beyond sign language technology. The finding that users are consulted primarily for evaluation—not requirements gathering or design—reflects a broader pattern in assistive technology development where disabled people are positioned as passive evaluators rather than active co-creators. For practitioners developing sign language applications, the review offers concrete recommendations: explore diverse usage scenarios (educational tools, emergency alerts, medical information); ensure good processing quality to prevent communication breaks; address inter-sign transitions in continuous signing; provide multimodal design options; and involve D/HH researchers throughout the design lifecycle, not just during testing. The lesson on "role inversion"—placing hearing participants in minority group situations—offers a creative approach to building empathy and perspective-taking among hearing researchers. The scarcity of participatory design and the predominance of quantitative questionnaires suggest that ASLP research may be optimizing for measurable outcomes while missing deeper insights about user needs, cultural contexts, and communication quality that qualitative methods could reveal. For organizations building sign language technology, this review is a call to examine whether their own user research practices genuinely center Deaf community perspectives or merely check the box of "user testing."
Tags: sign language · automatic sign language processing · user studies · participatory design · Deaf community · systematic literature review · sign language recognition · sign language generation · sign language translation