Evaluation of Language Feedback Methods for Student Videos of American Sign Language
Matt Huenerfauth, Elaine Gale, Brian Penly, Sree Pillutla, Mackenzie Willard, Dhananjai Hariharan · 2017 · ACM Transactions on Accessible Computing (TACCESS) · doi:10.1145/3046788
Summary
This paper investigates how to best present video-based feedback to students learning American Sign Language (ASL), as part of a long-term project to build an automatic system that analyses student signing videos and provides immediate corrective feedback. The motivation is significant: deaf children benefit from early language exposure, and higher written literacy has been measured in deaf adults raised in ASL-using homes, yet new hearing parents of deaf children often struggle to find accessible, flexible ASL instruction. Two Wizard-of-Oz studies were conducted with ASL students at Rochester Institute of Technology (RIT). In Phase 1, 24 students in their second semester of ASL courses were assigned homework requiring them to record videos of themselves signing. The study compared three feedback conditions: VIDEO (students simply reviewed their own signing video), NOTES (students saw their video with a text panel listing errors and correct usages alongside), and POPUP (students saw their video with time-synchronized pop-up messages appearing at the exact moment of errors or correct performance). A human ASL expert served as the "wizard," identifying errors using a detailed codebook of 22 error and correct-usage categories covering facial expressions, hand positioning, fingerspelling, eye gaze, and body movement. An ASL instructor independently scored students' signing before and after viewing feedback to measure actual improvement. Phase 2 with 9 new students compared three variations of the POPUP approach: original technical terminology (POPUP), simplified language (SIMPLE), and simplified language with photographs illustrating correct ASL performance (PHOTO).
Key findings
In Phase 1, videos with feedback information (NOTES and POPUP) received significantly higher subjective scores than plain video review (VIDEO), confirming that annotated feedback is valued by students. Students significantly preferred the time-synchronized POPUP format over non-synchronized NOTES (median 9.5 vs 7). Crucially, an ASL instructor's blind evaluation showed that students who received any form of feedback (NOTES or POPUP) demonstrated significantly greater improvement in their signing compared to those who only watched their video (t-test p = 0.0008). However, there was no significant difference in actual signing improvement between NOTES and POPUP conditions, despite the subjective preference for POPUP. In Phase 2, students significantly preferred the PHOTO condition (median 9) over SIMPLE (median 8), indicating that visual illustrations of correct ASL facial expressions and hand positions enhanced the perceived quality of feedback. Surprisingly, simplified terminology (SIMPLE) was not significantly preferred over technical terminology (POPUP) — some students found the simplified wording less precise and informative. Qualitative feedback revealed students wanted even richer visual feedback, including short video clips of correct performance rather than static photographs. The detailed error codebook (22 categories) provides a comprehensive taxonomy of common ASL learner errors spanning facial expression timing, fingerspelling location and handshape, eye gaze, and body movement.
Relevance
This research has direct implications for the design of technology-mediated language education, particularly for visual-spatial languages like ASL where written feedback alone cannot convey the nuances of facial expression, hand positioning, and body movement. The finding that time-synchronized, visually illustrated feedback produces the highest student satisfaction offers a clear design template for ASL learning applications and for instructors providing electronic feedback on student video submissions. The work is timely given the growth of online ASL courses where instructor availability for real-time feedback is limited. For the broader accessibility community, the paper highlights the importance of ASL education as an accessibility enabler — improving ASL fluency among hearing people directly benefits deaf and hard-of-hearing individuals by expanding their communication opportunities. The Wizard-of-Oz methodology demonstrates a practical approach for evaluating feedback systems before the underlying computer vision technology is fully developed, and the error codebook provides a valuable resource for future automated ASL assessment tools.
Tags: sign language · deaf and hard of hearing · ASL education · video feedback · language learning · Wizard-of-Oz · facial expression