Comparing Methods of Displaying Language Feedback for Student Videos of American Sign Language
Matt Huenerfauth, Elaine Gale, Brian Penly, Mackenzie Willard, Dhananjai Hariharan · 2015 · ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility · doi:10.1145/2700648.2809859
Summary
This research investigates how to effectively provide feedback to students learning American Sign Language (ASL), specifically comparing different methods of presenting information about errors and correct usage in their signing videos. The work is motivated by an urgent accessibility need: over 80% of deaf children are born to hearing parents, and early exposure to ASL significantly benefits deaf children's language development and English literacy. ASL has also become the third most studied language at U.S. universities, with enrollments rising 19% between 2008-2013. The researchers are developing an automated system that would analyze videos of ASL students and provide immediate feedback about movement fluency—not semantic content, but observable aspects like facial expressions for grammatical markers, eye gaze during pointing, smoothness of movement, and fingerspelling hand position. This Wizard-of-Oz study simulates such a system to evaluate interface design before implementing full automation. Eight ASL students (1.5 semesters experience) completed a translation task producing 12 short ASL videos. A human expert analyzed each video using predefined error codes, and researchers created feedback stimuli in three conditions: VIDEO (simply replaying the student's recording), NOTES (video followed by text messages about errors), and POPUP (video with time-synchronized pop-up messages pointing to specific body parts during errors, plus summary notes). Students viewed feedback in each condition before re-recording, and an experienced ASL instructor evaluated all before/after videos in a blind assessment.
Key findings
Students strongly preferred receiving feedback over merely rewatching their videos. Subjective quality ratings showed a median score of 8.5 for feedback conditions versus 3 for video-only replay (statistically significant). When comparing feedback presentation methods, students rated time-synchronized POPUP feedback significantly higher than NOTES (median 9.5 vs 7), indicating that seeing error messages appear at specific moments in their video was more satisfying. Critically, feedback actually improved student performance. ASL instructor evaluations showed significantly greater improvement in students' Round 2 recordings when they had received feedback (NOTES or POPUP) compared to simply rewatching their video. However, there was no significant difference in actual performance improvement between POPUP and NOTES conditions—suggesting that while students preferred time-synchronized feedback, both methods were equally effective at helping them correct errors. The error codes used were deliberately limited to aspects an automated system could realistically detect: facial expression timing for grammatical markers (topic, negation, questions), eye gaze during pointing, hand position during fingerspelling, and movement smoothness. The finding that these limited codes produced measurable improvement suggests automated systems need not achieve full semantic understanding to be useful.
Relevance
This study provides a foundation for developing automated ASL tutoring systems that could benefit both the growing population of ASL students and, critically, hearing parents of deaf children who need to learn ASL quickly but may lack time for extensive classroom instruction. The research demonstrates that even limited, non-semantic feedback (focusing on movement properties rather than meaning) can measurably improve student signing. For practitioners developing language learning technology, the key insight is that time-synchronized feedback significantly improves user satisfaction, even when it does not improve actual learning outcomes compared to summary feedback. This suggests that systems should invest in temporal alignment of feedback with video when possible, as it affects user engagement and willingness to use the tool. The Wizard-of-Oz methodology is well-suited for testing interface designs before implementing computationally expensive automatic video analysis. Limitations include small sample size (8 participants), single evaluator for scoring, and the possibility that automated systems will make more errors than human "wizards." Future work will explore how students respond to less accurate automated feedback.
Tags: American Sign Language · ASL · deaf education · language learning · video feedback · Wizard-of-Oz · computer-assisted instruction · sign language · hearing parents