Effect of Sign-recognition Performance on the Usability of Sign-language Dictionary Search
Saad Hassan, Oliver Alonzo, Abraham Glasser, Matt Huenerfauth · 2021 · ACM Transactions on Accessible Computing · doi:10.1145/3470650
Summary
This paper investigates how the performance of sign-language recognition technology affects user satisfaction with ASL dictionary search systems. Unlike written languages where users can type unfamiliar words to look them up, ASL learners who encounter an unfamiliar sign cannot easily search for it—they must either manually select linguistic features (handshape, location, movement) or submit a video of themselves performing the sign for automatic recognition. The researchers conducted three Wizard-of-Oz studies with hearing ASL students at Rochester Institute of Technology to understand which properties of search results affect user perception. Using a prototype that displayed predetermined results (controlling for recognition accuracy), participants watched a video of an ASL sign, recorded themselves performing it via webcam, and then evaluated the search results. The studies examined three factors: the position of the desired word in results (rank 1, 5, 10, or 20), whether the result appeared above or below the fold (requiring scrolling), and the precision/similarity of surrounding results.
Key findings
User satisfaction decreased significantly as the position of the desired word moved lower in the results list (Friedman test χ² = 182.682, p < 0.01), with satisfaction dropping below the Likert scale midpoint between positions 10 and 20. The "above-the-fold" effect was significant: when the desired result required scrolling, user satisfaction dropped sharply at the fold boundary—position 6 for interfaces showing 6 results per page, position 8 for interfaces showing 8 results. The precision of surrounding results also significantly affected satisfaction (χ² = 16.526, p < 0.01), with users rating high-precision lists (containing visually similar signs) more favorably than low-precision lists, even when the target word appeared in the same position. Critically, the normalized Discounted Cumulative Gain (nDCG) metric—which incorporates both position and relevance of surrounding items—correlated significantly with user satisfaction across all studies, while the simpler binary DCG metric (commonly reported in ASL dictionary research) did not correlate with user judgments when precision varied.
Relevance
This research provides essential guidance for two audiences: designers of ASL dictionary interfaces and researchers developing sign-recognition algorithms. For interface designers, the findings suggest that the number of results displayed per page should be informed by the accuracy of the underlying recognition technology—if the system reliably places desired words in the top-k results, displaying k items above the fold maximizes satisfaction. For recognition researchers, the study establishes that reporting only top-k accuracy is insufficient; users also care about the quality of surrounding results, so researchers should adopt metrics like nDCG that capture overall list quality. The work addresses a critical gap in ASL accessibility technology, as approximately 500,000 people in the U.S. use ASL as their primary language, and growing interest in ASL learning creates demand for effective dictionary tools. Limitations include the focus on hearing ASL learners rather than Deaf users, and the use of isolated signs rather than signs encountered in sentence context.
Tags: American Sign Language · ASL · sign language recognition · dictionary search · information retrieval · deaf and hard of hearing · usability · search interfaces · computer vision