Design and Psychometric Evaluation of an American Sign Language Translation of the System Usability Scale
Matt Huenerfauth, Kasmira Patel, Larwan Berke · 2017 · Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17) · doi:10.1145/3132525.3132540
Summary
This paper addresses a significant gap in accessibility research methodology: the lack of standardized usability questionnaires available in American Sign Language (ASL) for Deaf and Hard-of-Hearing (DHH) participants. The System Usability Scale (SUS) is one of the most widely used instruments in HCI for measuring perceived usability, but its English text format creates a barrier for many deaf adults in the U.S. who have lower English reading literacy and prefer to communicate in ASL. The researchers created ASL-SUS, a formal ASL video translation of the ten-item SUS questionnaire, following established survey translation methodologies. The translation was produced by a bilingual expert team including two native ASL signers from the Deaf community and a faculty member with ASL computational linguistics expertise. The team prioritized meaning equivalence over word-for-word transliteration, using ASL linguistic structures such as rhetorical questions, facial expressions, and spatial reference points to convey the original intent. High-definition videos of a native ASL signer performing each item were professionally recorded. The translation quality was evaluated through a back-translation study with nine advanced ASL interpreting students, whose feedback led to revisions of six items. The instrument was then validated through two user studies with DHH participants evaluating a university website.
Key findings
The ASL-SUS demonstrated strong criterion validity: scores from 30 DHH participants correlated significantly with an independently administered adjective rating scale (Pearson's r=0.684, p<0.001). When compared to published correlations between the original English SUS and the same adjective scale (r=0.822), no statistically significant difference was found (p=0.09), confirming that ASL-SUS measures the same construct as the English version. Internal reliability, measured by Cronbach's alpha, was 0.69 for ASL-SUS — borderline but comparable to the alpha of 0.79 obtained from 10 additional DHH participants who took the original English SUS. The mean ASL-SUS score (52.25) was close to the mean English SUS score from DHH participants (50.5), suggesting consistent measurement. None of the 30 DHH participants in the main study reported difficulty understanding or responding to the ASL-SUS videos. The back-translation evaluation identified that 2 of the 10 items needed major revision and 4 needed minor changes, demonstrating the importance of this validation step.
Relevance
This work has direct practical value for any researcher or practitioner conducting usability studies who wants to include DHH participants. The ASL-SUS instrument is freely available online, removing a barrier to inclusive user research. The paper also serves as a methodological template for translating other standardized instruments into sign languages — a process that differs significantly from spoken-language translation due to the visual-spatial nature of sign languages and the need for video delivery. For accessibility practitioners, the study underscores that inclusive research requires more than just recruiting diverse participants; the research instruments themselves must be accessible. The relatively small validation sample (30 participants) and single-website evaluation context are limitations the authors acknowledge, and future work to evaluate additional psychometric properties like construct validity and test-retest reliability would strengthen the instrument further.
Tags: sign language · deaf accessibility · usability testing · survey translation · psychometrics · American Sign Language · research methods · inclusive research