MobileASL: Intelligibility of Sign Language Video as Constrained by Mobile Phone Technology
Anna Cavender, Richard E. Ladner, Eve A. Riskin · 2006 · Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '06) · doi:10.1145/1168987.1169001
Summary
This paper presents MobileASL, a video compression research project aimed at enabling Deaf people to communicate in American Sign Language (ASL) over mobile phone networks. In 2006, Deaf people in the US were limited to text messaging on mobile phones, forcing communication in English rather than their preferred language, ASL. While mobile video phones existed, the bandwidth constraints of cell phone networks (15-30 kbps upload on GPRS) meant that standard video compression could not produce intelligible sign language video at available bit rates. The research exploits the visual nature of sign language, drawing on eye-tracking studies showing that sign language receivers maintain high-resolution foveal vision around the signer's face. Two sign-language-specific encoding techniques were studied: region-of-interest (ROI) encoding, where the face area receives higher quality encoding at the expense of surrounding regions, and reduced frame rates (10 fps vs 15 fps), which yield better quality individual frames at a given bit rate. The study comprised a focus group with four Deaf Community members to understand needs and desires for mobile video communication, followed by a user study with eighteen Deaf adults (10 Deaf, 5 hearing, 3 CODAs) who evaluated 18 different encoding configurations across three bit rates (15, 20, 25 kbps) on Sprint PPC 6700 PDA-style phones.
Key findings
Focus group participants expressed strong desire for mobile video calling, identifying use cases ranging from casual conversation to emergencies, and noted that text is cumbersome because English is not the native language of most Deaf people in the US. They preferred that text messaging remain available as a fallback for poor connections. Privacy was not a concern — participants noted that sign language conversations can be observed in person just as easily. In the video compression study, participants showed statistically significant preferences for higher bit rates (25 kbps preferred over 20 kbps, which was preferred over 15 kbps; F(2,34)=51.12, p<.01). For frame rate, 10 fps was preferred over 15 fps (F(1,17)=4.59, p<.05), likely because fewer frames per second meant more bits per frame and thus better image quality — participants did not notice the frame rate difference. For ROI encoding, participants significantly preferred a moderate quality increase in the face region (ROI value of -6, doubling face quality) over no ROI or extreme ROI (-12, quadrupling face quality). The -12 ROI caused too much distortion in the hand/body regions. The best-rated encoding combination was 25 kbps, 10 fps, -6 ROI. The "understand" and "use" ratings were highly correlated (r(16)=.85, p<.01), indicating that perceived intelligibility directly predicted willingness to use the technology.
Relevance
This research addresses a fundamental telecommunications accessibility issue: the exclusion of Deaf people from real-time mobile communication in their preferred language. While hearing people could make phone calls anywhere, Deaf people were constrained to text — a medium in a language that is often their second language. The insight that sign language video can be made intelligible at low bit rates by exploiting how signers perceive visual information (focusing on the face) is a creative application of psycholinguistic research to engineering. The region-of-interest approach demonstrates that accessible video compression is not just about general quality but about allocating quality where it matters most for the specific communication modality. Although mobile network bandwidth has increased dramatically since 2006, the principles identified here — that face quality matters more than background quality for sign language, and that frame quality matters more than frame rate — remain relevant for sign language video in bandwidth-constrained situations such as developing countries, rural areas, or congested networks. The participatory approach of involving the Deaf Community in defining requirements and evaluating results exemplifies best practice in accessible technology development.
Tags: deaf · American Sign Language · mobile phone · video compression · video relay · telecommunication · sign language · accessibility
Standards referenced: H.264