Understanding Emotion through Multimedia: Comparison between Hearing-Impaired People and People with Hearing Abilities
Rumi Hiraga, Nobuko Kato · 2006 · Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '06) · doi:10.1145/1168987.1169012
Summary
This paper investigates how hearing-impaired and normal-hearing people recognise emotions conveyed through drum performances and accompanying visual stimuli. The research is part of a larger project to build a "performance enhancement machine" (PEM) — a system that would enable hearing-impaired and normal-hearing people to play music together by generating visual cues representing the dominant emotion in a performance. The authors's six years of teaching hearing-impaired students to use computers for music revealed that many hearing-impaired people are interested in and enjoy playing music. The experiment tested emotion recognition (joy, fear, anger, sadness) across four stimulus types: drum performance alone (sound), drum performance paired with a drawing expressing the same emotion, and drum performance paired with one of two types of motion pictures (amoeba and fountain effects from Windows Media Player). Twenty-four drum performances were recorded by three groups: hearing-impaired students, normal-hearing amateurs, and professional drummers, played on MIDI drum sets and acoustic tams. Eleven hearing-impaired college students (hearing loss over 100 decibels) and 15 normal-hearing students participated in the evaluation.
Key findings
The difference in ability to recognise intended emotions between hearing-impaired and normal-hearing participants was statistically insignificant across all stimulus categories — a striking finding suggesting that both groups encode and decode emotions through similar processes. Drawing stimuli produced the highest recognition rates for both groups, even though participants reported difficulty when they felt the drawing and performance conveyed conflicting emotions. Fear was the most poorly recognised emotion across both groups and all stimulus types. Visual stimuli were especially effective for performances whose intended emotions were not clear from sound alone — recognition rates for the worst-recognised performances increased when accompanied by visual information. Hearing-impaired subjects showed a strong preference for motion picture stimuli, while normal-hearing subjects preferred drawings and motion pictures roughly equally. A supplemental drawing-only experiment one month later confirmed no significant difference in emotion recognition from drawings between the two groups. Sadness was better recognised by hearing-impaired subjects, while the other three emotions were better recognised by normal-hearing subjects. Interestingly, 7 of 11 hearing-impaired and 8 of 15 normal-hearing subjects reported perceiving different emotions from paired performance-drawing stimuli, yet the drawing category still achieved the highest recognition rates overall.
Relevance
This research challenges assumptions about what hearing-impaired people can and cannot perceive in musical contexts. The finding that emotion recognition ability is comparable between hearing-impaired and normal-hearing people supports the development of inclusive music systems where both groups can participate together. For accessibility practitioners, the key insight is that visual representations of musical emotion — particularly drawings over abstract motion effects — can effectively bridge the gap for performances whose emotional content is unclear from sound alone. The PEM system concept, where visual cues enable collaborative music-making across hearing abilities, represents an approach to inclusive design that goes beyond accommodation toward genuine shared experience. The research also highlights an important methodological point: hearing-impaired participants sat on a wood floor to receive vibrotactile information from the drums, suggesting that whole-body vibration is part of how they engage with music.
Tags: hearing impairment · music accessibility · emotion recognition · multimodal · music visualisation · drum performance · inclusive design