Towards Large Scale Evaluation of Novel Sonification Techniques for Non Visual Shape Exploration
Andrea Gerino, Lorenzo Picinali, Cristian Bernareggi, Nicolò Alabastro, Sergio Mascetti · 2015 · ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility · doi:10.1145/2700648.2809848
Summary
This paper presents six sonification techniques for exploring shapes on touchscreen devices and evaluates them through Invisible Puzzle, an iPhone application enabling large-scale unsupervised testing. The research addresses a critical need: people who are blind need access to images for everyday activities (reading maps) and education (developing visuospatial skills for STEM), but tactile solutions are bulky, expensive, and limited in detail. The sonification modes are built on two exploration paradigms. The 2-D paradigm allows free exploration—touching any pixel plays a tone (100-1000 Hz) based on luminance, mimicking how blind people explore tactile drawings on swell paper. The 1-D paradigm restricts exploration to vertical movement, simultaneously sonifying all pixels on the horizontal "flush line" being touched. Five 1-D modes vary in audio rendering: Variable Frequency Pure (VF Pure) and VF Noise use continuous frequency mapping; Fixed Frequency Pure (FF Pure), FF Noise, and FF Music divide the line into 24 sectors with discrete frequencies. Sound spatialization conveys horizontal position using Interaural Level Differences (up to 20 dB) and Interaural Time Differences (up to 1 ms), creating a piano-keyboard-like mapping where low frequencies appear left and high frequencies right. The system was developed through user-centered design with feedback from multiple users including four people who are blind.
Key findings
Evaluation with 131 sighted and 18 blind subjects showed that all sonification techniques enabled shape recognition after brief training. The 1-D FF Pure mode achieved the best trade-off between accuracy (71.3% correct for sighted, 82.2% for blind), speed (mean 10.6s exploration for sighted, 15.2s for blind), and user satisfaction. Key findings from statistical analysis: - No significant accuracy differences between sonification modes for sighted users, but significant differences in exploration time (2-D and 1-D FF Music were slowest) - 2-D mode required significantly more concentration and was least enjoyable - 1-D FF Pure was rated most enjoyable and generated highest interest in replaying - Music sound generator was rated most pleasant, but 1-D FF Music unexpectedly rated least enjoyable overall—likely because the music demanded higher attention - Subjects who play musical instruments achieved significantly higher accuracy (75% vs 67%) - Blind subjects achieved similar or better results than sighted subjects, with 1-D FF Pure yielding 82.2% accuracy Remarkably, 90% of subjects who gave wrong answers on the first task gave correct answers after repeating it, demonstrating rapid learning. The gamification approach (progressive difficulty following Csikszentmihalyi's Flow theory) kept users engaged through 16 tasks without supervisor intervention.
Relevance
This research makes two significant contributions: novel sonification techniques and a scalable evaluation methodology. The finding that 1-D exploration modes are faster than 2-D while maintaining accuracy has practical implications—users can perceive shapes in seconds rather than the extended exploration required for free-form tactile exploration. The remote, unsupervised evaluation approach via a gamified mobile app represents a methodological innovation for accessibility research. Traditional studies struggle to recruit sufficient participants with visual impairments; this approach collected 149 tests in under two weeks with minimal management effort. The gamification elements (progressive difficulty, immediate feedback, engagement metrics) maintained participation without supervisor presence. For practitioners, the specific technical parameters are valuable: frequency range 100-1440 Hz, 24 concurrent sound generators for FF modes, ILD up to 20 dB, ITD up to 1 ms. The finding that musical instrument experience correlates with performance suggests that sonification literacy could be developed through training. The planned release as a public game could eventually enable evaluation with thousands of blind users, establishing robust evidence for which sonification approaches work best across diverse users.
Tags: sonification · blindness · visual impairment · touchscreen · image exploration · audio feedback · spatial audio · gamification · remote evaluation · visuospatial skills