Enhancing Scanning Input with Non-Speech Sounds
Stephen A. Brewster, Veli-Pekka Raty, Atte Kortekangas · 1996 · Proceedings of the Second Annual ACM Conference on Assistive Technologies (Assets '96) · doi:10.1145/228347.228350
Summary
This paper proposes adding structured non-speech sounds (earcons) to scanning input systems used by people with severe motor disabilities who can only operate a single switch. Scanning input works by sequentially highlighting items in a grid; the user presses their switch when the highlight reaches the desired target. The authors identify a fundamental mismatch in conventional scanning systems: scanning is inherently a temporal task (timing a switch press to coincide with the highlight reaching the target), yet it is presented purely as a spatial/visual task (watching a highlight move across a grid). Research in psychology has shown that the auditory modality is often superior to the visual for temporal tasks, and humans have a basic ability to perceive and predict rhythms from a very young age. The paper leverages these natural rhythmic abilities by adding earcons — abstract, structured audio messages built from musical motives — to encode the scanning position. The system was built on an Apple Macintosh HyperCard stack with sounds played through a Yamaha TG100 MIDI sound module. This work was part of the TIDE ACCESS Project 1001, aimed at creating a mobile communication device for people with speech-motor and language-cognitive impairments.
Key findings
The earcon design used pitch to encode row position (high pitch for top rows, descending to low pitch for bottom rows) and arpeggios of notes within a row's octave to encode column position, with the first note accented and the last note lengthened to create rhythmic structure. For row scanning on a 4x4 grid, each row was assigned a base octave (C1 at 1056Hz for top row down to C4 at 130Hz for bottom). For item scanning within a row, four notes (C, E, G, B) were played as an arpeggio. The system addressed the problem of long scanning delays (up to 5 seconds per row for users with limited motor control) breaking rhythm perception — humans cannot perceive rhythm when gaps exceed 1800ms. The solution was to repeat the earcon pattern with diminishing volume to fill long gaps, maintaining rhythmic continuity. A preliminary evaluation with one participant (a child at a Finnish rehabilitation hospital) showed favourable results: the participant was able to use the sonically-enhanced system, appeared more interested and engaged, and seemed to perform better with sound, though no formal analysis was conducted due to the limited evaluation scope. The authors noted the child treated the sound-enhanced system more like a game.
Relevance
This paper addresses a problem that remains highly relevant for switch access users: the cognitive and perceptual demands of scanning input, which is still one of the primary access methods for people with severe motor impairments. The insight that scanning is a temporal task being poorly served by purely visual feedback applies to modern AAC devices, on-screen keyboards, and environmental control systems that use scanning. For practitioners designing scanning interfaces, the paper provides concrete guidance: leverage the auditory modality's natural superiority for temporal tasks, design sound patterns that support rhythm perception to help users predict when to press their switch, and use pitch mapping to encode position information. The earcon-based approach also reduces the visual attention demands of scanning, which is particularly beneficial for users who have difficulty maintaining visual focus on a moving highlight. While modern scanning systems have incorporated various audio feedback options, the principled approach of mapping scanning's temporal structure to musical rhythm patterns remains underexplored in commercial products.
Tags: scanning input · earcons · sonification · auditory display · switch access · augmentative and alternative communication · multimodal interaction · motor disability