Speaking Through Pictures: Images vs. Icons
Xiaojuan Ma, Jordan Boyd-Graber, Sonya Nikolova, Perry R. Cook · 2009 · Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '09) · doi:10.1145/1639642.1639672
Summary
This paper investigates whether freely available web images can serve as effective substitutes for professionally designed icons in assistive communication systems for people with aphasia. Existing icon-based systems like Lingraphica and C-VIC (Computerized Visual Communication) help people with aphasia construct sentences by arranging pictorial symbols that are translated into spoken or written language. However, these systems suffer from severely limited vocabularies — typically only a few thousand icons — making it difficult for users to communicate about everyday topics like local news, new medications, or current events. Adding new icons requires specialist designers, creating a frustrating bottleneck. The authors propose using web images sourced from databases like the ESP Game and ImageNet as a scalable alternative. They conducted two studies: Experiment 1 recruited 24 elderly participants from a senior center to compare how well web images versus Lingraphica icons conveyed word meaning when embedded in sentences. Experiment 2 tested 50 individuals with aphasia at a rehabilitation center, presenting both icons and images alongside audio pronunciations and asking participants to select the correct word from five options. The between-subjects and within-subjects designs respectively controlled for modality bias and individual variability in the aphasia population.
Key findings
Experiment 1 found no statistically significant difference between images and icons at the sentence level, with both modalities scoring similarly on human-coded accuracy (icons: 0.76, images: 0.77). Both performed well for nouns but struggled with adjectives and verbs. Images were slightly better for understanding nouns, while icons performed marginally better for other word classes. Experiment 2, conducted with people with aphasia, confirmed these results: error rates for icons were higher than images in 13 out of 25 nouns tested. Accuracy and entropy measures showed images slightly outperformed icons, though not significantly. People with high and medium cognitive levels were faster with images, while those with low cognitive levels were faster with icons. Response times were generally comparable, with no significant speed difference between modalities. Abstract nouns like "appointment," "government," and "trouble" were difficult for both images and icons, while concrete nouns were easier to interpret in both formats.
Relevance
This research has significant practical implications for expanding communication options for people with aphasia. By demonstrating that web images can match professionally designed icons in conveying meaning, the study opens the door to vastly larger visual vocabularies sourced from the internet — potentially millions of images versus the few thousand available in commercial icon sets. For accessibility practitioners, this suggests that assistive communication tools need not be constrained by expensive, slow icon development processes. The findings also highlight important design considerations: concrete concepts are easier to represent visually than abstract ones regardless of format, and cognitive level influences which modality works best. The study's limitations include relatively small sample sizes and the inherent variability in the aphasia population, but the consistent finding across both elderly and aphasic participants strengthens confidence in the results.
Tags: aphasia · augmentative and alternative communication · visual communication · icon-based communication · assistive technology · web images · elderly users · cognitive accessibility