Measuring the Performance of a Location-Aware Text Prediction System
Luís Filipe Garcia, Luís Caldas De Oliveira, David Martins De Matos · 2015 · ACM Transactions on Accessible Computing (TACCESS) · doi:10.1145/2739998
Summary
This study rigorously evaluates whether adding location awareness to word and sentence prediction improves communication efficiency for AAC users. The researchers developed Eugénio, a Portuguese AAC system that combines traditional word prediction with sentence prediction and location-specific language models. Using GPS for outdoor locations and Bluetooth beacons for indoor spaces, the system automatically switches between language models trained on location-specific vocabulary—for example, offering different predictions in a classroom versus a cafeteria versus home. The evaluation used a Location-Vocabulary Corpus (LocVocCorpus) of approximately 2,500 sentences collected from undergraduate students for three contexts: classroom, school cafeteria, and home. Twenty-four participants without disabilities completed user tests copying sentences using an on-screen QWERTY keyboard with eight word prediction slots and four sentence prediction slots. The researchers compared location-specific language models against an all-purpose model trained on all locations combined, measuring Words Per Minute (WPM) and Keystroke Saving Rate (KSR). To complement the user tests, they also ran software simulations modeling optimal user behavior under varying levels of sentence reuse.
Key findings
User tests showed modest improvements with location-specific models: 2.4% increase in WPM (23.5 vs 22.9) and 1.3% increase in KSR (72.3% vs 71.3%), but these differences were not statistically significant. However, software simulations revealed a 12-percentage-point gap between actual and optimal performance, suggesting untapped potential being lost to interface design, training, or cognitive load issues rather than the location-aware approach itself. The study introduced a "mixed" language model strategy that combines location-specific and all-purpose approaches. Mixed models performed better under low sentence-reuse conditions (0-50% reuse), while pure location-specific models excelled under high sentence-reuse scenarios (75-100%). Participants strongly favored sentence prediction, using it for 89.8% of sentences while word prediction was used for only 20.2% of words. Sentence prediction correlated positively with WPM (0.409, p<0.001), while word prediction correlated negatively (-0.464, p<0.001), suggesting that sentence-level completion is more efficient for communication rate.
Relevance
This research provides important quantitative evidence for AAC developers considering context-aware features. While the location-aware approach shows theoretical promise, the modest real-world improvements suggest that interface optimization and user training may yield greater gains than sophisticated prediction algorithms alone. The finding that users intuitively preferred sentence prediction over word prediction has direct implications for AAC interface design—systems should prioritize whole-utterance completion. The study's key limitation—testing with non-disabled users—is significant since AAC users exhibit different sentence-reuse patterns and may benefit more from location-aware predictions. The gap between simulated optimal performance and actual user performance highlights that even well-designed prediction systems fail if users cannot efficiently access predictions. For practitioners, this underscores the importance of interface usability testing alongside algorithm development.
Tags: augmentative and alternative communication · word prediction · sentence prediction · location-awareness · context-aware computing · language models · communication rate