The Feasibility of Eyes-Free Touchscreen Keyboard Typing
Keith Vertanen, Haythem Memmi, Per Ola Kristensson · 2013 · Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2513383.2513399
Summary
This poster paper proposes a novel approach to eyes-free touchscreen typing where users imagine a QWERTY keyboard somewhere on the device and tap out an entire sentence without seeing the keyboard and without receiving any intermediate feedback about letters or words typed. This contrasts with existing approaches for blind users — the split-tapping method of VoiceOver (slow), Braille-based chording input like Perkinput and BrailleTouch (requiring learning a new input method), and word-at-a-time systems like Fleksy (requiring intermediate confirmation). The key innovation is deferring all recognition until the complete sentence is entered, avoiding the distraction of letter-by-letter or word-by-word feedback. The decoder algorithm uses three components: a keyboard topology model with 2D Gaussian distributions for each key, a point transformation algorithm that geometrically compensates for the user's imagined keyboard position (scaling, translating, and rotating tap points), and a 9-gram character language model trained on billions of words from Twitter, Usenet, and blog data (9.8M parameters, 67MB compressed).
Key findings
The decoder achieved an 18.5% character error rate on blind touchscreen typing from novice users, and on one-third of all sentences, the highly noisy input was decoded with zero errors. The point transformation step is critical: since users imagine the keyboard at an arbitrary position, size, and angle on the screen, the raw tap coordinates do not correspond to any fixed keyboard layout. The algorithm compensates by searching over possible scaling (x and y independently), translation (horizontal and vertical), and rotation (up to 20 degrees) transformations, plus adjustments to the Gaussian variance. The decoder uses a grid search to find the optimal transformation, then applies greedy fixed decoding. The approach demonstrates that statistical language models can recover intended text from extremely noisy spatial input — effectively treating the entire sentence as a single recognition problem rather than a sequence of individual key presses.
Relevance
This research proposes a fundamentally different paradigm for eyes-free text input on touchscreens. Rather than making each key press accessible (as VoiceOver does) or requiring users to learn an alternative input method (as Braille-based systems do), it lets users type naturally on an imagined keyboard and relies on computational intelligence to decode the noisy input. This approach could dramatically increase text entry speed for blind users compared to the 4.5 WPM typically achieved with VoiceOver keyboard tapping. The 18.5% error rate is still too high for practical use, but the one-third of sentences decoded perfectly demonstrates the approach's potential. For accessibility practitioners, the key insight is that powerful language models can compensate for imprecise motor input — a principle applicable beyond visual impairment to motor impairments and other conditions affecting touch accuracy. The paper is a brief feasibility study with offline decoding, but it laid groundwork for real-time eyes-free typing systems.
Tags: visual impairment · text entry · eyes-free interaction · touchscreen · keyboard · language model · mobile accessibility