Exploring Aural Navigation by Screenless Access
Mikaylah Gross, Joe Dara, Christopher Meyer, Davide Bolchini · 2018 · Proceedings of the 15th International Web for All Conference (W4A 2018) · doi:10.1145/3192714.3192815
Summary
This paper introduces screenless access, a paradigm for browsing aural information architectures using one-handed, in-air gestures recognised by an off-the-shelf Myo armband, freeing blind and visually impaired users from the constraint of holding and touching a smartphone screen. The problem is fundamental: blind people typically travel with one hand occupied (cane, guide dog lead, companion’s arm), yet mobile screen readers require continuous two-handed interaction on a small touchscreen. Users must halt travel to use their phones, creating dangerous conflicts between mobility and information access. Screenless access eliminates the screen as an interaction surface entirely — the phone stays in the user’s pocket while the Myo armband (a Bluetooth-connected elastic band using electromyography to classify hand gestures with vibrotactile feedback) recognises six gestures: wave right/left (navigate forward/backward in menus), double tap (select), finger spread (go up in hierarchy/cancel), and fist rotate clockwise/counterclockwise (next/previous item in the same level). The system, called AudioSword, implements two navigation concepts: Binary Aural Browsing (dichotomically splitting menu items into left and right sets for spatial anchoring, enabling faster access than linear scanning) and Free Flows (automatic sequential reading of menu items, similar to "auto playlists," reducing gesture effort). Each gesture triggers an earcon (audio cue) providing immediate feedback about the action performed. The gesture-to-action mapping was iteratively refined over six months with both sighted and blind users.
Key findings
A study with ten blind and visually impaired participants (ages 26-61, 5 totally blind, 5 light perception) at a workforce development organisation completed five increasingly complex navigation tasks using a Yelp-like information architecture. The average task success rate was 76.7%, with simpler tasks (1-3 steps) achieving 89% success and more complex ones (7+ steps) dropping to 61%. The SUS usability score was 83.9 — well above the "good" threshold of 68. NASA-TLX scores showed low mental and physical demand, low stress, and low effort across all tasks, though pace was perceived as too slow (TTS was set to ~200 words per minute for researcher note-taking, slower than participants’ usual screen reader speed). Errors fell into two categories: system errors (54.7% were missed gesture recognition, 22.3% incorrect recognition) and design errors (54.3% were gestures performed incorrectly by users). Participants exhibited two distinct navigation styles: autonomous (69% of task instances — rapid, confident, self-correcting) and step-by-step (22% — methodical, waiting for feedback between each gesture). Critically, participants spontaneously adapted gestures to fit their comfort: curling fingers during wave gestures, tapping thumb to four fingers instead of just the middle finger for double tap, rotating hands to different orientations, and performing fist-and-rotate simultaneously rather than sequentially. The system recognised all these unprompted adaptations. Participants adopted four distinct arm-body postures (hip level, elbow pivot, torso level, neck level) without being instructed, all of which worked for gesture recognition. Qualitative feedback revealed strong enthusiasm: "I don’t have to pull my phone out of my pocket and mess with it. I can just do a gesture and be done" (P6). Privacy was a major motivator — no visible screen means no one can see what you’re doing. However, participants raised concerns about social conspicuousness of arm gestures ("People are going to see me doing these gestures and be like ‘what is with him?’" P4) and physical fatigue from prolonged arm movement.
Relevance
This paper challenges a foundational assumption in mobile accessibility: that blind users must interact through a touchscreen. By demonstrating that complex web navigation can be performed proficiently through in-air gestures alone, with low cognitive load and high user satisfaction, it opens a design space for truly hands-free mobile interaction. For accessibility practitioners, several findings are directly applicable: the Binary Aural Browsing approach to structuring audio menus spatially (left/right split) rather than linearly; the use of earcons for gesture feedback; the importance of allowing gesture customisation (users will adapt gestures to their comfort, and systems should accommodate this); and the value of Free Flows for reducing interaction effort during content scanning. The social conspicuousness concern is a critical design constraint that mirrors findings in other wearable accessibility research — assistive devices that draw attention can be rejected regardless of their functional benefits. As wearable sensors become smaller and more accurate, the screenless access paradigm could extend beyond the Myo armband to smartwatches, rings, or textile-integrated sensors that are less visually conspicuous.
Tags: blind · visual impairment · gesture interaction · wearable technology · mobile accessibility · screen reader · aural navigation · information architecture · electromyography · hands-free