Creating 'a Simple Conversation': Designing a Conversational User Interface to Improve the Experience of Accessing Support for Study
Francisco Iniesto, Tim Coughlan, Kate Lister, Peter Devine, Nick Freear, Richard Greenwood, Wayne Holmes, Ian Kenny, Kevin McLeod, Ruth Tudor · 2023 · ACM Transactions on Accessible Computing · doi:10.1145/3568166
Summary
This paper presents the ADMINS (Assistants for the Disclosure and Management of Information about Needs and Support) project, which developed a virtual assistant named "Taylor" to help students disclose disabilities and arrange support at The Open University (UK). The OU supports over 20,000 students with declared disabilities, who previously had to complete a 17-question static web form (Disability Support Form, or DSF) to access accommodations. The researchers conducted two trials. A beta trial with 22 students who had already disclosed disabilities and 3 disability support advisors identified accessibility challenges and user experience issues. After implementing improvements, a larger main trial was conducted with 134 newly registering students who disclosed disabilities using both the VA and the existing form-based process in a counterbalanced within-participants design. The VA was built using Microsoft Azure Cognitive Services, including LUIS (Language Understanding) for intent recognition and QnA Maker for handling FAQs. A key design feature is multimodality: students can interact via text or speech for both input and output. The conversational flow covers disability disclosure, alternative study formats, tutorial arrangements, exam accommodations, and communication preferences. The VA produces a summary at the end that advisors can review. The design was informed by participatory methods including workshops with 13 students and 8 advisors, and analysis of 46 recorded telephone conversations between advisors and students. The research addresses a gap in understanding how CUIs can reduce the administrative burden that disproportionately affects people with disabilities—a burden characterized by restricted pathways, unequal outcomes, high effort, time dependencies, negative emotional responses, and multiplicity of processes.
Key findings
The main trial showed a clear preference for the VA: 64.9% of participants preferred the virtual assistant compared to 23.9% who preferred the form (11.2% had no preference). This preference was statistically significant (p < .001) and did not vary by disability type or gender. Technology Acceptance Model (TAM) scores were positive across all dimensions: ease of use (5.66/7), perceived usefulness (5.28/7), attitude (5.42/7), and intent to use (5.35/7). The custom Conversational User Interface Accessibility Questionnaire (CUIAQ) showed overall positive accessibility (5.61/7), with high scores for conversation sequence (6.03) and ease of navigation (5.96), but lower scores for assistive technology compatibility (4.79). Qualitative feedback revealed why participants valued the VA: it felt like "having an actual conversation with somebody," was faster to complete (40 minutes vs. days for the form), and the multimodal input/output helped those with dyslexia who appreciated hearing information spoken while also seeing it written. However, participants who preferred the form valued the open text fields that allowed more detailed responses, and the ability to review and edit answers. Some felt the VA constrained their responses and that automated processing might lose nuance that a human reader would understand. Accessibility barriers identified included: text-to-speech not working reliably on some devices, the VA being overly verbose, misinterpreting user input (e.g., "help" as a request for disability support rather than conversation assistance), and limited ability to expand on closed yes/no questions.
Relevance
This research demonstrates that conversational interfaces can meaningfully improve the experience of administrative processes for people with disabilities. The finding that nearly two-thirds preferred the VA to forms—despite the VA being a proof-of-concept with known limitations—suggests significant potential for CUIs in administrative accessibility. For practitioners designing CUIs, the study identifies four ways CUIs can benefit accessibility: flexibility (speech or text input/output, adjustable pace), assistance (interactive clarification), focus (one topic at a time), and alternative pathways (for those who find forms inaccessible). However, the research also cautions that CUIs introduce new barriers: verbosity that is hard to process, misinterpretation of user intent, unreliable speech recognition, and narrow response options that constrain expression. The CUIAQ instrument developed for this study provides a validated tool for evaluating CUI accessibility, covering sequence, navigation, predictability, AT compatibility, preference preservation, demandingness, response time, option clarity, and communication modalities. Limitations include the low representation of students with visual impairments (0.8%), speech impairments (3%), and hearing impairments (6%) in the main trial sample, despite these being groups that might particularly benefit from or face barriers with CUIs. The study was also conducted during COVID-19, which may have influenced preferences for online tools.
Tags: conversational user interface · chatbot · virtual assistant · disability disclosure · higher education · administrative burden · accessibility
Standards referenced: WCAG