Measuring Text Comprehension for People with Reading Difficulties Using a Mobile Application
Andreas Säuberli · 2021 · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3441852.3476474
Summary
This extended abstract presents a mobile touchscreen application designed to measure text comprehension for people with reading difficulties, specifically targeting users of Easy Language — simplified text used to make information accessible to people with intellectual disabilities, prelingual hearing impairments, non-native readers, and others. Traditional comprehension testing is expensive, time-consuming, and difficult to implement with these target groups due to inter-reader variability and the need for large sample sizes. The researcher developed a cross-platform app (Android, iOS, and web) using Flutter that presents texts and comprehension questions, records response data including reading times, and sends results to a server. The app was designed for independent use: participants register via QR code, are never asked for personal data, and response data is completely anonymous. The application supports multiple-choice comprehension questions and cloze tests, with task implementations iteratively tested by experts familiar with people with intellectual disabilities. To validate the app, the researcher conducted an experiment with 16 participants (8 female, 8 male, ages 18-38) with mild to moderate intellectual disabilities recruited from an educational program in Austria. Participants read eight short Easy German news articles — each between 8 and 12 sentences — both on paper and on a tablet (iPad), answering three comprehension questions per text across two sessions over 10 days.
Key findings
Using Many-Facet Rasch Measurement to model the interaction between participant proficiency, item difficulty, and test conditions, the study found 0.00 separation between the app and paper conditions — meaning there was no difference in task difficulty between the mobile application and traditional paper-and-pencil testing. There was no significant interaction between items and conditions either. While more questions were answered correctly on paper than on the tablet, Rasch analysis attributed this to randomization effects rather than modality differences. Reading times were significantly faster on the tablet, with a linear mixed model confirming a 9.97-second difference (p < 0.001) favoring the app condition. Participants rated the Easy Language texts as very easy overall (mean 4.37 out of 5), suggesting the text sample may have been too homogeneous in difficulty. The study provides initial evidence that computer-based testing is a valid measurement tool for comprehension assessment with people with intellectual disabilities — an important finding because it enables testing in participants' familiar environments rather than lab settings, allows automatic timing, and scales to larger sample sizes needed for reliable readability research.
Relevance
This work addresses a methodological gap in accessibility research: how to reliably measure whether simplified texts actually achieve their goal of being comprehensible to target audiences. For practitioners working on plain language or Easy Language content, validated comprehension testing tools are essential — without them, text simplification relies on formulas like Flesch Reading-Ease that have been criticized for oversimplifying what makes text comprehensible. The finding that mobile-based testing produces equivalent results to paper testing is practically significant because it removes barriers to conducting comprehension research with hard-to-reach populations, enabling data collection in familiar environments rather than laboratories. For organizations producing Easy Language content (government agencies, healthcare providers, legal services), this type of tool could support quality assurance by empirically testing whether their simplified materials are actually understood by the intended audience. The open-source availability of the application further lowers barriers to adoption in accessibility research.
Tags: readability · intellectual disability · easy language · reading accessibility · cognitive accessibility · mobile application · comprehension assessment