A Large User Pool for Accessibility Research with Representative Users
Marianne Dee, Vicki L. Hanson · 2014 · Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS) · doi:10.1145/2661334.2661361
Summary
This paper reports on the creation and management of the SiDE (Social Inclusion in the Digital Economy) User Pool at the University of Dundee, Scotland — a pool of over 800 older adults and people with disabilities established between 2009 and 2014 to provide accessibility researchers with ready access to representative participants. The paper addresses a fundamental methodological problem in accessibility research: difficulty recruiting representative users in a timely manner, which leads to problematic workarounds including reusing the same participants across studies, simulating impairments with able-bodied proxies, and using inappropriately small sample sizes. The SiDE User Pool was part of a multi-year, multidisciplinary research effort involving the Universities of Dundee and Newcastle, focused on inclusion of older adults and disabled users in rapidly changing technologies. Over the pool's lifetime, 863 individuals were recruited (peak 817 in Fall 2012, 702 current as of Spring 2014), with 694 participating in at least one research study across 32 studies involving 58 activities. The pool was deliberately over-sampled for people with disabilities and targeted both age-related advocacy groups (pensioners' forums) and disability-related organisations (local society for the blind, skills centres for people with disabilities). The pool collaborated with the CREATE team at the University of Miami to use standardised cognitive and abilities test batteries, providing researchers with baseline participant characteristics. A unique institutional feature was the Dundee User Centre, which serves as both a computer training centre for older adults and a research evaluation resource.
Key findings
The paper identifies several critical lessons for establishing and maintaining user pools for accessibility research. Reciprocity was the primary motivation for participation — not financial compensation. Participants genuinely wanted to "give something back" and contribute to research that could help others; many were uncomfortable receiving gift vouchers, viewing participation itself as the reward. Face-to-face recruitment was far more effective than advertising — travelling to local communities and speaking directly with groups produced better-matched, more diverse participants than posters or newspaper ads. A single point of contact (SPOC) — a dedicated pool manager — was essential for building trust with a population that was anxious about cold calls from strangers and nervous about visiting an academic department. The manager, being of similar age to participants, could empathise and build rapport. The pool manager role required personal skills (empathy, patience, communication) as much as organisational ones. The pool fell short of representing the oldest old, people in care homes, minority ethnic communities, and those with the highest support needs. Maintaining engagement was the hardest task — keeping participants informed through newsletters, websites, emails, and phone calls was critical for retention (only 159 of 863 withdrew over four years). A one-day workshop with four user groups — including one group with no interest in technology — proved "eye-opening" for researchers, as the no-interest group challenged technologists' assumption that new technology would be embraced. The goal of involving users throughout all research phases (not just evaluation) was the least successful — researchers tended to seek evaluation participants rather than co-designers, though co-design workshops and focus groups provided some early-stage input.
Relevance
This paper addresses a structural problem in accessibility research that has only intensified: the difficulty of conducting rigorous user studies with representative participants. The problematic workarounds the paper identifies — proxy users, simulated impairments, tiny samples, repeated use of experienced participants — remain widespread and undermine the validity of accessibility research. For accessibility researchers and organisations, the SiDE User Pool model offers practical guidance on building sustainable participant infrastructure. Key insights include: invest in community relationships rather than advertising; hire a dedicated manager with strong interpersonal skills; treat participation as reciprocal rather than transactional; provide a single trusted contact point; use standardised assessment tools for participant characterisation; and recognise that maintaining a pool is harder than creating one. The paper's honest assessment of failures (under-representation of the most vulnerable groups, limited success in involving users beyond evaluation) is as valuable as its successes. The tension between researchers' evaluation-focused needs and the aspiration for genuine participatory design reflects a broader challenge in accessibility research that remains unresolved. The model is particularly relevant for institutions considering longitudinal accessibility research programmes where repeated access to diverse participants with disabilities is essential.
Tags: research methodology · user research · participant recruitment · aging · accessibility research · participatory design · older adults