Eye Tracking Scanpath Analysis on Web Pages: How Many Users?
Sukru Eraslan, Yeliz Yesilada, Simon Harper · 2016 · Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications (ETRA) · doi:10.1145/2857491.2857519
Summary
This paper investigates how many participants are needed for reliable eye tracking scanpath analysis on web pages — a question that has been debated in usability research for over 30 years but never specifically addressed for scanpath studies. While the classic "magic number" of five users for usability testing (Nielsen and Landauer, 1993) is widely cited, this figure applies to finding usability problems, not to analysing eye movement sequences. The researchers used their Scanpath Trend Analysis (STA) algorithm, which identifies the most commonly followed visual path across multiple users by analysing fixation sequences in terms of web page visual elements rather than raw pixel coordinates. The STA algorithm differs from other approaches by producing a "trending scanpath" rather than an absolute shared path, making it more robust to individual variation. The study used data from 81 participants across two universities (Manchester and Middle East Technical University) who performed browsing and searching tasks on six web pages of varying visual complexity (Apple, Babylon, AVG, Yahoo, Godaddy, BBC). Pages were segmented into visual elements using the VIPS algorithm, and scanpath similarity was measured using the Levenshtein String-edit distance with a substitution matrix based on Euclidean distances between visual elements.
Key findings
The results demonstrate that approximate scanpath results can be achieved with substantially fewer users than the full sample of 65. To reach 75% similarity to the full group's trending scanpath, 27 users are needed for searching tasks and 34 users for browsing tasks. The difference between task types is significant: searching tasks produce more consistent scanpaths because users have a specific target, while browsing tasks yield more varied eye movements as users attend to whatever attracts their attention without a defined goal. To reach higher similarity thresholds, the numbers increase: 85% similarity requires 45 users for browsing and 36 for searching; 90% requires 51 and 41 respectively; and 95% requires 58 and 46. After 52 users for searching tasks, the trending scanpath stabilises completely. Cross-validation using two independent sets of three web pages confirmed these findings, with average differences between sets of only 8.06% for searching and 11.32% for browsing tasks. The relationship between number of users and similarity follows a power curve (S = 15.936 * i^0.440 for browsing; S = 17.665 * i^0.439 for searching), showing diminishing returns as more participants are added.
Relevance
This study provides practical guidance for researchers planning eye tracking studies on web pages, helping them balance the cost and time of recruiting participants against the reliability of their results. For accessibility research specifically, scanpath analysis can reveal how users with different abilities navigate web content, and knowing the minimum viable sample size makes such studies more feasible. The finding that task type significantly affects the required sample size is important for study design — accessibility studies involving directed information-seeking tasks can use fewer participants than those studying exploratory browsing behaviour. The connection to Experiential Transcoding is particularly relevant for accessibility: understanding trending scanpaths can inform how web pages are re-ordered for small screens or audio presentation, ensuring the most commonly attended content is presented first. The open dataset provided by the researchers also enables other accessibility researchers to conduct secondary analyses without the expense of running new eye tracking sessions.
Tags: eye tracking · scanpath analysis · usability testing · sample size · web accessibility · research methods · visual attention