Stratified Sampling

Stratified sampling is a statistical technique that divides a population into non-overlapping subgroups (strata) that share some characteristic, then draws a random sample from each stratum. In accessibility evaluation, stratified sampling is used to pick test pages by first clustering a site's pages according to their 'error profile' — the distribution of automatically-detected accessibility issues — and then randomly sampling from each cluster. Compared with simple random sampling, this approach is more likely to capture pages that exhibit rare or unusual accessibility issues, which would otherwise be missed. Empirical studies show that stratified sampling with error profiles can reduce WCAG-conformance measurement inaccuracy by tens of percentage points compared with ad hoc or purely random sampling.