Evaluator Effect

The phenomenon in accessibility and usability evaluation where different evaluators examining the same interface detect different sets of problems and may reach different conclusions about the same issues. The evaluator effect means that no single evaluation can achieve 100% reliability, as the process requires subjective judgement influenced by each evaluator's experience, background, and interpretation of guidelines. Research has shown that even expert accessibility auditors disagree on findings, and the W3C considers a check reliably testable only when 80% of knowledgeable evaluators agree. The evaluator effect is a key motivation for developing structured evaluation methods and tools that reduce individual variation.