Are Users the Gold Standard for Accessibility Evaluation?
1. Are Users the Gold Standard
for Accessibility Evaluation?
7th April - W4A’14
Amaia Aizpurua1, Myriam Arrue Simon Harper2, Markel Vigo3
U. of the Basque Country University of Manchester
http://dx.doi.org/10.1145/2596695.2596705
1: @amaiaaizpurua
2: @sharpic
3: @markelvigo
2. Motivation
• User testing for accessibility evaluation
• Encouraged by the community
• No established procedures
– What?
– When?
– How?
27th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
3. In practice
• Evaluating web accessibility for blind users
– User testing method
• Effectiveness in identifying accessibility problems
[Mankoff et al. 2005]
– Accessibility guidelines
• Partial coverage of user problems
[Power et al. 2012]
37th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
4. Study results
• Accessibility: conformance vs. perceived
Website AA conformance level Perceived
47th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
Median Mode SD
6 7 1.95
6 6 1.42
2 1 1.62
6 6 1.95
1: very inaccessible
7: very accessible
Satisfied SC Non-satisfied SC
73% 27%
69% 31%
52% 48%
36% 64%
5. Why this mismatch?
• Guidelines vs. user problems
• Eliciting users’ problems
– Experience/report problems differently: web
expertise
– Interaction context, tasks type
• Identifying accessibility issues
– Combination of causes, contextual
information
– Under the influence of the evaluator effect
57th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
6. Discussion
• Inclusive participatory evaluation
– Following the considerations of co-operative
evaluation
– Evaluator and user evaluating a website
collaboratively
– Instead of interpreting data afterwards, work
on problems during the session
• Improve effectiveness
67th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
7. Conclusions
• User testing may not be the gold standard
for web accessibility evaluation
• Users are a gold mine
• Co-operative accessibility evaluation
• Bridge the gap?
77th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
