SlideShare a Scribd company logo
1 of 8
Are Users the Gold Standard
for Accessibility Evaluation?
7th April - W4A’14
Amaia Aizpurua1, Myriam Arrue Simon Harper2, Markel Vigo3
U. of the Basque Country University of Manchester
http://dx.doi.org/10.1145/2596695.2596705
1: @amaiaaizpurua
2: @sharpic
3: @markelvigo
Motivation
• User testing for accessibility evaluation
• Encouraged by the community
• No established procedures
– What?
– When?
– How?
27th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
In practice
• Evaluating web accessibility for blind users
– User testing method
• Effectiveness in identifying accessibility problems
[Mankoff et al. 2005]
– Accessibility guidelines
• Partial coverage of user problems
[Power et al. 2012]
37th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
Study results
• Accessibility: conformance vs. perceived
Website AA conformance level Perceived
47th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
Median Mode SD
6 7 1.95
6 6 1.42
2 1 1.62
6 6 1.95
1: very inaccessible
7: very accessible
Satisfied SC Non-satisfied SC
73% 27%
69% 31%
52% 48%
36% 64%
Why this mismatch?
• Guidelines vs. user problems
• Eliciting users’ problems
– Experience/report problems differently: web
expertise
– Interaction context, tasks type
• Identifying accessibility issues
– Combination of causes, contextual
information
– Under the influence of the evaluator effect
57th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
Discussion
• Inclusive participatory evaluation
– Following the considerations of co-operative
evaluation
– Evaluator and user evaluating a website
collaboratively
– Instead of interpreting data afterwards, work
on problems during the session
• Improve effectiveness
67th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
Conclusions
• User testing may not be the gold standard
for web accessibility evaluation
• Users are a gold mine
• Co-operative accessibility evaluation
• Bridge the gap?
77th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
Thank you!
87th April - W4A’14
Contact
amaia.aizpurua@ehu.es | @amaiaaizpurua
myriam.arrue@ehu.es
simon.harper@manchester.ac.uk | @sharpic
markel.vigo@manchester.ac.uk | @markelvigo

More Related Content

Viewers also liked

Biographical Prewrite & Outline
Biographical Prewrite & OutlineBiographical Prewrite & Outline
Biographical Prewrite & Outline
eardales
 
Macbeth Rough Draft 3
Macbeth Rough Draft 3Macbeth Rough Draft 3
Macbeth Rough Draft 3
eardales
 
ファーエンドテクノロジーのサービスを支えるOSS
ファーエンドテクノロジーのサービスを支えるOSSファーエンドテクノロジーのサービスを支えるOSS
ファーエンドテクノロジーのサービスを支えるOSS
Mutsumi IWAISHI
 
MCTC A&PII Vessels
MCTC A&PII VesselsMCTC A&PII Vessels
MCTC A&PII Vessels
n k
 
Harry Potter Rough Draft
Harry Potter Rough DraftHarry Potter Rough Draft
Harry Potter Rough Draft
eardales
 

Viewers also liked (11)

Biographical Prewrite & Outline
Biographical Prewrite & OutlineBiographical Prewrite & Outline
Biographical Prewrite & Outline
 
何なのc?
何なのc?何なのc?
何なのc?
 
Macbeth Rough Draft 3
Macbeth Rough Draft 3Macbeth Rough Draft 3
Macbeth Rough Draft 3
 
Jaws ug shimane-1
Jaws ug shimane-1Jaws ug shimane-1
Jaws ug shimane-1
 
ファーエンドテクノロジーのサービスを支えるOSS
ファーエンドテクノロジーのサービスを支えるOSSファーエンドテクノロジーのサービスを支えるOSS
ファーエンドテクノロジーのサービスを支えるOSS
 
MCTC A&PII Vessels
MCTC A&PII VesselsMCTC A&PII Vessels
MCTC A&PII Vessels
 
アプリ開発検証はLXC+Ansibleで楽ちんにやろう!
アプリ開発検証はLXC+Ansibleで楽ちんにやろう!アプリ開発検証はLXC+Ansibleで楽ちんにやろう!
アプリ開発検証はLXC+Ansibleで楽ちんにやろう!
 
いろいろ引き出し作って見ました
いろいろ引き出し作って見ましたいろいろ引き出し作って見ました
いろいろ引き出し作って見ました
 
Frankenstein Outline
Frankenstein  OutlineFrankenstein  Outline
Frankenstein Outline
 
Harry Potter Rough Draft
Harry Potter Rough DraftHarry Potter Rough Draft
Harry Potter Rough Draft
 
DNSって結局何?って人向けのRoute53の話など
DNSって結局何?って人向けのRoute53の話などDNSって結局何?って人向けのRoute53の話など
DNSって結局何?って人向けのRoute53の話など
 

Similar to Are Users the Gold Standard for Accessibility Evaluation?

Privacy Concerns vs. User Behavior in Community Question Answering
Privacy Concerns vs. User Behavior in Community Question AnsweringPrivacy Concerns vs. User Behavior in Community Question Answering
Privacy Concerns vs. User Behavior in Community Question Answering
Nicolas Kourtellis
 

Similar to Are Users the Gold Standard for Accessibility Evaluation? (20)

Towards a Quality Framework for Online Educator Professional Development Acti...
Towards a Quality Framework for Online Educator Professional Development Acti...Towards a Quality Framework for Online Educator Professional Development Acti...
Towards a Quality Framework for Online Educator Professional Development Acti...
 
Crowdsourcing Software Evaluation
Crowdsourcing Software EvaluationCrowdsourcing Software Evaluation
Crowdsourcing Software Evaluation
 
Adaptive software based feedback acquisition a personas-based design
Adaptive software based feedback acquisition a personas-based designAdaptive software based feedback acquisition a personas-based design
Adaptive software based feedback acquisition a personas-based design
 
MOOCs at UVa for eLearn October 2015
MOOCs at UVa for eLearn October 2015MOOCs at UVa for eLearn October 2015
MOOCs at UVa for eLearn October 2015
 
Comparing the Effectiveness of Electronic Diary and UX Curve Methods in Mul...
Comparing the Effectiveness of  Electronic Diary and UX Curve Methods  in Mul...Comparing the Effectiveness of  Electronic Diary and UX Curve Methods  in Mul...
Comparing the Effectiveness of Electronic Diary and UX Curve Methods in Mul...
 
Quality frameworks for e-learning (SIEAD 2018, Brazil)
Quality frameworks for e-learning (SIEAD 2018, Brazil)Quality frameworks for e-learning (SIEAD 2018, Brazil)
Quality frameworks for e-learning (SIEAD 2018, Brazil)
 
Rob howe - assessment strategies in a digital age
Rob howe   - assessment strategies in a digital ageRob howe   - assessment strategies in a digital age
Rob howe - assessment strategies in a digital age
 
User experience at Imperial: a case study of qualitative approaches to Primo ...
User experience at Imperial: a case study of qualitative approaches to Primo ...User experience at Imperial: a case study of qualitative approaches to Primo ...
User experience at Imperial: a case study of qualitative approaches to Primo ...
 
Learning Analytics and MOOCs
Learning Analytics and MOOCsLearning Analytics and MOOCs
Learning Analytics and MOOCs
 
A Challenge to Web Accessibility Metrics and Guidelines: Putting People and P...
A Challenge to Web Accessibility Metrics and Guidelines: Putting People and P...A Challenge to Web Accessibility Metrics and Guidelines: Putting People and P...
A Challenge to Web Accessibility Metrics and Guidelines: Putting People and P...
 
Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...
Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...
Paul Clough Sheffield iSchool Evaluating Info Searching in Digital Cultural H...
 
Rob Howe - The honest journey to full e-submission and feedback using blackboard
Rob Howe - The honest journey to full e-submission and feedback using blackboardRob Howe - The honest journey to full e-submission and feedback using blackboard
Rob Howe - The honest journey to full e-submission and feedback using blackboard
 
Learning Analytics: New thinking supporting educational research
Learning Analytics: New thinking supporting educational researchLearning Analytics: New thinking supporting educational research
Learning Analytics: New thinking supporting educational research
 
IWMW 2002: Designing for Usability
IWMW 2002: Designing for UsabilityIWMW 2002: Designing for Usability
IWMW 2002: Designing for Usability
 
The design of adaptive acquisition of users feedback an empirical study (rcis...
The design of adaptive acquisition of users feedback an empirical study (rcis...The design of adaptive acquisition of users feedback an empirical study (rcis...
The design of adaptive acquisition of users feedback an empirical study (rcis...
 
Collecting meaningful feedback on Information literacy training: results of a...
Collecting meaningful feedback on Information literacy training: results of a...Collecting meaningful feedback on Information literacy training: results of a...
Collecting meaningful feedback on Information literacy training: results of a...
 
UX in 10 Minutes - Usability Testing - Presented at SEMPO
UX in 10 Minutes - Usability Testing - Presented at SEMPOUX in 10 Minutes - Usability Testing - Presented at SEMPO
UX in 10 Minutes - Usability Testing - Presented at SEMPO
 
Evidencing compliance of quality training and assessment (SNR 15) through e-l...
Evidencing compliance of quality training and assessment (SNR 15) through e-l...Evidencing compliance of quality training and assessment (SNR 15) through e-l...
Evidencing compliance of quality training and assessment (SNR 15) through e-l...
 
Privacy Concerns vs. User Behavior in Community Question Answering
Privacy Concerns vs. User Behavior in Community Question AnsweringPrivacy Concerns vs. User Behavior in Community Question Answering
Privacy Concerns vs. User Behavior in Community Question Answering
 
How to Conduct Usability Studies: A Librarian Primer
How to Conduct Usability Studies: A Librarian PrimerHow to Conduct Usability Studies: A Librarian Primer
How to Conduct Usability Studies: A Librarian Primer
 

Recently uploaded

Jigani Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangal...
Jigani Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangal...Jigani Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangal...
Jigani Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangal...
amitlee9823
 
Whitefield Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Ba...
Whitefield Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Ba...Whitefield Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Ba...
Whitefield Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Ba...
amitlee9823
 
Anupama Kundoo Cost Effective detailed ppt with plans and elevations with det...
Anupama Kundoo Cost Effective detailed ppt with plans and elevations with det...Anupama Kundoo Cost Effective detailed ppt with plans and elevations with det...
Anupama Kundoo Cost Effective detailed ppt with plans and elevations with det...
sriharipichandi
 
call girls in Dakshinpuri (DELHI) 🔝 >༒9953056974 🔝 genuine Escort Service 🔝✔️✔️
call girls in Dakshinpuri  (DELHI) 🔝 >༒9953056974 🔝 genuine Escort Service 🔝✔️✔️call girls in Dakshinpuri  (DELHI) 🔝 >༒9953056974 🔝 genuine Escort Service 🔝✔️✔️
call girls in Dakshinpuri (DELHI) 🔝 >༒9953056974 🔝 genuine Escort Service 🔝✔️✔️
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
Brookefield Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Brookefield Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Brookefield Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Brookefield Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
amitlee9823
 
RT Nagar Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bang...
RT Nagar Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bang...RT Nagar Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bang...
RT Nagar Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bang...
amitlee9823
 
Vip Mumbai Call Girls Bandra West Call On 9920725232 With Body to body massag...
Vip Mumbai Call Girls Bandra West Call On 9920725232 With Body to body massag...Vip Mumbai Call Girls Bandra West Call On 9920725232 With Body to body massag...
Vip Mumbai Call Girls Bandra West Call On 9920725232 With Body to body massag...
amitlee9823
 

Recently uploaded (20)

Jigani Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangal...
Jigani Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangal...Jigani Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangal...
Jigani Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bangal...
 
The hottest UI and UX Design Trends 2024
The hottest UI and UX Design Trends 2024The hottest UI and UX Design Trends 2024
The hottest UI and UX Design Trends 2024
 
WhatsApp Chat: 📞 8617697112 Call Girl Baran is experienced
WhatsApp Chat: 📞 8617697112 Call Girl Baran is experiencedWhatsApp Chat: 📞 8617697112 Call Girl Baran is experienced
WhatsApp Chat: 📞 8617697112 Call Girl Baran is experienced
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
 
8377087607, Door Step Call Girls In Kalkaji (Locanto) 24/7 Available
8377087607, Door Step Call Girls In Kalkaji (Locanto) 24/7 Available8377087607, Door Step Call Girls In Kalkaji (Locanto) 24/7 Available
8377087607, Door Step Call Girls In Kalkaji (Locanto) 24/7 Available
 
Whitefield Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Ba...
Whitefield Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Ba...Whitefield Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Ba...
Whitefield Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Ba...
 
Anupama Kundoo Cost Effective detailed ppt with plans and elevations with det...
Anupama Kundoo Cost Effective detailed ppt with plans and elevations with det...Anupama Kundoo Cost Effective detailed ppt with plans and elevations with det...
Anupama Kundoo Cost Effective detailed ppt with plans and elevations with det...
 
Hire 💕 8617697112 Meerut Call Girls Service Call Girls Agency
Hire 💕 8617697112 Meerut Call Girls Service Call Girls AgencyHire 💕 8617697112 Meerut Call Girls Service Call Girls Agency
Hire 💕 8617697112 Meerut Call Girls Service Call Girls Agency
 
💫✅jodhpur 24×7 BEST GENUINE PERSON LOW PRICE CALL GIRL SERVICE FULL SATISFACT...
💫✅jodhpur 24×7 BEST GENUINE PERSON LOW PRICE CALL GIRL SERVICE FULL SATISFACT...💫✅jodhpur 24×7 BEST GENUINE PERSON LOW PRICE CALL GIRL SERVICE FULL SATISFACT...
💫✅jodhpur 24×7 BEST GENUINE PERSON LOW PRICE CALL GIRL SERVICE FULL SATISFACT...
 
Gamestore case study UI UX by Amgad Ibrahim
Gamestore case study UI UX by Amgad IbrahimGamestore case study UI UX by Amgad Ibrahim
Gamestore case study UI UX by Amgad Ibrahim
 
call girls in Dakshinpuri (DELHI) 🔝 >༒9953056974 🔝 genuine Escort Service 🔝✔️✔️
call girls in Dakshinpuri  (DELHI) 🔝 >༒9953056974 🔝 genuine Escort Service 🔝✔️✔️call girls in Dakshinpuri  (DELHI) 🔝 >༒9953056974 🔝 genuine Escort Service 🔝✔️✔️
call girls in Dakshinpuri (DELHI) 🔝 >༒9953056974 🔝 genuine Escort Service 🔝✔️✔️
 
Hingoli ❤CALL GIRL 8617370543 ❤CALL GIRLS IN Hingoli ESCORT SERVICE❤CALL GIRL
Hingoli ❤CALL GIRL 8617370543 ❤CALL GIRLS IN Hingoli ESCORT SERVICE❤CALL GIRLHingoli ❤CALL GIRL 8617370543 ❤CALL GIRLS IN Hingoli ESCORT SERVICE❤CALL GIRL
Hingoli ❤CALL GIRL 8617370543 ❤CALL GIRLS IN Hingoli ESCORT SERVICE❤CALL GIRL
 
Call Girls Jalgaon Just Call 8617370543Top Class Call Girl Service Available
Call Girls Jalgaon Just Call 8617370543Top Class Call Girl Service AvailableCall Girls Jalgaon Just Call 8617370543Top Class Call Girl Service Available
Call Girls Jalgaon Just Call 8617370543Top Class Call Girl Service Available
 
VIP Model Call Girls Kalyani Nagar ( Pune ) Call ON 8005736733 Starting From ...
VIP Model Call Girls Kalyani Nagar ( Pune ) Call ON 8005736733 Starting From ...VIP Model Call Girls Kalyani Nagar ( Pune ) Call ON 8005736733 Starting From ...
VIP Model Call Girls Kalyani Nagar ( Pune ) Call ON 8005736733 Starting From ...
 
Brookefield Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Brookefield Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Brookefield Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Brookefield Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
 
Pooja 9892124323, Call girls Services and Mumbai Escort Service Near Hotel Hy...
Pooja 9892124323, Call girls Services and Mumbai Escort Service Near Hotel Hy...Pooja 9892124323, Call girls Services and Mumbai Escort Service Near Hotel Hy...
Pooja 9892124323, Call girls Services and Mumbai Escort Service Near Hotel Hy...
 
Sector 105, Noida Call girls :8448380779 Model Escorts | 100% verified
Sector 105, Noida Call girls :8448380779 Model Escorts | 100% verifiedSector 105, Noida Call girls :8448380779 Model Escorts | 100% verified
Sector 105, Noida Call girls :8448380779 Model Escorts | 100% verified
 
Q4-W4-SCIENCE-5 power point presentation
Q4-W4-SCIENCE-5 power point presentationQ4-W4-SCIENCE-5 power point presentation
Q4-W4-SCIENCE-5 power point presentation
 
RT Nagar Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bang...
RT Nagar Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bang...RT Nagar Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bang...
RT Nagar Call Girls Service: 🍓 7737669865 🍓 High Profile Model Escorts | Bang...
 
Vip Mumbai Call Girls Bandra West Call On 9920725232 With Body to body massag...
Vip Mumbai Call Girls Bandra West Call On 9920725232 With Body to body massag...Vip Mumbai Call Girls Bandra West Call On 9920725232 With Body to body massag...
Vip Mumbai Call Girls Bandra West Call On 9920725232 With Body to body massag...
 

Are Users the Gold Standard for Accessibility Evaluation?

  • 1. Are Users the Gold Standard for Accessibility Evaluation? 7th April - W4A’14 Amaia Aizpurua1, Myriam Arrue Simon Harper2, Markel Vigo3 U. of the Basque Country University of Manchester http://dx.doi.org/10.1145/2596695.2596705 1: @amaiaaizpurua 2: @sharpic 3: @markelvigo
  • 2. Motivation • User testing for accessibility evaluation • Encouraged by the community • No established procedures – What? – When? – How? 27th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
  • 3. In practice • Evaluating web accessibility for blind users – User testing method • Effectiveness in identifying accessibility problems [Mankoff et al. 2005] – Accessibility guidelines • Partial coverage of user problems [Power et al. 2012] 37th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
  • 4. Study results • Accessibility: conformance vs. perceived Website AA conformance level Perceived 47th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation? Median Mode SD 6 7 1.95 6 6 1.42 2 1 1.62 6 6 1.95 1: very inaccessible 7: very accessible Satisfied SC Non-satisfied SC 73% 27% 69% 31% 52% 48% 36% 64%
  • 5. Why this mismatch? • Guidelines vs. user problems • Eliciting users’ problems – Experience/report problems differently: web expertise – Interaction context, tasks type • Identifying accessibility issues – Combination of causes, contextual information – Under the influence of the evaluator effect 57th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
  • 6. Discussion • Inclusive participatory evaluation – Following the considerations of co-operative evaluation – Evaluator and user evaluating a website collaboratively – Instead of interpreting data afterwards, work on problems during the session • Improve effectiveness 67th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
  • 7. Conclusions • User testing may not be the gold standard for web accessibility evaluation • Users are a gold mine • Co-operative accessibility evaluation • Bridge the gap? 77th April - W4A’14 Are Users the Gold Standard for Accessibility Evaluation?
  • 8. Thank you! 87th April - W4A’14 Contact amaia.aizpurua@ehu.es | @amaiaaizpurua myriam.arrue@ehu.es simon.harper@manchester.ac.uk | @sharpic markel.vigo@manchester.ac.uk | @markelvigo