This document evaluates the effectiveness of the Accessibility Evaluation Assistant (AEA), a tool designed to support novice auditors in evaluating website accessibility. A trial was conducted with 38 undergraduate students using the AEA and WCAG 2.0 methods. Results showed the AEA led to higher reliability, validity, and sensitivity in identifying accessibility issues compared to WCAG 2.0. Student feedback indicated the AEA was easier to use and understand than WCAG 2.0. The study demonstrates the AEA is effective for supporting novices but further improvements could be made to reduce errors and ambiguity.

1. Evaluation of the Effectiveness of a Tool
to Support Novice Auditors
Christopher Bailey
Dr. Elaine Pearson
Teesside University
c.p.bailey@tees.ac.uk

2. Practitioner Context
• Computing students need support with accessibility as:
– Lack general awareness of accessibility.
– Minimal inclusion in UG and/or PG curriculum.
– Evaluation is only one element of accessibility.
– Projects include developing live websites.
– Limited face-to-face student/tutor instruction.
– Limited time to dedicate to accessibility.
– Limited access to expertise/end users for testing.
– Need skills to enhance employability.

3. Research Context
• Evaluation report (audit) has motivational and educational value
(Sloan, 2006).
• Manual evaluation is important (WCAG 2.0, UWEM, BW).
• Evaluation support tools aimed at experienced evaluators.
• The expertise level of the evaluator is particularly significant (WCAG
2.0, Brajnik, 2010; BW, Yesilada et al, 2009).
• Fewer false positives and false negatives, faster, and more
confident in judgements.
• Comprehension, Knowledge and Effort (Alonso et al, 2010).

4. Accessibility Evaluation Assistant
• Educational evaluation support tool for novices
• Support functions:
– User Group
– Site Features
– Check Categories
• Structured Walkthrough Method – Translation of expert process
– Title and Summary of Accessibility Principle (Heuristic)
– The User Group(s) affected
– The nature of problem caused and barrier presented
– A step-by-step checking procedure (manual and/or tool support
– Guidance for verification (interpreting results of tool)
– An example video tutorial

5. How Effective is AEA?
• Initial trial showed promising validity and reliability (Bailey &
Pearson, 2011)
• Compare relative effectiveness to WCAG 2.0
• Reliability (reach same decision)
• Validity
– Correctness (in matching experienced evaluator)
– Sensitivity (identifying true barriers)
• Usefulness
• Usability
• Efficiency

6. Trial Methodology
• 38 Undergraduate Computing Students, 12 week elective
Accessibility and Adaptive Technology Module.
• Conducted within constraints of curriculum as an assessment
• 4 Tasks:
– 2 Evaluation Exercises: Sunsail and Harley Davidson Home
Pages
– 2 Reflective Pieces: Personas/User Group, Experience of
Evaluation (Compare Methods, Describe Problems).
• Evaluate 15 AEA Heuristics, equivalent 17 WCAG 2.0 Success
Criteria, relevant to both webpages.
• Check criteria is Met, Not Met or Partly Met and explain/justify their
decision.

7. Results: Reliability
Week 1 Week 2
Grp.
Method/Website Rel. Method/Website Rel.
1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 63%
2 WCAG 2.0 –
67% AEA - Sunsail 78%
Harley Davidson
3 WCAG 2.0 –
AEA - Sunsail 71% 59%
Harley Davidson
4 WCAG 2.0 - Sunsail 70% AEA - Harley Davidson 63%
Week 1 Week 2 Overall
Method WCAG WCAG WCAG
AEA AEA AEA
2.0 2.0 2.0
Reliability 72% 68.5% 70.5% 61% 71.25% 64.75%

8. Results: Validity (Correctness)
Week 1 Week 2
Grp.
Method/Website Validity Method/Website Validity
1 AEA - Harley Davidson 73% WCAG 2.0 - Sunsail 49%
2 WCAG 2.0 – 59% AEA - Sunsail 73%
Harley Davidson
3 AEA - Sunsail 66% WCAG 2.0 – 49%
Harley Davidson
4 WCAG 2.0 - Sunsail 50% AEA - Harley Davidson 62%
Week 1 Week 2 Overall
Method WCAG WCAG WCAG
AEA AEA AEA
2.0 2.0 2.0
Validity 69.5% 54.5% 67.5% 49% 68.5% 51.75%

9. Results: Validity (Sensitivity - AEA)
Sunsail Home Page Harley Davidson Home Page
Heuristic Failure Detection Heuristic Failure Detection
Images of Text 72% Images of Text 88%
Text Size 89% Colour Contrast 59%
Keyboard Navigation 94% Text Size 88%
Skip Navigation 89% Keyboard Navigation 76%
Image Text Skip Navigation 94%
94%
Alternatives Image Text Alternatives 82%
Headings 83% Headings 76%
Form Labels 39% Form Labels 76%
Valid (X)HTML Code 94% Identify Language of 76%
Site Map 100% Page
Average 85% Valid (X)HTML Code 100%
Search Function 94%
Average 83%

10. Results: Validity (Sensitivity)
• Overall Comparison
Sunsail Home Page Harley Davidson Home Page
Method Average Method Average
Detection Rate Detection Rate
AEA 85% AEA 83%
WCAG 2.0 80% WCAG 2.0 74%

11. Qualitative Feeback (AEA)
• Usable, Useful, Efficient?
Positive Negative
Easy/Simple to Use/Concise Check explanation too brief
Easy to Understand/Clear Poor UI Usability/Videos too
Terminology small
Explanation Guides User Checks Require Individual
Judgement
Categorisation/Grouping of Hard to Judge Met/Not Met
Checks
Speed of Check Process No Advice on Solving
Problems
Helpful Videos Bugs and Errors

12. Qualitative Feeback (WCAG)
• Usable, Useful, Efficient?
Positive Negative
Detailed Explanations Confusing/Difficult to
Understand
Linked to Regulation and Complex/Hard to Use
Industry
Real Examples in More
Documentation Knowledge/Experience
Required
Fewer Individual Hard to Judge Relevance
Judgements Required of Check
Easy to Navigate No Explanation for
Documentation Performing Check

13. Findings and Implications
• Use of AEA leads to higher levels of reliability and correctness, and
novices are able to identify true barriers.
• Subjective judgements influence results (evaluator effect).
• We can provide a method, but we still can’t control:
– Thoroughness
– Effort
– Competence
• Novices can relate heuristics to WCAG 2.0 Success Criteria
• Structured approach makes it easier to apply, repeat and learn.
• Complement existing methods for use in industry.

14. Future Work
• Enhance Structured Walkthrough approach to further limit errors:
– Remove ambiguities to reduce misinterpretation
– Enhance guidance for decision making
– Develop a simple metric for determining severity
• Redevelop AEA to increase flexibility of delivery
• Trial in another institution
• Add functionality to formally support WCAG 2.0 evaluation
• Longitudinal studies:
– Further demonstrate effectiveness of AEA
– Barriers which novices consistently identify/miss
– Identify best strategies to teach novices (evaluation, simulation,
demonstration, personas)

15. Student Experience
• “….using the AEA are that you are told precisely
what you are checking clearly and concisely, making
the process faster and simpler. With WCAG 2.0 the
sentences are extremely long…. I had to repeatedly
read the check.”
• “The AEA is very easy to understand, follow and
implement the checks. The step-by-step instructions
are not only helpful, but informative and made my
understand why I should be performing the check”.

16. Student Experience
• “The advantages of the AEA tool are speed, simplicity
and ease-of-use. Using the AEA definitely felt more
effective as it follows the WCAG 2.0 guideline but
streamlines the majority of checks allowing you bypass
the technical jargon and bureaucracy in WCAG.”
• “I found the AEA tool significantly easier and more
intuitive that the WCAG 2.0 approach….it guides the
tester to a relevant section, explains what to look for in
an easy to understand manner as well as explaining the
purpose of the test.”

17. Evaluation of the Effectiveness of a Tool
to Support Novice Auditors
http://arc.tees.ac.uk/aea
Christopher Bailey
Dr. Elaine Pearson
Teesside University
c.p.bailey@tees.ac.uk