1. ASSESSING THE EFFORT OF REPAIRING THE
ACCESSIBILITY OF WEB SITES
Nádia Fernandes, Luís Carriço
University of Lisbon
ICCHP, Linz, Austria, July 11-13, 2012

2. ICCHP, Linz, Austria, July 13, 2012 2

3. ICCHP, Linz, Austria, July 13, 2012 3
Introduction
• 40-50% of the Web content uses templates.
• Evaluations are performed in pages as a whole.
• Obfuscating results.
• Available metrics may be misleading.

4. ICCHP, Linz, Austria, July 13, 2012 4
Objectives
1. Assess the effort of repairing a site’s accessibility that was
originally developed using templates
Requirements:
• Develop a new metric,
• Template detection algorithm
2. Introducing accessible templates as a form of rapidly
repairing a page

5. ICCHP, Linz, Austria, July 13, 2012 5
The Accessibility Repairing Effort Metric (AREM)
The metric considers the sum of the number of fails and
warnings reported by accessibility evaluation
techniques, excluding repeated instances.
“Primitive elements”- elements of a page, when an element that
is part of a template is only considered once.

6. ICCHP, Linz, Austria, July 13, 2012 6
The Accessibility Repairing Effort Metric (AREM)
The purpose of this metric is:
1. assess the quality measurement of the accessibility of site
construction (effort person.month);
2. and not the perceived quality of the site towards end-users.

7. ICCHP, Linz, Austria, July 13, 2012 7
The Platform for Accessibility Evaluation
Basis:
• QualWeb evaluator
• Fast Match algorithm

8. ICCHP, Linz, Austria, July 13, 2012 8
Procedure
1. The DOM trees are obtained
2. The pages are compared using the Fast Match algorithm
• Comparative function
• Result: The “primitive” nodes
3. The accessibility evaluation is executed (QualWeb evaluator)
4. Reports generation (new reports)
5. The repairing effort estimation according to AREM is computed.

9. ICCHP, Linz, Austria, July 13, 2012 9
Experimental Study
• Objective: understand the advantage of using AREM and
validate the template detection method.
• Object of study: 15 sites with templates (Alexa Top 100)
5 high 5 low
level level
5 medium
level

10. ICCHP, Linz, Austria, July 13, 2012 10
Experimental Study
• QualWeb was applied with the template aware option set.
• The values for the AREM using the element primitiveness.
• We used a conservative rate metric to obtain values of
accessibility quality to compare with AREM.
rate conservative =

11. ICCHP, Linz, Austria, July 13, 2012 11
Metrics Results – AREM (I)
100000
80000
60000
40000
20000
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Web sites
Non-primitive Primitive
1 – Higher level of template usage, 15 – Lower level of template usage

12. ICCHP, Linz, Austria, July 13, 2012 12
Metrics Results - Conservative rate (I)
0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Web sites
Non-primitive Primitive
1 – Higher level of template usage, 15 – Lower level of template usage

13. ICCHP, Linz, Austria, July 13, 2012 13
Metrics Results – AREM (II)
100000
80000
60000
40000
20000
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Web sites
Non-primitive Primitive
1 – Higher level of template usage, 15 – Lower level of template usage

14. ICCHP, Linz, Austria, July 13, 2012 14
Metrics Results - Conservative rate (II)
0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Web sites
Non-primitive Primitive
1 – Higher level of template usage, 15 – Lower level of template usage

15. ICCHP, Linz, Austria, July 13, 2012 15
Metrics Results - Conservative rate (III)
0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05
0.00
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Web sites
Non-primitive Primitive
1 – Higher level of template usage, 15 – Lower level of template usage

16. ICCHP, Linz, Austria, July 13, 2012 16
Validating the Approach
% errors on template
Web sites % template
detection
Higher level of 56% 14%
template detection 51% 8%
Mid level of template 33% 6%
detection 31% 12%
Lower level of 18% 7%
template detection 15% 1%
The number of incorrect elements detected by Fast
Match algorithm is less than 10% (average).

17. ICCHP, Linz, Austria, July 13, 2012 17
Discussion
• The validation of the template detection yielded a deviation.
• The algorithm had more failures in sites which use more
template based components.
• Regarding the AREM metric:
• The difference between the computed values is high;
• Less 30% of repairing issues (primitive elements);
• Depending on the sites this value can decrease substantially.
• These results confirm and support our previous experiment’s
results.

18. ICCHP, Linz, Austria, July 13, 2012 18
Conclusion
• Templates can be very important, reducing the effort of
correction.
• The metric defined is a real indicator of the work that have to
be done, unlike certain quality metric that can be misleading.
• We performed a validation experiment of both metric and
framework and conclude that the template detection
algorithm has a high efficacy.

19. ICCHP, Linz, Austria, July 13, 2012 19
Future Work
1. Improvements of Fast-Match algorithm to guarantee a
higher accuracy level;
2. A large-scale evaluation of the fast match algorithm.

20. ICCHP, Linz, Austria, July 13, 2012 20
Thank you
nadiaf@di.fc.ul.pt