Presentation at the Open University's Computers and Learning Research Group (CALRG) Conference 2015 on Learning Analytics and Accessibility - detecting accessibility deficits with Learning Analytics approaches

1. Learning Analytics and Accessibility – what
can be done and pragmatic considerations
Martyn Cooper (IET)
CALRG Conference 2015

2. Introduction
• Work is part of the eSTEeM project LA4DS-STEM
• Learning Analytics (LA) requires “Big Data”
• Particular interest in retention and pass rates
• Research Questions:
• What can LA approaches tell us about the
accessibility (to disabled students) of modules?
• If they tell us anything is the approach useful?

3. The Central Hypothesis
• In modules where the completion rate and the pass rate
are significantly lower for disabled students than for
nondisabled students then this is indicative of
accessibility challenges in that module
Questions:
• What is significance indicated by in this case?
• How confident are we that we are not measuring other
factors that impact on performance such as: motivation;
family circumstances; ability; educational background; etc.?

4. The Data Set
• The “big data”:
–All Science and MCT Modules from presentation
2009B to and including 2013J (5 years)
–1452 presentations in total
–% Completion and % Pass Rates
–Disabled students vs Non-disabled students
–Data set needed some “cleaning-up” before analysis
Disabled
No Yes
Module Presentation Total No. % complete % pass No. % complete % pass
A 2009B 1827 1605 63.4 61.6 222 56.3 52.7
B 2009J 2609 2282 67.4 66.3 327 59.0 56.9
C 2010B 1662 1492 61.5 60.1 170 60.6 58.2
… … … … … … … … …

5. Odds Ratios Explanation
• Need a statistical useful comparison
–Using odds ratios [J.T.E. Richardson personal communication]
• If the probability of the members of Group 1 exhibiting a
particular outcome is p then the odds of this are p/(1 − p)
• If the probability of the members of Group 2 exhibiting
that outcome is q, then the odds of this are q/(1 − q)
• The odds ratio is the ratio between these odds (i.e. [p/(1
− p)]/[q/(1 − q)], which equals [p(1 − q)]/[q(1 − p)])
• Odds ratios vary from 0 (when p = 0 or q = 1) to infinity
(when p = 1 or q = 0)

6. Odds Ratios Explanation cont.
• An odds ratio of 1 means that there is no difference in
the odds of the two groups’ members exhibiting the
outcome (when p = q)
• An odds ratio less than 1 means that the members of
Group 1 are less likely to exhibit the outcome than are
the members of Group 2; and an odds ratio greater than
1 means that the members of Group 1 are more likely to
exhibit the outcome than are the members of Group 2
• N.B. - Whether an odds ratio is significantly different
from 1 depends on the odds ratio itself and on the
number of members in each group

7. Completion % All Modules
-60.0
-40.0
-20.0
0.0
20.0
40.0
60.0
80.0
100.0
120.0
1
32
63
94
125
156
187
218
249
280
311
342
373
404
435
466
497
528
559
590
621
652
683
714
745
776
807
838
869
900
931
962
993
1024
1055
1086
1117
1148
1179
1210
1241
1272
1303
1334
1365
1396
1427
Disabled-Nondisabled % Complete
Disabled-Nondisabled % Complete
Cases % difference
> 35% when low
number disabled
students registered

8. Completion % - Modules with
> 25 disabled students registered
-30.0
-20.0
-10.0
0.0
10.0
20.0
30.0
40.0
1
15
29
43
57
71
85
99
113
127
141
155
169
183
197
211
225
239
253
267
281
295
309
323
337
351
365
379
393
407
421
435
449
463
477
491
505
519
533
547
561
575
589
603
617
631
645
659
Nondisabled-Disabled % Complete
Nondisabled-Disabled % Complete

9. Odds Ratios Completion
All Modules in Data Set
0.000
2.000
4.000
6.000
8.000
10.000
12.000
14.000
16.000
18.000
20.000
1
36
71
106
141
176
211
246
281
316
351
386
421
456
491
526
561
596
631
666
701
736
771
806
841
876
911
946
981
1016
1051
1086
1121
1156
1191
1226
1261
1296
1331
Odds Ratios Complete (>1 if non-disabled outperform disabled)
Odds Ratios Complete (>1 if non-disabled outperform
disabled)
All cases Odds Ratio
>10 occur when low
number of disabled
students registered

10. Thresholds for Decisions
• The learning analytics needs to lead to a decision about
which modules include significant accessibility barriers
for remedial action
– The LA can tell you where a problem might be not what it is
• Thresholds for decision making are arbitrary but
informed by the data
– A reasonable threshold for identifying accessibility problems
seems to be an odds ratio of 4.0 or more

11. Future Work
• Liaison with Science Accessibility Specialist particularly
with reference to S104 and planning for the new Level 1
gateway modules
• Liaison with the Science and MCT Data Wranglers to
look for any correlation with the their data
• Focus group(s) with MCT and Science staff of mock-ups
of Learning Analytics Dashboards
• Paper for LAK16 comparing with qualitative data from
end of module surveys

12. Discussion Points
• Learning Analytics approaches seem to be able to
identify major accessibility issues in modules
– However this needs testing and only possible by a detailed
accessibility assessment of the module’s media and activities
(this work not currently funded)
• LA approaches only valid on modules with a significant
number of disabled students – suggest a minimum of 25
• Even with 25 disabled students per module really need
to evaluate over multiple presentations to identify issues
– Does this mean the approach is less useful than responding to
student complaints, or proper accessibility evaluation in
production, etc.?

13. Questions

14. Institute of Educational Technology
The Open University
Walton Hall
Milton Keynes
MK7 6AA
www.open.ac.uk