2. While you wait…
http://tiny.cc/data-form
»If you haven’t had the chance
to do so, please take some
time to look at the data and
disadvantaged students pre-
session googleform.
02/03/2017 Data and disabled students 2
3. The net of meanings - 1
02/03/2017 Data and disabled students 3
4. The net of meanings - 2
02/03/2017 Data and disabled students 4
5. In this session we will…
Explore :
»Ethical issues
»Disability definitions and consequences
»Potential scenarios
»Your priorities
»A real life case study
02/03/2017 Data and disabled students 5
6. “learning analytics is the measurement, collection,
analysis and reporting of data about learners and their
contexts, for purposes of understanding and optimising
learning and the environments in which it occurs”
SoLAR – Society for Learning Analytics Research
02/03/2017 Data and disabled students 6
8. 86 issues in 9 groups
Group Name Question Main type Importance Responsibility
2 Consent Adverse impact of opting
out on individual
If a student is allowed to opt out of data collection and
analysis could this have a negative impact on their
academic progress?
Ethical 1 Analytics Committee
7 Action Conflict with study goals What should a student do if the suggestions are in conflict
with their study goals?
Ethical 3 Student
8 Adverse impact Oversimplification How can institutions avoid overly simplistic metrics and
decision making which ignore personal circumstances?
Ethical 1 Educational researcher
9. Group Name Question Main type Importance Responsibility
2 Consent Adverse impact of opting
out on individual
If a student is allowed to opt out of data collection and
analysis could this have a negative impact on their
academic progress?
Ethical 1 Analytics Committee
7 Action Conflict with study goals What should a student do if the suggestions are in conflict
with their study goals?
Ethical 3 Student
8 Adverse impact Oversimplification How can institutions avoid overly simplistic metrics and
decision making which ignore personal circumstances?
Ethical 1 Educational researcher
jisc.ac.uk/guides/code-of-practice-for-learning-
analytics
12. Accessibility Considerations for Learning Analytics
1. Remember that learning analytics is not assessment
2. Avoid the labelling of individuals and reinforcing of prejudice and stereotypes
3. Maintain disabled students’ confidentiality
4. Handle the inference of disabilities from the analytics appropriately
5. Ensure that the analytics do not unfairly single out disabled students
6. Use analytics to identify modules where there appear to be accessibility issues
7. Ensure that student-facing analytics are accessible
8. Ensure that interventions are worded appropriately
» https://analytics.jiscinvolve.org/wp/2016/12/14/accessibility-considerations-for-learning-analytics/
Jisc Learning Analytics Accessibility Webinar
13. Contacts
Paul Bailey paul.bailey@jisc.ac.uk
Niall Sclater niall.sclater@jisc.ac.uk
Further Information:
http://www.analytics.jiscinvolve.org
Join: analytics@jiscmail.ac.uk
Jisc Learning Analytics 2017
15. The data discussion
1. How do you encourage disabled
students to disclose?
2. Which of your institutional data
sources might be relevant to
supporting disabled students?
3. Which disabled students are
visible in your data?
4. How mature is the technology?
Discussion:
02/03/2017 Data and disabled students 15
16. Brainstorm Scenarios
» Design of learner data
› What is collected (ethics, level of detail e.g. disability and other co-morbid factors,
process and outcome),
› Design of interface (usability and accessibility),
» Support learner progress
› Documenting/disclosing barriers provides an additional method of early identification for support by
tracking progress,
› Informing course design and learner attainment
› Improving learning and teaching practice
› Comparing progress of disabled versus non disabled learners
» Evaluate institutional/support services
› Usage of institution wide assistive technology (e.g. text to speech)
› Library uptake of productivity software, ebook usage,
02/03/2017 Data and disabled students 16
17. Design: Stakeholder engagement:
Senior manager
»“We've invited a range of stakeholders to
be involved in our learning analytics
steering group – including support staff
and people with accessibility needs”
02/03/2017 Data and disabled students 17
Roll of roles…
Senior manager
18. Design: Learner engagement
Student Union President
»“Students are actively involved in
deciding what information they see on
their own personal dashboard.”
02/03/2017 Data and disabled students 18
Roll of roles…
Senior manager
Student union president
19. Design: Exam arrangements
Examination Officer
»“Some people need access
arrangements for exams. It’s always
involved lots of consultation and
meetings. Now its just a touch of a
button”
02/03/2017 Data and disabled students 19
Roll of roles…
Senior manager
Student union president
Examination officer
20. Support: Consistency
Study skills tutor
»Now there’s less room for students
to slip through the net because all
the support services have the same
information at the same time
so can work together.
02/03/2017 Data and disabled students 20
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
21. Support: Responsiveness
Head of disability service
» Student data helps me track the
progress of students who have disclosed
a barrier to learning so we can respond
more swiftly.
02/03/2017 Data and disabled students 21
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
Head of disability service
22. Support: Prioritise
Dyslexia specialist
» I used to spend ages chasing students
who missed their appointments. Now I
can instantly check their other progress
and leave them alone if they’re
succeeding.
02/03/2017 Data and disabled students 22
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
Head of disability service
Dyslexia specialist
23. Evaluate: Support strategies
Assistive technologist
»As well as asking about a learners
disability we've tried to capture
more specific detail about the
technology strategies recommended
through the assessment.
02/03/2017 Data and disabled students 23
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
Head of disability service
Dyslexia specialist
Assistive technologist
24. Evaluate:Teaching approaches
Lecturer
»I can see how changes to my resources
and activities have impacted on
everyone's engagement, and
particularly benefited my disabled
students.
02/03/2017 Data and disabled students 24
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
Head of disability service
Dyslexia specialist
Assistive technologist
Lecturer
25. Evaluate:Teaching approaches
Learning technologist
»“I can see who is using the learning
platform and how often .This makes it
easy to see where content might be
difficult to access.”
02/03/2017 Data and disabled students 25
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
Head of disability service
Dyslexia specialist
Assistive technologist
Lecturer
Learning technologist
26. Evaluate: Library technology support
Library manager
» We can now monitor the usage/uptake
of enabling technology software in our
library. This helps us to adopt a more
targeted strategy for promotion of
productivity tools to enhance the
support we offer.
02/03/2017 Data and disabled students 26
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
Head of disability service
Dyslexia specialist
Assistive technologist
Lecturer
Learning technologist
Library manager
27. Evaluate: E-books and journals:
Collections manager
»“I can see who is using the resources
such as e-books and e-journals. If
there are any anomalies I can ask why.
By exploring use by different
categories of student we can plan
more effective intervention and
support”
02/03/2017 Data and disabled students 27
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
Head of disability service
Dyslexia specialist
Assistive technologist
Lecturer
Learning technologist
Library manager
Collections manager
28. Evaluate: blended learning
E-learning manager
»“I can begin to correlate
outcomes for disabled student
with online provision in
different subject areas.
Now I have proof that CPD in
blended learning pays
dividends for disabled
students."
02/03/2017 Data and disabled students 28
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
Head of disability service
Dyslexia specialist
Assistive technologist
Lecturer
Learning technologist
Library manager
Collections manager
E-learning manager
29. Evaluate: Data planning
Data analyst
»“I can begin to plan for the future
in ways that can extend what is
currently possible to do.With my
colleagues I can begin to shape
our data to really meet the needs
of a wider group of learners.”
02/03/2017 Data and disabled students 29
Roll of roles…
Senior manager
Student union president
Examination officer
Study skills tutor
Head of disability service
Dyslexia specialist
Assistive technologist
Lecturer
Learning technologist
Library manager
Collections manager
E-learning manager
Data analyst
30. Where this fits in your institution
http://bit.ly/2lFgpwv
02/03/2017 Data and disabled students 30
31. What Can Analytics Contribute to Accessibility in
e-Learning Systems and to Disabled Student’s
Learning
Martyn Cooper, Rebecca Ferguson and Annika Wolff
32. Context to the research
• OU – distance educator
• Larger than average no. of disabled students
• Greater challenges in responding to individual needs of
disabled learners at a distance
• Students can declare a disability. But don’t necessarily know
what type and no two disabled students are the same, anyway
33. The question
• Can learning analytics be used to identify modules with
accessibility deficits?
34. First pass
• Look at average completion rates.
– 1338 modules analysed
– Can show 50% completion rate if 1 of 2 students with declared
disability drops out.
– Low numbers can skew results
– Solution: analyse only modules with >25 disabled students = 668
modules
35. Refined approach using odds ratios
• Odds ratios can determine for 2 groups whether one group is
more or less likely to achieve an outcome than another group.
• It is a relative measure of the odds of one outcome occurring,
given a particular criteria compared to odds of it happening in
absence of the criteria
• In this case:
– Outcome is success of students on the course
36. Using odds ratios to find
accessibility issues
• A bigger odds ratio = bigger disparity between groups
• But - need to find threshold above which you can say there is a problem
Threshold of > 3 looks
sufficient to identify
where accessibility is
most likely factor to
explain difference
37. Summary
• Low numbers make applying statistical measures very difficult
• Not suitable for a large number of modules
• Identifies where there might be a problem – but not how to fix
it
38. Possible Future work
• Use research to find ‘critical learning paths’ to identify
accessibility issues on individual modules.
39. What next
»Follow up email with
› feedback form
› PDF of slides/notes
› Links to Google form,Tricider votes, Niall and Julia blog posts
»Link to recording (if we remembered to press Record!)
»New blog post summarising issues and questions arising
from session.
02/03/2017 Data and disabled students 39
40. jisc.ac.uk
One CastleparkTower Hill Bristol BS2 0JA
customerservices@jisc.ac.uk
T 020 3697 5800
Thank you for listening
Subject specialists Accessibility & Inclusion
Julia.Taylor@jisc.ac.uk
Alistair.McNaught@ jisc.ac.uk
02/03/2017 Data and disabled students 40
Margaret.McKay@ jisc.ac.uk
Notas del editor
“Enshrined in the Jisc Code of practice is the principle that learning analytics should be for the benefit of all students“
Alistair
Holding slide – while you wait
It is night. I am standing alone in 25 acres of scrubland in rural Nicaragua looking at a the Seven Sisters in a sky brimming with stars.
A man with a rifle is walking towards me, demanding to know my business. I speak the Spanish of a 4 year old.
The gun is pointing towards me. I persuade him that I am part of the project team he is supposed to be guarding rather than shooting.
He asks why I am not with the rest of the team in the hut. Why am I out with the crickets, the dust and the snakes?
I explain I love stars and planets.
He tells me there was a news report that day about an asteroid that might hit earth in the 25th century. And what is the difference between an asteroid and a comet?
The chirp of the crickets grows louder as I search my Beginners Spanish vocabulary for the chapter on the formation of the solar system. I pick up a stick and – by the light of his torch - draw the sun in the dust.
I have a brainwave. Use complicated scientific words – they’re nearly all derived from Latin. It works. He is genuinely engaged.
That night I dream I am talking to him and I see his meanings falling through the air. I have a net of language but the weave is wide and many of the meanings slip through without me catching them. I know that as I learn Spanish the weave of the net will become smaller and smaller and the most delicate of meanings will be caught. Then I will more fully know the people to whom I speak.
Alistair
That net of language that catches finer and finer meanings from a speaker’s voice is exactly like the net of data that catches meaning from an institution’s processes, a tutor’s teaching, a library’s content or a learner’s progress. In this session we will look at how we can use the net of meanings to shape practice and personalise support.
Just like there are ethical implications of our communication, our speaking and listening, there are also ethical implications for our net of data. And that’s where we start this session.
Alistair
…have made you aware of the:
ethical issues surrounding learning analytics and their resonance with disabled students
opportunities and issues around disability definitions
range of potential positive scenarios within institutions
considerations relevant to your own institutional ambitions
In universities, colleges and schools we have lots of data, but it has traditionally been stored on paper and in filing cabinets – it’s been hard to access it and to analyse it. But this is changing. Many institutions are now collecting data more consistently and storing it in ways that it can be accessed easily and merged with other datasets to obtain greater insight on the effectiveness of educational processes.
“learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs”
SoLAR – Society for Learning Analytics Research
Jisc’s learning analytics project consists of three core strands: a learning analytics service, a toolkit to help institutions develop their capacities, and a community for sharing experience across institutions.
Jisc has produced a code of practice for learning analytics, built from an extensive literature review. Questions which informed the code include:
If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress?
What should a student do if the suggestions are in conflict with their study goals?
How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances?
The code of practice is available at http://jisc.ac.uk/guides/code-of-practice-for-learning-analytics
A screenshot of Tribal Student Insight, one of the products integrated with Jisc’s learning analytics architecture. It consists of a dashboard showing a student’s predicted risk scores for the modules she is taking.
Screenshots from Study Goal, the Jisc student app for learning analytics, showing an activity feed, course engagement and attainment data and a graph showing how your engagement compares with one of your friends over time.
Accessibility Considerations for Learning Analytics
Remember that learning analytics is not assessment
Avoid the labelling of individuals and reinforcing of prejudice and stereotypes
Maintain disabled students’ confidentiality
Handle the inference of disabilities from the analytics appropriately
Ensure that the analytics do not unfairly single out disabled students
Use analytics to identify modules where there appear to be accessibility issues
Ensure that student-facing analytics are accessible
Ensure that interventions are worded appropriately
https://analytics.jiscinvolve.org/wp/2016/12/14/accessibility-considerations-for-learning-analytics/
Contacts
Paul Bailey paul.bailey@jisc.ac.uk
Niall Sclater niall.sclater@jisc.ac.uk
Further Information:
http://www.analytics.jiscinvolve.org
Join: analytics@jiscmail.ac.uk
Straw poll
University of Wolverhampton identified >3o different sources of support for students.
How can learning analytics support consistency and clarity? If you have different definitions of ‘disability data’ you end up telling different stories.
What data do you collect? Does it have implicit accessibility data (e.g. from bursary,
May not be getting the data due to funding changes (DSA) BUT the data would still be useful – How will you get students to continue to disclose?
There is a need for evidence of impact (OFFA) Learning Analytics has great potential to shape provision
Are learners needs reflected in the data you collect, how could this be improved?
Brainstorm scenarios
Data 'stages' to consider: Deferent roles will play different parts but each will inform the other.
Design of learner data
What is collected (ethics, level of detail e.g. disability and other co-morbid factors, process and outcome),
Design of interface (usability and accessibility),
Support learner progress
Documenting/disclosing barriers provides an additional method of early identification for support by tracking progress,
Informing course design and learner attainment
Improving learning and teaching practice
Comparing progress of disabled versus non disabled learners
Evaluate institutional/support services
Usage of institution wide assistive technology (e.g. text to speech)
Library uptake of productivity software, ebook usage. Etc.
Like to expand on this further - and think about prospective scenarios within a typical institution. In addition … and in true Academy Awards style lets think about the roll of rolls – what key people might be involved or indeed be responsible.
Design stage and in particular Stakeholder engagement
“We've invited a range of stakeholders to be involved in our learning analytics steering group - including support staff and people with accessibility needs” ”
Senior managers might typically be involved in making decisions to initiate this
Design : Learner engagement
“Students are actively involved in deciding what information they see on their own personal dashboard.” (Student Union President might typically be involed )
Here we’re considering the impact of learner advocacy - nothing about us without us
Design stage - Exam arrangements
“Some people need access arrangements for exams. It’s always involved lots of consultation and meetings. Now its just a touch of a button” (Examination Officer)
Support – thinking about Consistency of support
Now there’s less room for students to slip through the net because all the support services have the same information at the same time so can work together. (Study skills support tutor)
Support opportunities: responsiveness.
Student data helps me track the progress of students who have disclosed a barrier to learning so we can respond more swiftly. (Head of disability services)
Support opportunities: Prioritise – don’t chase students who don’t need it!
I used to spend ages chasing students who missed their appointments. Now I can instantly check their other progress and leave them alone if they’re succeeding. (Dyslexia specialist)
Evaluate: Support strategies
As well as capturing information about a learners disability we've tried to find out more specific detail about the individual nature of a learners' need (e.g. keyboard only user, uses screen reading/magnification/text to speech software, has mental health issues) to help evaluate and audit our course demands and learner expectations .
Evaluate: Teaching approaches
I can see how changes to my resources and activities have impacted on everyone's engagement, and particularly benefited my disabled students.
(Lecturer)
Evaluate: Teaching approaches
“I can see who is using the learning platform and how often . This makes it easy to see where content might be difficult to access.”
Evaluate: Library technology support
We can now monitor the usage/uptake of enabling technology software in our library. This helps us to adopt a more targeted strategy for promotion of productivity tools to enhance the support we offer. (Library manager)
Learner data is used to adapt services and where identified to change/improve support services. e.g. access to information on study skills tools, different ways to introduce library services, additional CPD for students and staff on enabling technology approaches, engagement with IT staff (ref Loughbourough Uni Pedestal for Progression project)
In your organisation how likely is this practice to occur: answer on the poll.
A: Already widespread practice
B: Already well developed pockets
C: Likely to be a willingness
D Likely to be resistance
Evaluate: E-books and journals:
““I can see who is using the resources such as e-books and e-journals. If there are any anomalies I can ask why. By exploring use by different categories of student we can plan more effective intervention and support” Collections manager
Evaluate: blended learning“I can begin to correlate outcomes for disabled student with online provision in different subject areas. Now I have proof that CPD in blended learning pays dividends for disabled students.“E-learning manager
Evaluate: Data planning
“I can begin to plan for the future in ways that can extend what is currently possible to do. Witj my colleagues I can begin to shape our data to really meet the needs of a wider group of learners.” Data analyst
5 minute activity
http://bit.ly/2lFgpwv or http://www.tricider.com/brainstorming/2egTN8IYdK7
We are now like to invite you to reflect on some more scenarios in the context of your own organisation and your own role. I’ll add a link to this poll in the chat pane - http://bit.ly/2lFgpwv and we’d like to invite you to visit this page now – we’ll take 5 minutes to:
1.Review the 8 scenarios on the webpage
2.For each of these statements we’d like to invite you to vote for the top 3 that you think are most important for you, that you feel you would prioritise or indeed be the most helpful or achievable.
3.If you have time to - or you wish to – you might feel compelled to add a comment - please do add your own ‘argument’ or viewpoint under the pros and cons section
4.You may alternatively feel that you want to add an additional scenario – one that hasn’t been mentioned here.
Lets take some time and come back in 5 minutes –
This poll will be open for 100 plus days which means that you’ll be able to review what others have added, to reflect on these priorities in the context of your own organisation, share with colleagues and use as We’ve looked at the art of the possible, and now we are going to take the opportunity to hear about an example of practice learn about how one organisation has implemented approaches to use learner data to support disabled students.
What’s possible - Scenarios that make good use of learner information
What Can Analytics Contribute to Accessibility in e-Learning Systems and to Disabled Student’s Learning
Martyn Cooper, Rebecca Ferguson and Annika Wolff
Context to the research
OU – distance educator
Larger than average no. of disabled students
Greater challenges in responding to individual needs of disabled learners at a distance
Students can declare a disability. But don’t necessarily know what type and no two disabled students are the same, anyway
The question
Can learning analytics be used to identify modules with accessibility deficits?
First pass
Look at average completion rates.
1338 modules analysed
Can show 50% completion rate if 1 of 2 students with declared disability drops out.
Low numbers can skew results
Solution: analyse only modules with >25 disbaled students = 668 modules
Refined approach using odds ratios
Odds ratios can determine for 2 groups whether one group is more or less likely to achieve an outcome than another group.
It is a relative measure of the odds of one outcome occurring, given a particular criteria compared to odds of it happening in absence of the criteria
In this case:
Outcome is success of students on the course
Using odds ratios to find accessibility issues
A bigger odds ratio = bigger disparity between groups
But - need to find threshold above which you can say there is a problem
Threshold of > 3 looks sufficient to identify where accessibility is most likely factor to explain difference
Summary
Low numbers make applying statistical measures very difficult
Not suitable for a large number of modules
Identifies where there might be a problem – but not how to fix it
Possible Future work
Use research to find ‘critical learning paths’ to identify accessibility issues on individual modules.
What next?
Follow up email with
feedback form
PDF of slides/notes
Links to Google form, Tricider votes, Niall and Julia blog posts
Link to recording (if we remembered to press Record!)
New blog post summarising issues and questions arising from session.
Thank you for listening
Subject specialists Accessibility & Inclusion
Margaret.McKay@ jisc.ac.uk
Julia.Taylor@jisc.ac.uk
Alistair.McNaught@ jisc.ac.uk