Presentation at humane event on digital transformation in higher education (http://www.humane.eu/events/seminars-and-conferences/2018/aveiro-042018/).
Learning analytics is hot. But are learning dashboards scalable and sustainable solutions for providing actionable feedback to students? Is learning analytics applicable in more traditional higher education settings? This talk will share experiences and lessons learned from two European projects (ABLE and STELA) that aimed at developing learning dashboards for more traditional higher education institutions and integrating it within actual educational practices. The talk will challenge your beliefs regarding “chances of success” and predictive models in higher education.
Learning dashboards for actionable feedback: the (non)sense of chances of success and predictive models
1. Learning dashboards for
actionable feedback
the (non)sense of chances of success and predictive models
Tinne De Laet
Tinne.DeLaet@kuleuven.be
@TinneDeLaet
2. “Learning analytics is about
collecting traces that learners
leave behind and using those
traces to improve learning.”
- Erik Duval
Learning Analytics and Educational Data Mining, Erik Duval’s Weblog, 30 January 2012, https://erikduval.wordpress.com/2012/01/30/learning-analytics-and-educational-data-mining/ 2
Learning Analytics?
3. Learning Dashboards?
3Dashboard Confusion, Stephen Few, Intelligent Enterprise, March 20, 2004
“A dashboard is a visual display of the
most important information needed to
achieve one or more objectives;
consolidated and arranged on a single
screen so the information can be monitored
at a glance.”
- Stephen Few
4. Successful Transition from secondary to higher
Education using Learning Analytics
enhance a successful transition from
secondary to higher education by means of
learning analytics
design and build analytics dashboards,
dashboards that go beyond identifying at-risk
students, allowing actionable feedback for all
students on a large scale.
Achieving Benefits from Learning Analytics
research strategies and practices for using
learning analytics to support students during
their first year at university
developing the technological aspects of
learning analytics,
focuses on how learning analytics can be used
to support students.
4
www.stela-project.eu
@STELA_project
2015-1-UK01-KA203-013767
www.ableproject.eu
@ABLE_project_eu
562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD
5. STELA ♥ ABLE
5
actionable feedback
student-centered
program level
inclusive
first-year experience
institution-wide
Learning Analytics
actual implementation
6. [!] Feedback must be “actionable”.
6
Warning!
Male students have
10% less probability to
be successful.
You are male.
Warning!
Your online activity is
lagging behind.
action?
?
action?
?
9. [!] Start with the available data.
Lots of data may eventually become
available in the future …
…. already start with what is available
9
(*)
(*) Zarraonandia, T., Aedo, I., Díaz, P., & Montero, A. (2013). An augmented lecture feedback system to support learner and teacher communication.
British Journal of Educational Technology, 44(4), 616-628.
11. Study advisor – student conversations
11
Should I consider
another program?
Can I still finish the
bachelor in 3 years?
How should I compose
my program for next
year?
What is the personal
situation?
How can I help?
What is the best
next step?
12. [!] Use all available expertise.
12
visualization experts
practitioners / end-users
researchers LA
researchers first-year
study success
Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue.
In IEEE Transactions on Learning Technology (http://ieeexplore.ieee.org/document/7959628/).
14. [!] Wording matters.
14
73% chance of success
73% of students of earlier
cohorts with the same
study efficiency obtained
the bachelor degree
http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
15. LISSA dashboard
15
Three examination periods
observations, interviews,
questionnaires
pilot with two engineering programs
Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology
16. LISSA: evaluation – observations
16
15 observations
insights
(-) factual
(+) interpretative
(!) reflective
Charleer S., Vande Moere A., Klerkx J., Verbert K., De Laet T. (2017). Learning Analytics Dashboards to Support Adviser-Student Dialogue. In IEEE Transactions on Learning Technology
17. Evaluation – interviews
“When students see the numbers, they are
surprised, but now they believe me.
Before, I used my gut feeling, now I feel
more certain of what I say as well”.
“It’s like a main thread
guiding the
conversation.”
“I can talk about what to do with the results,
instead of each time looking for the data and
puzzling it together.”
“Students don’t know where to look during the
conversation, and avoid eye contact.
The dashboard provides them a point of focus”.
“A student changed her
study method in June and
could now see it paid off.”
LISSA supports a personal dialogue.
the level of usage depends on the experience
and style of the study advisors
fact-based evidence at the side
narrative thread
key moments and student path help to
reconstruct personal track
“I can focus on the
student’s personal
path, rather than on
the facts.”
“Now, I can blame
the dashboard and
focus on
collaboratively looking
for the next step to
take.”
17
18. LISSA: status
18
26 programs >4500 students
114 student advisors
training of study advisors
http://blog.associatie.kuleuven.be/tinnedelaet/lissa-learning-dashboard-supporting-student-advisers-in-traditional-higher-education/
Millecamp M., Gutiérrez F., Charleer S., Verbert K., De Laet T.# (2018). A qualitative evaluation of a learning dashboard to support advisor-student
dialogues. Proceedings of the 8th International Learning Analytics & Knowledge Conference. LAK. Sydney, 5-9 March 2018 (pp. 1-5) ACM.
dashboards for three examination
periods
19. LISSA: evaluation – student
questionnaires
19
26 programs @KU Leuven
291 student questionnaires
first examination period
“Confronting, but
useful”
“I want to use this
dashboard at home.”
“Also show the sub-grades
for labs, … ”
“How can I know the data is
trustworth?”
“Can’t these visualizations be
send to students?” “Crisp and clear.”
20. 20
0
0
1
1
1
1
4
2
1
4
4
3
29
21
36
37
49
42
176
112
156
132
141
169
80
155
93
116
92
72
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1. The dashboard is clarifying and surveyable.
2. The shown information regarding my study
situation is correct.
3. The shown position with respect to my fellow
students (histograms per exam and global…
4. A conversation with my student advisors helped
me to gain insight in my study trajectory.
5. The visualisation is of added value to the
conversation with the student advisor.
6. The shown information provide me insight in
my current situation.
Student questionnaire January 2018 (N=291)
Strongly Disagree Disagree Neither Agree or Disagree Agree Strongly Agree
21. [!] Do not oversimplify. Show
uncertainty.
21
• reality is complex
• measurement is limited
• individual circumstances
• need for nuance
• trigger reflection
http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
22. [!] Be careful with predictive
algorithms.
22
http://blog.associatie.kuleuven.be/tinnedelaet/the-nonsense-of-chances-of-success-and-predictive-models/
• reality is complex
• measurement is limited
• individual circumstances
• need for nuance
• trigger reflection
24. [!] Start with the available data.
24
data already available?
administrative (examples)
student records course grades
systems (examples)
LMS access logs advisor meetings
)
Broos T., Verbert K., Van Soom C., Langie G., De Laet T.# (2018). Small data as a conversation starter for learning analytics: exam results dashboard for first-year students in higher
education. Journal of Research in Innovative Teaching & Learning, , 1-14.
25. [!] Think beyond the obvious data.
25
• Don’t think too traditional.
• Many institutions are collecting survey
data for educational research.
26. [!] Not all data is usable.
26
example data from a traditional course with “VLE as a file system”
test scores
activity/week (#days)
weeks of the year
27. [!] Not all data is usable.
27
example data from a course with flipped classroom & blended learning
exam scores
activity (# of modules used)
Not a single student
using less than 10
modules passed the
course.
Most of the successful
students used 15
modules or more.
28. [!] Keep Learning Analytics in
mind when designing learning
activities.
28
Learning
Analytics
Learning Design
INFORM
ENABLE
If LA indeed contributes to improved
learning design…
… don’t make it an afterthought
30. data already available?
administrative (examples)
student records course grades
[!] Think beyond the obvious data.
30
systems (examples)
LMS access logs advisor meetings
surveys (examples)
quality insurance LASSI
31. ~ 30 LASSI questions
(shortened version)
“Learning Skills”
Example: When preparing for an
exam, I create questions that I
think might be included.
Example: I find it difficult to
maintain my concentration
while doing my coursework.
Example: I find it hard to stick
to a study schedule.
raw scores
(selected 5 out of 10)
CONCENTRATION
MOTIVATION
FAILURE ANXIETY
TEST STRATEGY
TIME MANAGEMENT
norm scores
(in Flemish HE context)
Example: STRONG
Example: AVERAGE
Example: LOW
Example: VERY STRONG
Example: VERY WEAK
31
Meta cognitive abilities
Pinxten, M., Van Soom, C., Peeters, C., De Laet, T., Langie, G., At-risk at the gate: prediction of study success of first-year science and engineering students in an
open-admission university in Flanders—any incremental validity of study strategies? Eur J Psychol Educ (2017).
readySTEMgo Erasmus+ project https://iiw.kuleuven.be/english/readystemgo
32. Dashboard learning skills
32
students complete LASSI
questionnaire
students received personalized email
with invitation for dashboard
4367 students in 26 programs
in 9 faculties @KU Leuven
demo:
https://learninganalytics.set.kuleuven.be/lassi-1718/ (KU Leuven login)
2 programs @TU Delft
33. Feedback model
1. What is this about?
2. How am I doing?
3. How does this relates to
others?
4. Why is this relevant?
5. What can I do about it?
33
34. 34
3. How does this relates to
others?
2. How am I doing?
1. What is this about?
@studyProgram@
@yourScore@
35. 4. Why is this relevant?
5. What can I do about it?
35
39. Students that click through
Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.
In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.
39
better learning skills
40. More intense users
Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017, July). Dashboard for Actionable Feedback on Learning Skills: Scalability and Usefulness.
In International Conference on Learning and Collaboration Technologies (pp. 229-241). Springer, Cham.
40
worse learning skills
41. [!] Give students “the key”.
41
• Student has the key to own
data.
• Student takes initiative to
share/discuss own data.
• GDPR as opportunity!
43. [!] Acceptance precedes impact.
43
• Involve stakeholders from the start and
value their input!
COmmunication
COoperation
• Demonstrate usefulness.
• Take care of ethics and privacy.
• Best scenario:
students & study advisors as ambassadors
COCO
44. Impact?
survey before intervention
2nd year students 2016-2017
experiences first-year feedback
41 vragen, 5-point Likert scale
pen & paper
dashboards
LISSA
LASSI (learning skills)
3 x REX (grades)
Survey after intervention
2nd year students 2017-2018
45. Impact?
During the first year I received sufficient information regarding my academic achievements.
45
Engineering Science (p<0.001)
46. Impact?
The information I received helped to position myself with respect to my peers.
46
Engineering Science (p<0.001)
48. [!] Context matters!
• available data
• national and institutional regulations
and culture
• educational vision
• educational system, size of population ..
• …
Don’t just copy existing LA solutions!
48
49. Summary
case studies 11 findings/recommendations
[!] Use all available expertise.
[!] Start with the available data.
[!] Look beyond the obvious data.
[!] Not all data is usable.
[!] Wording matters.
[!] Don’t oversimplify. Show uncertainty.
[!] Beware of predictive algorithms.
[!] Keep Learning Analytics in mind when designing
learning activities.
[!] Give students “the key” to their data.
[!] Acceptance precedes impact.
[!] Context matters!
humble approach
small data
involvement of stakeholders, especially practitioners
actionable feedback
scalability
traditional university settings
Is this Learning Analytics?
51. Project team @
51
Sven Charleer
AugmentHCI, Computer Science department
PhD researcher ABLE
Katrien Verbert
AugmentHCI, Computer Science department
Copromotor of STELA & ABLE
Carolien Van Soom
Leuven Engineering and Science Education Center
Head of Tutorial Services of Science
Copromotor of STELA & ABLE
Greet Langie
Leuven Engineering and Science Education Center
Vicedean (education) faculty of Engineering Technology
Copromotor of STELA & ABLE
Tinne De Laet
Leuven Engineering and Science Education Center
Head of Tutorial Services of Engineering Science
Coordinator of STELA
KU Leuven coordinator of ABLE
Francisco Gutiérrez
AugmentHCI, Computer Science department
PhD researcher ABLE
Tom Broos
Leuven Engineering and Science Education Center
AugmentHCI, Computer Science department
PhD researcher STELA
Martijn Millecamp
AugmentHCI, Computer Science department
PhD researcher ABLE
Special thanks to study advisors for their cooperation, advice, feedback, and support!
Jasper, Bart, Riet, Hilde, An, Katrien, …
♥
