The increasing presence of technology mediation offers an unprecedented opportunity to use detailed data sets about the interactions that occur while a learning experience is being enacted. Areas such as Learning Analytics or Educational Data Mining have explored numerous algorithms and techniques to process these data sets. Additionally, technology now offers the opportunity to increase the immediacy of interventions. However, not much emphasis has been placed on how to extract truly actionable knowledge and how to bring it effectively as part of a learning experience. In this talk, we will use the concept of feedback as the focus to establish a specific connection between the knowledge derived from data-analysis procedures and the actions that can be immediately deployed in a learning environment. We will discuss how there is a trade-off between low-level automatic feedback and high-level complex feedback and how technology can provide efficient solutions for the case of large or highly diverse cohorts.

1. Provision of personalized feedback at
scale using learning analytics
Centre for the Enhancement of Teaching and Learning
Faculty of Education
Hong Kong University 19/May/2017
Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
Antoniettaflickr.com

2. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 2
MaureenBarlinflickr.com
New
Design
Mindset
The role
of data
and feedback
Personalised feedback
at scale

3. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 3
MaureenBarlinflickr.com
New
Design
Mindset

4. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 4
Simple information transfer is not working
Mazur, E. (2009). Farewell, lecture. Science, 323(5910), 50-51.
Krugazorflickr.com

5. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 5

6. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 6
Active Learning Works
Engage students in
the learning process
Bonwell, C. C., & Eison, J. A. (1991). Active Learning: Creating Excitement in the Classroom ASHEERIC Higher Education Report No. 1.
Washington, DC, USA: George Washington University.

7. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 7
“… robust correlations between student
involvement in a subset of ‘educationally
purposive activities’, and positive outcomes of
student success and development, including
satisfaction, persistence, academic achievement
and social engagement”
Trowler, V. (2010). Student engagement literature review. York, UK: The Higher Education Academy.
shuaGandersonflickr.com

8. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 8
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: beliefs, techniques, and illusions. Annu Rev Psychol, 64,
417-444. doi:10.1146/annurev-psych-113011-143823
ChristianWeidingerflickr.com
We, as learners may
• Not know how to promote
comprehension, retention, transfer.
• Not assess properly our own
learning
• Be biased when judging our
learning
• Rely too much on social beliefs

9. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 9
JansonHewsflickr.com
Frontier between physical and virtual spaces is blurring

10. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 10
Beware of technology creating the illusion of rational thinking
JenRflickr.com

11. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 11
“… teaching in higher education will
necessarily shift the balance of its efforts
towards a greater investment in design
as a way of coping with otherwise
intolerable pressures on staff and
resources.”
Goodyear, P. (2015). Teaching as Design. HERDSA Review of Higher Education, 2, 27-50.

12. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 12
"There is no such thing as a neutral design"
JeremyBrooksflickr.com
Thaler, R. H., & Sunstein, C. R. (2008). Nudge. Great Britain: Yale University Press.
“People make good choices in contexts in which
they have experience, good information, and
prompt feedback"

13. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 13
“38 meta-analyses investigating 105 correlates of achievement, based on
3,330 effect sizes from almost 2 million students”
Schneider, M., & Preckel, F. (2017). Variables Associated With Achievement in Higher Education: A Systematic Review of Meta-
Analyses. Psychological Bulletin. doi:10.1037/bul0000098
• The effectivity of courses is strongly related to what teachers do.
• The effectivity of teaching methods depends on how are implemented
• Teachers can improve the instructional quality of their courses by
making a number of small changes
- providing detailed task-focused and improvement-oriented
feedback
• The combination of teacher-cantered and student-cantered
instructional elements is more effective than either form of instruction
alone
Variables associated with achievement

14. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 14
MaureenBarlinflickr.com
New
Design
Mindset
The role
of data
and feedback

15. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 15
Students are less likely to engage in pre-class activities if they
are not interactive, do not provide formative feedback, and
not coherently linked with the face-to-face activities
O'Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. The Internet and Higher
Education, 25, 85-95. doi:10.1016/j.iheduc.2015.02.002
DanKlimkeflickr.com

16. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics
If you could choose one…
• Over 800 meta-analyses of student
achievements
• 100 factors with potential influence
• Feedback in top five
• (74 meta-analyses) Most effective
form: video, audio, computer-
assisted instructional feedback,
and/or related goals
16
Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses related to achievement. New York: Routledge.

17. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 17
Boud, D., & Molloy, E. (Eds.). (2013). Feedback in Higher and Professional Education: Understanding it and doing it well. London
and New York: Routledge.
FarukAteşflickr.com
“Feedback is a process whereby learners obtain information about
their work in order to appreciate the similarities and differences
between the appropriate standards for any given work, and the
qualities of the work itself, in order to generate improved work”

18. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 18
Black, P., & Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1),
7-74. doi:10.1080/0969595980050102
GarethChristopherflickr.com
Innovations designed to strengthen the frequent
feedback that students receive about their
learning yield substantial learning gains

19. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 19
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in
Higher Education: Learning for the Longer Term. London and New York: Routledge.
Perceived as an administrative chore
instead of a pedagogical necessity
MarcinWicharyflickr.com

20. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 20
IanRansleyflickr.com
How to scale sustainable practices?
Carless, D., Salter, D., Yang, M., & Lam, J. (2011). Developing sustainable feedback practices. Studies in Higher Education, 36(4),
395-407. doi:10.1080/03075071003642449
• Multi-stage assignments
• Dialogic feedback
• Technology supported
• Self-evaluation

21. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 21
Baker, R. S. (2016). Stupid Tutoring Systems, Intelligent Humans. International Journal of Artificial Intelligence in Education. doi:
10.1007/s40593-016-0105-0
From artificial intelligence to intelligence amplification
Instructors are informed about student engagement
but it is up to them to decide if/when/how to act
Tecnaliaflickr.com

22. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 22
DmitryGrigorievflickr.com
Learning Analytics: measure, collect, analyse data
about learners to understand and improve their
learning and the environment in which it occurs
1st International Conference on Learning Analytics and Knowledge [Online]. https://tekri.athabascau.ca/analytics. [Accessed 2017].

23. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 23
• Collect data about how students
engage in a learning experience
• Interpret the observations in the
context of the instructional design
• Translate knowledge into
personalised student support
actions

24. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 24
Bartimote-Aufflick, K., Reimann P., Pardo, A. The perspective theory brings to learning analytics in the classroom: A realist approach
Manuscript in preparation.
Learning Analytics Model

25. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 25
MaureenBarlinflickr.com
New
Design
Mindset
The role
of data
and feedback
Personalised feedback
at scale

26. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 26
Example of Highly Instrumentalized Learning Design

27. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 27
Weekly real time feedback

28. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 28
No statistically significant difference in the rating of feedback (2013 edition,
M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17

29. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 29
You should take a more
careful look at how symbols
are encoded in the video.
Would you be able to encode/
decode UAL symbols without
looking at the video?
Good initial work. However,
did you understand the trick
to handle encoding with a
variable number of bits?
Would you be able to provide
an example?
Good work. Would you be
able to come up with your
own machine language and
your encoding scheme?
Remember that it has to be
unambiguous.
Thorough work with the task
about machine language
encoding. Give it a quick
review before the midterm.
Q1 Q2 Q3 Q4
Instructor

30. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 30
Algorithm

31. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 31
Automatic
Email

32. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 32
Helpful feedback
Effect size (Cohen’s d) = 0.49.
Medium positive effect
Midterm Scores
Effect size (Cohen’s d) = 0.21.
Small positive effect
Pardo, A., Jovanovic, J., Dawson, S., Gašević, D. & Mirriahi, N. (2017). Using Learning Analytics to Scale the Provision of Personalised
Feedback. Manuscript in preparation

33. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 33
ontasklearning.org

34. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 34
• Support instructors to
create personalised
feedback
• Simple rule-base
knowledge encoding
• Provide appropriate
view of data sources
• Scale to large and highly
diverse cohorts
• Will be released as
open-source project
Q3/4 2017
• First pilots in Q1/2 2017
• Tutorial in LAK 2017
• Contact us if interested
ontasklearning.org

35. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 35
MTC0316flickr.com
Conclusions
• New L&T design mindset required
• Feedback is effective to promote
student engagement
• Learning analytics has the
potential to support students at
scale
• Use data to provide personalised
student support

36. Abelardo Pardo Provision of Personalised Feedback at Scale Using Learning Analytics 36

37. Provision of personalized feedback at
scale using learning analytics
Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
Antoniettaflickr.com
Centre for the Enhancement of Teaching and Learning
Faculty of Education
Hong Kong University 19/May/2017