Feedback is problematic, and in higher education the situation is degrading quickly due to pressures to cater to massive cohorts while maintaining effectiveness. Technology can help achieve scalable feedback, and we show a few examples illustrating the process.
Separation of Lanthanides/ Lanthanides and Actinides
Using technology to provide feedback on learning strategies
1. EksoBionicsflickr.com
Using technology
to provide
feedback on
learning strategies
Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
Digital Learning Week
University of South Australia
16 March 2016
2. Abelardo Pardo Using technology to provide feedback on learning strategies 2
MichaelCoghlanflickr.com
About
Feedback Feedback
Higher Ed
Tech to scale
feedback Examples
3. Abelardo Pardo Using technology to provide feedback on learning strategies 3
MichaelCoghlanflickr.com
About
Feedback
4. Abelardo Pardo Using technology to provide feedback on learning strategies
"information provided by an agent (teacher, peer,
book, parent, self, experience) regarding aspects
of one’s performance or understanding”
4
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. doi:
10.3102/003465430298487
FarukAteşflickr.com
5. Abelardo Pardo Using technology to provide feedback on learning strategies
Innovations designed to strengthen the frequent
feedback that students receive about their
learning yield substantial learning gains
5
Black, P., & Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1),
7-74. doi:10.1080/0969595980050102
GarethChristopherflickr.com
6. Abelardo Pardo Using technology to provide feedback on learning strategies
If you could choose one…
• More than 500 meta-analyses
of student achievements
• 100 factors with potential
influence
• Feedback in top five
• (74 meta-analyses) Most
effective form: video, audio,
computer-assisted
instructional feedback, and/or
related goals
6
Hattie, J. A. (1999). Influences on student learning. Inaugural professorial address, University of Auckland, New Zealand
7. Abelardo Pardo Using technology to provide feedback on learning strategies 7
Feedback is inherent in and a
prime determiner of processes that
constitute self regulated learning
Galflickr.com
Butler, D. L., & Winne, P. H. (1995). Feedback and Self-Regulated Learning: A Theoretical Synthesis. Review of Educational
Research, 65(3), 245-281
8. Abelardo Pardo Using technology to provide feedback on learning strategies
Feedback models
1. Task Level (understanding,
performance)
2. Process Level (what to do to
understand, perform)
3. Self-regulation level (detecting
and directing effort)
4. Self level (personal evaluation
and affect)
8
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. doi:
10.3102/003465430298487
Orkomedixflickr.com
9. Abelardo Pardo Using technology to provide feedback on learning strategies
What is the recipe?
1. encourage contact between student
and faculty
2. develops reciprocity and cooperation
among students
3. uses active learning techniques
4. gives prompt feedback
5. emphasizes time on task
6. communicates high expectations
7. respects diverse talents and ways of
learning
9
Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. American Association
for Higher Education Bulletin, 39, 3-7.
Chickering, A. W., & Ehrmann, S. C. (1996). Implementing the seven principles: Technology as lever. American Association for
Higher Education Bulletin, 49, 3-6.
10. Abelardo Pardo Using technology to provide feedback on learning strategies 10
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in
Higher Education: Learning for the Longer Term. London and New York: Routledge.
Perceived as an administrative chore
instead of a pedagogical necessity
MarcinWicharyflickr.com
11. Abelardo Pardo Using technology to provide feedback on learning strategies 11
Hounsell, D. (2008). The Trouble with Feedback. New Challenges, Emerging Strategies. Interchange(2).
Feedback has a Cinderella status
Bitslammerflickr.com
12. Abelardo Pardo Using technology to provide feedback on learning strategies
NicolasNovaflickr.com
Feedback is inherently problematic
12
Higgins, R., Hartley, P., & Skelton, A. (2010). Getting the Message Across: The problem of communicating assessment feedback.
Teaching in Higher Education, 6(2), 269-274. doi:10.1080/13562510120045230
Feedback as communication
13. Abelardo Pardo Using technology to provide feedback on learning strategies
1) Construct an understanding
2) Use it to regulate
13
Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good
feedback practice. Studies in Higher Education, 31(2), 199-218. doi:10.1080/03075070600572090
JamesButlerflickr.com
14. Abelardo Pardo Using technology to provide feedback on learning strategies 14
MichaelCoghlanflickr.com
About
Feedback Feedback
Higher Ed
15. Abelardo Pardo Using technology to provide feedback on learning strategies 15
JoelSageflickr.com
“sustainability of feedback is under threat… pervasive
student concerns about the provision of feedback in
an era of mass education"
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher
Education: Learning for the Longer Term. London and New York: Routledge.
16. Abelardo Pardo Using technology to provide feedback on learning strategies 16
AllenLaiflickr.com
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher
Education: Learning for the Longer Term. London and New York: Routledge.
Downward spiral: Student disenchantment mounts
when feedback is uninformative and
as a consequence, less feedback is provided
Solution: provide high-value feedback, rethink the role of
student, enhance relation between guidance and feedback
17. Abelardo Pardo Using technology to provide feedback on learning strategies
Feedback is delicate. It must be positive. High
fragility of confidence felt by many students
17
Ferguson, P. (2011). Student perceptions of quality feedback in teacher education. Assessment & Evaluation in Higher Education,
36(1), 51-62. doi:10.1080/02602930903197883
Doranflickr.com
18. Abelardo Pardo Using technology to provide feedback on learning strategies
The feedback question gets systematically
lower values in student surveys
18
Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005). The First Year Experience in Australian Universities: Findings from a
decade of National Studies. University of Melbourne: Centre for the Study of Higher Education.
Eleafflickr.com
19. Abelardo Pardo Using technology to provide feedback on learning strategies
• Ambiguity or opacity (What do you
mean?)
• Negativity
19
Huxham, M. (2007). Fast and effective feedback: are model answers the answer? Assessment & Evaluation in Higher Education,
32(6), 601-611. doi:10.1080/02602930601116946
• Lateness
• Uncertainty about criteria and
contexts
Loonyhikerflickr.com
20. Abelardo Pardo Using technology to provide feedback on learning strategies 20
PostmodernSleazeflickr.com
Boud, D., & Molloy, E. (2013). Rethinking models of feedback for learning: the challenge of design. Assessment & Evaluation in
Higher Education, 38(6), 698-712. doi:10.1080/02602938.2012.691462
Conventional feedback: Mark 1
Impractical due to resources
Reserved for important concepts only
Could be extended by technology if
careful selection is applied
Mark 2
From mechanistic to responsive
Curriculum-wide view
Foster self-regulation, feedback
seeking skills, calibrated
judgement, etc.
21. Abelardo Pardo Using technology to provide feedback on learning strategies 21
Active Learning Works
22. Abelardo Pardo Using technology to provide feedback on learning strategies 22
2 Hours
2 Hours
3 Hours
Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema,
Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer.
23. Abelardo Pardo Using technology to provide feedback on learning strategies 23
Students are less likely to
engage in pre-class activities if
they are not interactive, do not
provide formative feedback,
and not coherently linked with
the face-to-face activities
O'Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. The Internet and Higher
Education, 25, 85-95. doi:10.1016/j.iheduc.2015.02.002
DanKlimkeflickr.com
24. Abelardo Pardo Using technology to provide feedback on learning strategies 24
MichaelCoghlanflickr.com
About
Feedback Feedback
Higher Ed
Tech to scale
feedback
25. Abelardo Pardo Using technology to provide feedback on learning strategies
Instructive or process feedback is not common in
learning analytics
25
Gasevic, D., Dawson, S., & Siemens, G. (2015). Let's not forget: Learning analytics are about learning. TechTrends, 59(1), 64-75.
ChefCookeflickr.com
26. Abelardo Pardo Using technology to provide feedback on learning strategies 26
HungarianSnowflickr.com
Intelligence Amplification
Hierarchies of regulators to achieve amplification
We measure intelligence as the capacity to make
the appropriate selection
Ashby, W. R. (1957). An Introduction to Cybernetics. London: Chapman & Hall Ltd.
27. Abelardo Pardo Using technology to provide feedback on learning strategies 27
Baker, R. S. (2016). Stupid Tutoring Systems, Intelligent Humans. International Journal of Artificial Intelligence in Education. doi:
10.1007/s40593-016-0105-0
Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: A
Review of Evidence-Based Learning Analytics Interventions at the Open University UK. Journal of Interactive Media in
Education, 2016(1). doi:10.5334/jime.394
Change in focus, from artificial intelligence to
intelligence amplification
Tutors are informed about students struggling but it is up to
them to decide if/when to act
28. Abelardo Pardo Using technology to provide feedback on learning strategies 28
BrandonMartin-Andersonflickr.com
• Can data guide the provision of effective feedback
• Can this provision be done at scale?
29. Abelardo Pardo Using technology to provide feedback on learning strategies 29
JoshRepennekerFlickr.com
Can technology and
data be used to
change the economic
model of feedback,
achieve scale and
maintain effectiveness?
30. Abelardo Pardo Using technology to provide feedback on learning strategies
Simplistic View Emerging
“We are really shifting
the role of an educator
to someone who is
more of a data-
enabled detective”
Max Ventilla
AltSchool founder
(emphasis added)
30
Mead, Rebecca (2016). Learn Different. Silicon Valley disrupts education. The New Yorker, Annals of Technology, March 7 Issue
31. Abelardo Pardo Using technology to provide feedback on learning strategies 31
MichaelCoghlanflickr.com
About
Feedback Feedback
Higher Ed
Tech to scale
feedback Examples
32. Abelardo Pardo Using technology to provide feedback on learning strategies 32
• First year engineering
course
• Computer Systems
• Flipped Blended
Learning
• ~300 students
• Preparation activities
with formative feedback
and
summative assessment
• 42% satisfied with
feedback (meh)
33. Abelardo Pardo Using technology to provide feedback on learning strategies 33
2 Hours
2 Hours
3 Hours
Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema,
Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer.
2014 Edition
35. Abelardo Pardo Using technology to provide feedback on learning strategies 35
1. realtime feedback
2. convey engagement
3. compare with rest of
cohort
4. reset weekly
5. one click away from notes
Approach 1 (2014)
SeanDreilingerflickr.com
36. Abelardo Pardo Using technology to provide feedback on learning strategies 36
Tracking
37. Abelardo Pardo Using technology to provide feedback on learning strategies 37
Technology
Technology
Week N
38. Abelardo Pardo Using technology to provide feedback on learning strategies 38
Weekly real time feedback
39. Abelardo Pardo Using technology to provide feedback on learning strategies 39
No statistically significant difference in the rating of feedback (2013 edition,
M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17
40. Abelardo Pardo Using technology to provide feedback on learning strategies 40
1. encourage contact
between student and
faculty (forum)
2. uses active learning
techniques
3. gives prompt feedback
4. emphasizes time on task
5. kind, specific, helpful
Approach 2 (2015)Ben Manson flickr.com
BenMansonflickr.com
41. Abelardo Pardo Using technology to provide feedback on learning strategies 41
Week N
Technology
Human
42. Abelardo Pardo Using technology to provide feedback on learning strategies 42
Pardo, A., Mirriahi, N., Martínez-Maldonado, R., Jovanović, J., Dawson, S., & Gasevic, D. (2016). Generating Actionable Predictive
Models of Academic Performance. Paper presented at the International Conference on Learning Analytics and Knowledge,
Edinburgh.
1. Group students
2. Each group has an
estimated midterm score
3. Groups are determined by
conditions in engagement
measures
4. Can I act on this information?
Student Clustering based on Engagement
43. Abelardo Pardo Using technology to provide feedback on learning strategies 43
You should take a more
careful look at how symbols
are encoded in the video.
Would you be able to encode/
decode UAL symbols without
looking at the video?
Good initial work. However,
did you understand the trick
to handle encoding with a
variable number of bits?
Would you be able to provide
an example?
Good work. Would you be
able to come up with your
own machine language and
your encoding scheme?
Remember that it has to be
unambiguous.
Thorough work with the task
about machine language
encoding. Give it a quick
review before the midterm.
Q1 Q2 Q3 Q4
Instructor
44. Abelardo Pardo Using technology to provide feedback on learning strategies 44
Algorithm
45. Abelardo Pardo Using technology to provide feedback on learning strategies 45
Automatic
Email
46. Abelardo Pardo Using technology to provide feedback on learning strategies 46
1. Data collected weekly
2. Email sent at end of week
3. 4 weeks before the midterm
nateOneflickr.com
47. Abelardo Pardo Using technology to provide feedback on learning strategies 47
Significant difference in the rating of feedback (2014 edition, M=3.35, SD=1.03;
2015 edition, M=3.82, SD=0.90); t(389.78) = -4.88, p <10-6
Effect size (Cohen’s d) = 0.49. Medium positive effect
48. Abelardo Pardo Using technology to provide feedback on learning strategies 48
Significant difference in the midterm score (2014 edition, M=12.80, SD=4.79;
2015 edition, M=13.83, SD=4.89); t(684.5) = -2.86, p < 0.002
Effect size (Cohen’s d) = 0.21. Small positive effect
49. Abelardo Pardo Using technology to provide feedback on learning strategies 49
MatthewandHeaderflickr.com
Feedback: high impact/risk
Higher Ed is in dire need for feedback revamp
Hybrid methods: human intelligence amplification
Data supported approaches allow effective scaling
50. EksoBionicsflickr.com
Using technology
to provide
feedback on
learning strategies
Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
Digital Learning Week
University of South Australia
16 March 2016