Learning analytics is a discipline that uses data captured by technology during a learning experience to increase our level of understanding, increase its quality, and improve the environment in which it occurs. But these experiences need to be designed first. In this talk we start from the statement that there is no such thing as a neutral design. In the era of increasing technology mediation Learning experiences need to be designed considering the capacity to capture data, the possibility of making sense and derive knowledge from the data, and the need to act on that knowledge. In this talk we will explore some initiatives to make these connections explicit in a learning design. Using a flipped learning experience, we will explore how to embed data and data analysis as part of the design tasks.

1. Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
Articulating the connection between
Learning Analytics and Learning
Design
Faculty of Education
University of Hong Kong
16/May/2017
MarianoMantelflickr.com

2. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 2
Ivanflickr.com
Learning
Analytics
Data
Journey
Connection
Model
How to
integrate

3. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 3
Ivanflickr.com
Learning
Analytics

4. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 4
DmitryGrigorievflickr.com
Learning Analytics: measure, collect, analyse data
about learners to understand and improve their
learning and the environment in which it occurs
1st International Conference on Learning Analytics and Knowledge [Online]. https://tekri.athabascau.ca/analytics. [Accessed 2017].

5. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 5
EustaquioSantimanoflickr.com
Collect
Report
Analyse
Act
Refine
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics. Louisville, CO: Educause White Paper.

6. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 6
Long, P., & Siemens, G. (2011). Penetrating the Fog: Analytics in Learning and Education. EDUCAUSE Review, 48(5), 31-40.

7. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 7
Learning Analytics Model (v1)

8. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 8
Ivanflickr.com
Learning
Analytics
Data
Journey

9. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 9
trepeluflickr.com
Let’s explore clickstreams!

10. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 10
4.174.239.145 - user_2@bogus.com "user 2" [27/Jan/2016:00:00:26 +0000]
"GET resource_6 HTTP/1.1" 302 1428 "-" "Mozilla/5.0 (Macintosh; Intel Mac
OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
Clickstreams
Machine
address
User
id
User
name
When
date/time
4.174.239.145 - user_2@bogus.com "user 2" [27/Jan/2016:00:00:26 +0000]
"GET resource_6 HTTP/1.1" 302 1428 "-" "Mozilla/5.0 (Macintosh; Intel Mac
OS X 10.10; rv:39.0) Gecko/20100101 Firefox/39.0"
Which
resource One line written in a file for every click!

11. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 11
Clickstream - Video
4187,
2016-07-14 00:56:46.341946+00,
{“time”":0,"id":"xEJtdMQMcrs","event":"PLAY"},
https://mycourse.com/Material/HLP/HLP_notes.html,
embedded-video
User
id
When
date/time
Video played
from start
PageEvent
type

12. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 12
Clickstream - Question
7141,
2016-07-25 00:41:42.282416+00,
{“question_id":"COD-baseencoding-eqt_1","answer":0},
https://mycourse.com/Material/COD/Activities/encoding_videoeqt.html,
embedded-question
User
id
When
date/time
Question
answered
incorrectly
PageEvent
type

13. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 13
Large number of events per user
2016-08-09 00:05:43.124199+00,989,129.78.56.144,"{""outcome"": ""incorrect"", ""assessment"":
""summative"", ""score"": 44.4444444444444, ""exercise"": ""/data2u/static/exco_exc/DRM/
problem_13_c.html"", ""sequence"": ""DRM-exco-C""}",abelardopardo,196619,exco-answer
2016-08-09 00:05:44.140307+00,989,129.78.56.144,"{""score"": 44.4444444444444, ""exercise"": ""/data2u/
static/exco_exc/DRM/problem_03_c.html"", ""sequence"": ""DRM-exco-C""}",abelardopardo,196620,exco-view
2016-08-09 00:05:47.76122+00,968,49.182.128.186,"{""question_id"":""COD-numberofwires-eqt_1"",""answer"":
1}",abelardopardo,196621,embedded-question
2016-08-09 00:05:48.861036+00,806,129.78.56.151,"{""question_id"":""COD-encodeintegers-section-
eqt_4"",""answer"":0}",abelardopardo,196622,embedded-question
2016-08-09 00:05:48.959791+00,989,129.78.56.144,"{""outcome"": ""correct"", ""assessment"":
""summative"", ""score"": 50.0, ""exercise"": ""/data2u/static/exco_exc/DRM/problem_03_c.html"",
""sequence"": ""DRM-exco-C""}",abelardopardo,196623,exco-answer
2016-08-09 00:05:49.539162+00,989,129.78.56.144,"{""score"": 50.0, ""exercise"": ""/data2u/static/
exco_exc/DRM/problem_14_c.html"", ""sequence"": ""DRM-exco-C""}",abelardopardo,196624,exco-view
2016-08-09 00:05:49.571138+00,1311,129.78.56.159,"{""url"":""https://flip.ee.usyd.edu.au/elec1601/
Material/COD/index.html""}",abelardopardo,196625,resource-view
2016-08-09 00:05:50.050035+00,1069,129.78.56.199,"{""question_id"":""COD-integerencoding-videoeqt-
eqt_3"",""answer"":0}",abelardopardo,196626,embedded-question
2016-08-09 00:05:51.709295+00,806,129.78.56.151,"{""question_id"":""COD-encodeintegers-section-
eqt_4"",""answer"":0}",abelardopardo,196627,embedded-question
2016-08-09 00:05:51.962474+00,1069,129.78.56.199,"{""question_id"":""COD-integerencoding-videoeqt-
eqt_3"",""answer"":0}",abelardopardo,196628,embedded-question
2016-08-09 00:05:52.0819+00,806,129.78.56.151,"{""question_id"":""COD-encodeintegers-section-
eqt_4"",""answer"":""-1""}",abelardopardo,196629,embedded-question
2016-08-09 00:05:53.025356+00,1069,129.78.56.199,"{""question_id"":""COD-integerencoding-videoeqt-
eqt_3"",""answer"":1}",abelardopardo,196630,embedded-question
2016-08-09 00:05:54.756229+00,1311,129.78.56.159,"{""url"":""https://flip.ee.usyd.edu.au/elec1601/
Material/COD/COD_notes.html#range-accuracy-and-precision-of-the-floating-point-
representation""}",abelardopardo,196631,resource-view
2016-08-09 00:05:54.856333+00,989,129.78.56.144,"{""outcome"": ""incorrect"", ""assessment"":
""summative"", ""score"": 50.0, ""exercise"": “"/data2u/static/exco_exc/DRM/problem_03_c.html"",
""sequence"": ""DRM-exco-C""}",abelardopardo,196623,exco-answer

14. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 14
Logs without design context
• How many videos were played?
• How many questions answered correctly/incorrectly?
• How many pages were viewed?
• How many posts in the forum?
• Are posts and video plays together?*
• What type of resources are being used?*
• Correlation? Linear model? Student at risk?
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept.
Computers & Education, 54(2), 588-599. doi:10.1016/j.compedu.2009.09.008

15. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 15
4187,
2016-07-14 00:56:46.341946+00,
{“time”":0,"id":"xEJtdMQMcrs","event":"PLAY"},
https://mycourse.com/Material/HLP/HLP_notes.html,
embedded-video
4187, 2016-07-14 00:56:46.341946+00, VIDEO, xEJtdMQMcrs, PLAY, COD, W2
Combine logs with design
Student played a video about topic COD during Week 2

16. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 16
Logs with design context
• Is a student preparing the lecture?
• Are study sessions related to schedule?
• Reviewing or catching up?
• Is there distributed practice?
• Are there changes in strategies?
ChrisLewisflickr.com

17. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 17
Detect use of resources in sessions
Kahn, I., & Pardo, A. (2016). Data2U: Scalable Real time Student Feedback in Active Learning Environments. Paper presented at the
International Conference on Learning Analytics and Knowledge, Edinburgh, UK. doi:10.1145/2883851.2883911

18. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 18
Jovanovic, J., Gašević, D., Pardo, A., Dawson, S., & Mirriahi, N. (In Press. Accepted 20/Feb/2017). Learning Analytics to Unveil Learning
Strategies in a Flipped Classroom. Internet and Higher Education. doi:10.1016/j.iheduc.2017.02.001
Analyse Learning Strategies

19. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 19
Decision Trees
• Week 10
• Predicted score at
leaves (out of 40)
• Conditions at nodes
• If (EXC.in >=22) and
(VID.PL < 8.5) then
score = 6
Pardo, A., Mirriahi, N., Martínez-Maldonado, R., Jovanović, J., Dawson, S., & Gasevic, D. (2016). Generating Actionable Predictive Models
of Academic Performance. Paper presented at the International Conference on Learning Analytics and Knowledge, Edinburgh, UK. doi:
10.1145/2883851.2883870

20. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 20
Ivanflickr.com
Learning
Analytics
Data
Journey
Connection
Model

21. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 21
Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design.
American Behavioral Scientist, 57(10), 1439-1459. doi:10.1177/0002764213479367
Identifying and coding
learning and teaching contexts
is essential!
Checkpoints Process Analytics
JudithKleinflickr.com

22. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 22
Bakharia, A., Corrin, L., de Barba, P., Kennedy, G., Gašević, D., Mulder, R., . . . Lockyer, L. (2016). A Conceptual Framework linking
Learning Design with Learning Analytics. Paper presented at the International Conference on Learning Analytics and Knowledge,
Edinburgh, UK. doi:10.1145/2883851.2883944
The3Bsflickr.com
Comparative
Analytics
Cohort
Dynamics
Tool
Specific
Intervention
tools
Temporal
Analytics

23. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 23
RobertBrookflickr.com
Use actions to promote an objective
to drive the connection between
design elements and analytics affordances
Student Engagement

24. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 24
Bartimote-Aufflick, K., Reimann P., Pardo, A. The perspective theory brings to learning analytics in the classroom: A realist approach
Manuscript in preparation.
Learning Analytics Model v2

25. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 25
Questions guiding the design
• What can/will be observed when the design is
instantiated?
• How can the data be interpreted in the design
context?
• What actions can be derived from the analysis?
• During the enactment
Instructors
Students
• After the enactment (design refinement)

26. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 26
4 layer connection
1. Click (low) level events
2. Process to engage with a set of tasks
3. Use effective learning strategies
4. Overall effectiveness of the design
MichaelCoughlanflickr.com

27. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 27
Click/Low
level
Learning
process with
set of tasks
Used
learning
strategies
Overall
effectiveness
What can be
observed?
How to interpret data
Immediate
actions
Students
Instructor
Future actions
LA lense for LD

28. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 28
Example
• Blended course
• Large student cohort
• Face-to-face sessions require preparation
• Activities include: video, questions, readings,
exercises (summative)
• Pattern repeated weekly
• Interaction through discussion forum
• Three submissions throughout course
• Midterm examination

29. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 29
2 Hours
2 Hours
3 Hours
Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema,
Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer.

30. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 30
Click/Low
level
Learning
process with
set of tasks
Used
learning
strategies
Overall
effectiveness
What can be
observed?
Events in videos,
questions,
exercises
How to interpret data
If video viewed and
questions correct,
incorrect
Immediate
actions
Students
Suggestions to
engage with
resources
Instructor
Provide further
information about
topics with high
failure rate
Future actions
Learning process

31. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 31
Click/Low
level
Learning
process with
set of tasks
Used
learning
strategies
Overall
effectiveness
What can be
observed?
Topics in each
study session
How to interpret data
Obtain distribution,
depth and
effectiveness of
session.
Immediate
actions
Students Suggest plan
Instructor
Flag most popular
strategies
Future actions
Where to include
additional strategic
support
Learning strategies

32. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 32
Ivanflickr.com
Learning
Analytics
Data
Journey
Connection
Model
How to
integrate

33. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 33
Increasingly complex design space
ThomasLeth-Olsenflickr.com
Cognitive
Social
Material
(Tangible)
Material
(Intangible)

34. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 34
“… teaching in higher education will
necessarily shift the balance of its efforts
towards a greater investment in design as
a way of coping with otherwise intolerable
pressures on staff and resources.”
Goodyear, P. (2015). Teaching as Design. HERDSA Review of Higher Education, 2, 27-50.
TobiasToftflickr.com

35. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics
Design Patterns
35
Anneflickr.com
general repeatable solution to a commonly
occurring problem
• Creational
• Structural
• Behavioural

36. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics
Assume/Promote recording
36
NickLeeflickr.com

37. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics
Use focal points for LA
37
EricWustenhagenflickr.com
• Actions
• Expected reactions
• Rubrics

38. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 38
ontasklearning.org

39. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 39
• Support instructors to
create personalised
feedback
• Simple rule-base
knowledge encoding
• Provide appropriate
view of data sources
• Scale to large and highly
diverse cohorts
• Will be released as
open-source project
Q3/4 2017
• First pilots in Q1/2 2017
• Tutorial in LAK 2017
• Contact us if interested
ontasklearning.org

40. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 40
NASAJohnsonflickr.com
• LA needs LD
• Design may drive log analysis
to meaningful questions
• LA cues can be included in
LD
• Use of patterns and focal
points
Conclusions

41. Abelardo Pardo Articulating the Connection Between Learning Design and Learning Analytics 41

42. Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
Articulating the connection between
Learning Analytics and Learning
Design
Faculty of Education
University of Hong Kong
16/May/2017
MarianoMantelflickr.com