Keynote Address, Expanding Horizons 2012, Macquarie University http://staff.mq.edu.au/teaching/workshops_programs/expanding_horizons "Learning Analytics": unprecedented data sets and live data streams about learners, with computational power to help make sense of it all, and new breeds of staff who can talk predictive models, pedagogy and ethics. This means rather different things to different people: unprecedented opportunity to study, benchmark and improve educational practice, at scales from countries and institutions, to departments, individual teachers and learners. "Benchmarking" may trigger dystopic visions of dumbed down proxies for 'real teaching and learning', but an emu response is no good. For educational institutions, our calling is to raise the quality of debate, shape external and internal policy, and engage with the companies and open communities developing the future infrastructure. How we deploy these new tools rests critically on assessment regimes, what can be logged and measured with integrity, and what we think it means to deliver education that equips citizens for a complex, uncertain world.

1. Keynote Address, Expanding Horizons 2012, Macquarie University
http://staff.mq.edu.au/teaching/workshops_programs/expanding_horizons
Our Learning Analytics
are Our Pedagogy
Simon Buckingham Shum @sbskmi
Knowledge Media Institute, The Open University UK
simon.buckinghamshum.net
1

2. learning objective:
walk out with
better questions
than you can ask right now
2

3. Urgent need: quality dialogue between
analytics stakeholders, to accelerate
invention innovation
www.SoLAResearch.org
Follow: @SoLAResearch
3

4. The global demand for learning
Implications for
assessment and
feedback at John Daniel
massive scale?
http://www.col.org/resources/speeches/2012presentations/Pages/2012-02-01.aspx 4

5. is education
poised to become a
data-driven
enterprise
and science ? 5

6. Possibly 90% of the digital data we have
today was generated in the last 2 years
Volume outstrips old infrastructure
Variety Internet of things, e-business transactions, environmental
sensors, social media, audio, video, mobile…
Velocity The speed of data access and analysis is exploding
A quantitative shift on this scale is in fact a qualitative shift, requiring
new ways of thinking about
societal phenomena
6

7. edX: “this is big data, giving us the chance
to ask big questions about learning”
Will the tomorrow’s
educational researcher be
as helpless without an
analytics infrastructure, as
a geneticist without
genome databases, or a
physicist without CERN? 7

8. Lifelogging: explosion of data capture
and sharing about personal activities
http://www.mirror-project.eu
http://quantifiedself.com/guide 8

9. Educational Data Mining research community

10. Learning Analytics research community

11. different levels
of analytic
11

12. ‘Learning Analytics’ and
‘Academic Analytics’
Long, P. and Siemens, G. (2011), Penetrating the fog: analytics in learning and education. Educause Review Online,
46, 5, pp.31-40. http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education 12

13. Macro/Meso/Micro Learning Analytics
Macro:
region/state/national/international

14. Macro/Meso/Micro Learning Analytics
Macro:
region/state/national/international
Meso:
institution-wide

15. Macro/Meso/Micro Learning Analytics
Macro:
region/state/national/international
Meso:
institution-wide
Micro:
individual user actions
(and hence cohort)
Will institutions be dazzled by the
dashboards, or know what
questions to ask at each level?

16. Macro/Meso/Micro Learning Analytics
Macro:
region/state/national/international

17. US states are getting the infrastructure
in place
dataqualitycampaign.org
17

18. Shared Learning Collaborative
http://slcedu.org
18

19. National league tables for English schools
19

20. Macro/Meso/Micro Learning Analytics
Meso:
institution-wide

21. Analytics-savvy Leaders are the future?
Parr-Rud, O. (2012). Drive Your Business with Predictive Analytics. SAS White Paper
http://www.sas.com/reg/gen/corp/1800392
21

22. Business Intelligence companies see an
education market opening up
These are pedagogically agnostic:
they seek to optimize operational
efficiency whatever the sector
These may make pedagogical
assumptions: how will learning
design and assessment regimes
shape the analytics they offer?
http://www.sas.com/industry/education/highered 22

23. Business Intelligence companies see an
education market opening up
…but do they know anything about
the roles that language plays in
learning and knowledge
construction? 23

24. BI+HigherEd communities of practice
24

25. Business Intelligence
≠
Learning Analytics

26. Macro/Meso/Micro Learning Analytics
Micro:
individual user actions
(and hence cohort)

27. Analytics in your VLE:
Blackboard: feedback to students
http://www.blackboard.com/Platforms/Analytics/Overview.aspx
27

28. Purdue University Signals: real time traffic-
lights for students based on predictive model
Premise: academic success is defined as a function of
aptitude (as measured by standardized test scores and
similar information) and effort (as measured by participation
within the online learning environment).
Using factor analysis and logistic regression, a model was
tested to predict student success based on:
• ACT or SAT score
• Overall grade-point average
Predicted 66%-80% • CMS usage composite
of struggling • CMS assessment composite
students who • CMS assignment composite
needed help • CMS calendar composite
Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE
Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x 28

29. Desire2Learn visual analytics & predictive models
which can be interrogated on different dimensions
http://www.desire2learn.com/products/analytics
29

30. Desire2Learn visual analytics & predictive models
which can be interrogated on different dimensions
http://www.desire2learn.com/products/analytics
30

31. Socrato: train for SATs
http://www.socrato.com 31

32. Khan Academy: more data to teachers,
finer-grained feedback to students
http://www.thegatesnotes.com/Topics/Education/Sal-Khan-Analytics-Khan-Academy 32

33. Adaptive platforms generate fine-
grained analytics
https://grockit.com/research
33

34. Adaptive platforms generate fine-grained
analytics
http://knewton.com

35. Adaptive platforms generate fine-grained
analytics
http://oli.cmu.edu

36. The VLE—BI convergence
36

37. Hard distinctions between Learning +
Academic analytics may dissolve
…as they get joined up, each level enriches the others
Macro:
region/state/national/international
Meso:
institution-wide
Micro:
individual user actions
(and hence cohort)
Aggregation of user traces
enriches meso + macro analytics
with finer-grained process data

38. Hard distinctions between Learning +
Academic analytics may dissolve
…as they get joined up, each level enriches the others
Macro:
region/state/national/international
Meso:
institution-wide
Micro:
individual user actions
(and hence cohort)
Aggregation of user traces Breadth + depth from macro
enriches meso + macro analytics + meso levels add power to
with finer-grained process data micro analytics

39. …so everybody’s happy?
dawn of a new data-driven
enterprise + science?
39

40. wrong.
a very healthy debate
is brewing…
40

41. data (indeed technology)
is not neutral
data does not wholly
‘speak for itself’
41

42. Measurement tools are not neutral
“accounting tools...do not simply
aid the measurement of economic
activity, they shape the reality
they measure”
Du Gay, P. and Pryke, M. (2002)
Cultural Economy: Cultural Analysis and Commercial Life
Sage, London. pp. 12-13

43. Beyond big data hubris
1. Automating Research Changes the Definition of
Knowledge
2. Claims to Objectivity and Accuracy are Misleading
3. Bigger Data are Not Always Better Data
4. Not All Data Are Equivalent
5. Just Because it is Accessible Doesn’t Make it
Ethical
6. Limited Access to Big Data Creates New Digital
Divides
boyd, d. and Crawford, K. (2001). Six Provocations for Big Data.
Presented to: A Decade in Internet Time: Symposium on the
Dynamics of the Internet and Society, Oxford Internet Institute,
Sept. 21, 2011.
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431

44. Analytics provide maps = systematic
ways of distorting reality
“A marker of the health of the
learning analytics field will be the
quality of debate around what the
technology renders visible and
leaves invisible.”
Buckingham Shum, S. and Deakin Crick, R. (2012). Learning
Dispositions and Transferable Competencies: Pedagogy, Modelling
and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics &
Knowledge. (29 Apr-2 May, 2012, Vancouver, BC). ACM: New York.
Eprint: http://oro.open.ac.uk/32823

45. context
context
context
45

46. Analytics in English schools
RAISEonline platform: cohort visualization
46

47. Will our analytics reflect the progress that
‘Joe’ has made on so many other fronts –
but not his SATs?
47

48. Will our analytics reflect the progress that
‘Joe’ has made on so many other fronts –
but not his SATs?
?48

49. Video conferencing analytics
OU KMi’s Flashmeeting
Video conference spoken foreign language tutorials
Mentor 1 Mentor 2
AV Chat AV Chat
Session
2
3
49

50. Video conferencing analytics
OU KMi’s Flashmeeting
Video conference spoken foreign language tutorials
Mentor 1 Mentor 2
AV Chat AV Chat
1
Session
2
3
50

51. Video conferencing analytics
OU KMi’s Flashmeeting
Video conference spoken foreign language tutorials
— which mentor would you want to have?...
Mentor 1 Mentor 2
AV Chat AV Chat
1
Session
2
3
51

52. Video conferencing analytics
OU KMi’s Flashmeeting
Video conference spoken foreign language tutorials
— which mentor would you want to have?...
Mentor 1 Mentor 2
AV Chat AV Chat
1
Session
Mentor 1 is doing the best
job: at this introductory
2 level, students need
intensive input and
flounder if left
3
52

53. course completion
is only one proxy
for good learning
and what’s easy to
measure isn’t always
what’s most important
53

54. The Wal-Martification of education?
http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college 54
http://lak12.wikispaces.com/Recordings

55. The Wal-Martification of education?
“What counts as
data, how do you get
it, and what does it
actually mean?”
“The basic question is not
what can we measure?
The basic question is
“data narrowness” what does a good
“instrumental learning” education look like?
“students with no curiosity” Big questions.
http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college 55
http://lak12.wikispaces.com/Recordings

56. Course completion as a proxy for learning
http://annezelenka.com/2012/05/05/but-what-about-learning
56

57. Course completion as a proxy for learning
http://annezelenka.com/2012/05/05/but-what-about-learning
NOTE TO SELVES: We are the
HigherEd market who make it
worthwhile for major vendors to
design analytics focused on
57
maximising course completion
PS: HEIs may feel that they are
trapped by external expectations
and requirements. Systems thinking
required…

58. let’s just pretend that learning analytics took
seriously the revolution going on outside the
university front door…
we would need to devise
learning analytics for
this?...
58

59. Learning analytics for this?
“We are preparing students for jobs
that do not exist yet, that will use
technologies that have not been
invented yet, in order to solve
problems that are not even
problems yet.”
“Shift Happens”
http://shifthappens.wikispaces.com
59

60. Learning analytics for this?
“While employers continue to demand high academic
standards, they also now want more. They want people
who can adapt, see connections, innovate,
communicate and work with others. This is true in many
areas of work. The new knowledge-based economies in
particular will increasingly depend on these abilities.
Many businesses are paying for courses to promote
creative abilities, to teach the skills and attitudes that
are now essential for economic success…”
All our Futures: Creativity, culture & education, May 1999
60

61. Learning analytics for this?
“Knowledge of methods alone
will not suffice: there must be
the desire, the will, to employ
them. This desire is an affair
of personal disposition.”
John Dewey, 1933
Dewey, J. How We Think: A Restatement of the Relation of Reflective Thinking to the
Educative Process. Heath and Co, Boston, 1933
61

62. Learning analytics for this?
“The test of successful education
is not the amount of knowledge
that pupils take away from school,
but their appetite to know and
their capacity to learn.”
Sir Richard Livingstone, 1941
62

63. Learning analytics for this?
The Knowledge-Agency Window
co-generation
Expert-led enquiry Student-led enquiry
Knowledge
and use
Teaching as
Authenticity
learning design
Agency
Identity
Repetition,
Pre-scribed
Knowledge
Abstraction
Acquisition
Expert-led teaching Student-led revision
Teacher agency Student agency

64. Learning analytics for this?
Creativity, Culture and
Education (2009)
Changing Young Lives
2012. Newcastle: CCE.
http://www.creativitycultureeducation.org/
changing-young-lives-2012
64

65. Musicality ≠ Musical Reproduction
In those early days the children were taught from the start to develop
their own voice, whether literally singing, or through the
instrument they played. They were not taught music,
but musicality. Central to this tuition were the partimenti, many
pages of detailed music notes which pose many questions,
but leave the pupil to find the solutions. The
music is not a literal transcript, which the musician reads and reproduces.
set of rules and then
The partimenti establish, at the start, a
pose a set of conflicts for the musician to
resolve, in their own way.
65
http://bit.ly/U1vkNf

66. consider
assessment
for learning
(not summative assessment for
grading pupils, teachers,
institutions or nations)
66

67. Assessment for Learning
http://assessment-reform-group.org
67

68. Assessment for Learning
http://assessment-reform-group.org
68

69. Assessment for Learning
http://assessment-reform-group.org
To what extent
could automated
feedback be
designed and
evaluated with
emotional impact
in mind?
69

70. Assessment for Learning
http://assessment-reform-group.org
Can analytics
identify proxies
for such
advanced
qualities? 70

71. Assessment for Learning
http://assessment-reform-group.org
Do analytics provide
constructive next
steps?
71

72. Assessment for Learning
http://assessment-reform-group.org
How do we provide
analytics feedback
that does not
disempower and de-
motivate struggling
learners?
72

73. analytics for…
dispositions
discourse
social networks
See SoLAR Storm: Social Learning Analytics symposium
73
http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium

74. Social Learning Analytics
§ Analytics focused on social learning theories,
practices and platforms, e.g.
§ Discourse analytics: beyond quantitative
summaries of online writing, to qualitative
analysis
§ Social network analytics: visualizing effective
social ties for collective learning
§ Dispositional analytics: measuring students’
readiness to engage in lifelong, lifewide learning
Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge.
Vancouver, 29 Apr-2 May: ACM Press. Eprint: http://oro.open.ac.uk/32910
Buckingham Shum, S. and Ferguson, R., Social Learning Analytics. Educational Technology & Society (Special Issue on Learning & Knowledge Analytics, Eds. G.
Siemens & D. Gašević), 15, 3, (2012), 3-26. http://www.ifets.info Open Access Eprint: http://oro.open.ac.uk/34092

75. Discourse analytics on webinar
textchat
Given a 2.5 hour webinar, where in the live
textchat were the most effective learning
conversations?
Not at the start and end of a webinar
Sheffield, UK not as sunny but if we zoom in on a peak… See you!
as yesterday - still warm
bye for now!
Greetings from Hong Kong
bye, and thank you
Morning from Wiltshire,
80
sunny here! Bye all for now
60
40
20
0
9:28
9:32
10:13
11:48
12:00
12:05
12:04
9:36
9:40
9:41
9:46
9:50
9:53
9:56
10:00
10:05
10:07
10:07
10:09
10:17
10:23
10:27
10:31
10:35
10:40
10:45
10:52
10:55
11:04
11:08
11:11
11:17
11:20
11:24
11:26
11:28
11:31
11:32
11:35
11:36
11:38
11:39
11:41
11:44
11:46
11:52
11:54
12:03
-20
-40
Average Exploratory
-60
Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within Synchronous
Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

76. Discourse analytics on webinar
textchat
Given a 2.5 hour
webinar, where in the
live textchat were the
most effective learning
conversations?
Classified as
“exploratory
talk”
(more
substantive
100 for learning)
50
0
9:28
“non-
9:40
9:50
10:00
10:07
10:17
10:31
10:45
11:04
11:17
11:26
11:32
11:38
11:44
11:52
12:03
-50 exploratory”
Averag
-100
Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within Synchronous
Text Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

77. Discourse Network Analytics =
Concept Network + Social Network Analytics
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st
International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

78. KMi’s Cohere:
a web deliberation platform enabling semantic social
network and discourse network analytics
Rebecca is playing
the role of broker,
connecting 2 peers’
contributions in
meaningful ways
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st
International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

79. Analytics for C21 learning skills
Different social
network patterns
Questioning and
in different
challenging may
contexts may
load onto Critical
load onto
Curiosity
Learning
Relationships
Repeated
Sharing relevant attempts to pass
resources from an online test
other contexts may load onto
may load onto Resilience
Meaning Making
Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning
Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, Vancouver). Eprint: http://oro.open.ac.uk/32823

80. LearningEmergence.net: embedding
dispositional analytics into practice + tools
EnquiryBlogger: Wordpress plugins for reflective learning journals
80

81. Discourse analysis (Xerox Incremental Parser)
Detection of salient sentences in scholarly reports,
based on the rhetorical signals authors use:
BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:
Recent studies indicate … ... new insights provide direct evidence ... … little is known …
… the previously proposed … ... we suggest a new ... approach ... … role … has been elusive
Current data is insufficient …
… is universally accepted ... ... results define a novel role ...
CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:
… unorthodox view resolves … studies ... have provided important The goal of this study ...
paradoxes … advances Here, we show ...
In contrast with previous Knowledge ... is crucial for ... Altogether, our results ... indicate
hypotheses ... understanding
... inconsistent with past findings ... valuable information ... from studies
GENERALIZING: SURPRISE:
... emerging as a promising approach We have recently observed ...
surprisingly
Our understanding ... has grown
exponentially ... We have identified ... unusual
... growing recognition of the The recent discovery ... suggests Ágnes Sándor & OLnet Project:
http://olnet.org/node/512
intriguing roles
importance ...
De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine
Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

82. Human and machine analysis of a text for key
contributions
Document 1 19 sentences annotated 22 sentences annotated
11 sentences same as human annotation
Document 2 71 sentences annotated 59 sentences annotated
42 sentences same as human annotation
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation
De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine
Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

83. Closing thoughts
83

84. “The basic question is not what can
we measure?
The basic question is what does a
good education look like?
Big questions. (Gardner Campbell)
Do we value what we can measure,
or measure what we really value?
And just because this is tough to
do, doesn’t mean we don’t do it.
(Guy Claxton, BBC Radio 4
Education Debate, Nov. 2012)
84

85. Our analytics promote
values, pedagogy and
assessment regimes.
Are we clear which master
our analytics serve? Are we
happy to be judged by them?
simon.buckinghamshum.net
http:// sbskmi
http://twitter.com/ 85