1.
Keynote Address, Expanding Horizons 2012, Macquarie Universityhttp://staff.mq.edu.au/teaching/workshops_programs/expanding_horizonsOur Learning Analyticsare Our Pedagogy Simon Buckingham Shum @sbskmi Knowledge Media Institute, The Open University UK simon.buckinghamshum.net 1

2.
learning objective: walk out withbetter questionsthan you can ask right now 2

3.
Urgent need: quality dialogue betweenanalytics stakeholders, to accelerateinvention innovation www.SoLAResearch.org Follow: @SoLAResearch 3

4.
The global demand for learning Implications for assessment and feedback at John Daniel massive scale?http://www.col.org/resources/speeches/2012presentations/Pages/2012-02-01.aspx 4

5.
is educationpoised to become a data-driven enterprise and science ? 5

6.
Possibly 90% of the digital data we havetoday was generated in the last 2 years Volume outstrips old infrastructure Variety Internet of things, e-business transactions, environmental sensors, social media, audio, video, mobile… Velocity The speed of data access and analysis is exploding A quantitative shift on this scale is in fact a qualitative shift, requiring new ways of thinking about societal phenomena 6

7.
edX: “this is big data, giving us the chanceto ask big questions about learning” Will the tomorrow’s educational researcher be as helpless without an analytics infrastructure, as a geneticist without genome databases, or a physicist without CERN? 7

8.
Lifelogging: explosion of data captureand sharing about personal activities http://www.mirror-project.eu http://quantifiedself.com/guide 8

9.
Educational Data Mining research community

10.
Learning Analytics research community

11.
different levels of analytic 11

12.
‘Learning Analytics’ and‘Academic Analytics’Long, P. and Siemens, G. (2011), Penetrating the fog: analytics in learning and education. Educause Review Online,46, 5, pp.31-40. http://www.educause.edu/ero/article/penetrating-fog-analytics-learning-and-education 12

13.
Macro/Meso/Micro Learning Analytics Macro: region/state/national/international

14.
Macro/Meso/Micro Learning Analytics Macro: region/state/national/international Meso: institution-wide

15.
Macro/Meso/Micro Learning Analytics Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Will institutions be dazzled by the dashboards, or know what questions to ask at each level?

16.
Macro/Meso/Micro Learning Analytics Macro: region/state/national/international

17.
US states are getting the infrastructurein placedataqualitycampaign.org 17

18.
Shared Learning Collaborativehttp://slcedu.org 18

19.
National league tables for English schools 19

20.
Macro/Meso/Micro Learning Analytics Meso: institution-wide

21.
Analytics-savvy Leaders are the future?Parr-Rud, O. (2012). Drive Your Business with Predictive Analytics. SAS White Paperhttp://www.sas.com/reg/gen/corp/1800392 21

22.
Business Intelligence companies see aneducation market opening up These are pedagogically agnostic: they seek to optimize operational efficiency whatever the sector These may make pedagogical assumptions: how will learning design and assessment regimes shape the analytics they offer?http://www.sas.com/industry/education/highered 22

23.
Business Intelligence companies see aneducation market opening up …but do they know anything about the roles that language plays in learning and knowledge construction? 23

24.
BI+HigherEd communities of practice 24

25.
Business Intelligence ≠ Learning Analytics

26.
Macro/Meso/Micro Learning Analytics Micro: individual user actions (and hence cohort)

27.
Analytics in your VLE:Blackboard: feedback to studentshttp://www.blackboard.com/Platforms/Analytics/Overview.aspx 27

28.
Purdue University Signals: real time traffic-lights for students based on predictive model Premise: academic success is defined as a function of aptitude (as measured by standardized test scores and similar information) and effort (as measured by participation within the online learning environment). Using factor analysis and logistic regression, a model was tested to predict student success based on: • ACT or SAT score • Overall grade-point average Predicted 66%-80% • CMS usage composite of struggling • CMS assessment composite students who • CMS assignment composite needed help • CMS calendar composite Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x 28

29.
Desire2Learn visual analytics & predictive modelswhich can be interrogated on different dimensionshttp://www.desire2learn.com/products/analytics 29

30.
Desire2Learn visual analytics & predictive modelswhich can be interrogated on different dimensionshttp://www.desire2learn.com/products/analytics 30

31.
Socrato: train for SATshttp://www.socrato.com 31

32.
Khan Academy: more data to teachers,finer-grained feedback to studentshttp://www.thegatesnotes.com/Topics/Education/Sal-Khan-Analytics-Khan-Academy 32

33.
Adaptive platforms generate fine-grained analyticshttps://grockit.com/research 33

34.
Adaptive platforms generate fine-grainedanalytics http://knewton.com

35.
Adaptive platforms generate fine-grainedanalyticshttp://oli.cmu.edu

36.
The VLE—BI convergence 36

37.
Hard distinctions between Learning +Academic analytics may dissolve…as they get joined up, each level enriches the others Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Aggregation of user tracesenriches meso + macro analytics with finer-grained process data

38.
Hard distinctions between Learning +Academic analytics may dissolve…as they get joined up, each level enriches the others Macro: region/state/national/international Meso: institution-wide Micro: individual user actions (and hence cohort) Aggregation of user traces Breadth + depth from macroenriches meso + macro analytics + meso levels add power to with finer-grained process data micro analytics

39.
…so everybody’s happy?dawn of a new data-driven enterprise + science? 39

40.
wrong.a very healthy debate is brewing… 40

41.
data (indeed technology) is not neutraldata does not wholly ‘speak for itself’ 41

42.
Measurement tools are not neutral“accounting tools...do not simply aid the measurement of economic activity, they shape the reality they measure” Du Gay, P. and Pryke, M. (2002) Cultural Economy: Cultural Analysis and Commercial Life Sage, London. pp. 12-13

43.
Beyond big data hubris1. Automating Research Changes the Definition of Knowledge2. Claims to Objectivity and Accuracy are Misleading3. Bigger Data are Not Always Better Data4. Not All Data Are Equivalent5. Just Because it is Accessible Doesn’t Make it Ethical6. Limited Access to Big Data Creates New Digital Divides boyd, d. and Crawford, K. (2001). Six Provocations for Big Data. Presented to: A Decade in Internet Time: Symposium on the Dynamics of the Internet and Society, Oxford Internet Institute, Sept. 21, 2011. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1926431

44.
Analytics provide maps = systematicways of distorting reality“A marker of the health of the learning analytics field will be the quality of debate around what the technology renders visible and leaves invisible.” Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, 2012, Vancouver, BC). ACM: New York. Eprint: http://oro.open.ac.uk/32823

45.
contextcontextcontext 45

46.
Analytics in English schoolsRAISEonline platform: cohort visualization 46

47.
Will our analytics reflect the progress that‘Joe’ has made on so many other fronts –but not his SATs? 47

48.
Will our analytics reflect the progress that‘Joe’ has made on so many other fronts –but not his SATs? ?48

49.
Video conferencing analyticsOU KMi’s FlashmeetingVideo conference spoken foreign language tutorials Mentor 1 Mentor 2 AV Chat AV ChatSession 2 3 49

50.
Video conferencing analyticsOU KMi’s FlashmeetingVideo conference spoken foreign language tutorials Mentor 1 Mentor 2 AV Chat AV Chat 1Session 2 3 50

51.
Video conferencing analyticsOU KMi’s FlashmeetingVideo conference spoken foreign language tutorials — which mentor would you want to have?... Mentor 1 Mentor 2 AV Chat AV Chat 1Session 2 3 51

52.
Video conferencing analyticsOU KMi’s FlashmeetingVideo conference spoken foreign language tutorials — which mentor would you want to have?... Mentor 1 Mentor 2 AV Chat AV Chat 1Session Mentor 1 is doing the best job: at this introductory 2 level, students need intensive input and flounder if left 3 52

53.
course completion is only one proxy for good learning and what’s easy tomeasure isn’t alwayswhat’s most important 53

54.
The Wal-Martification of education?http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college 54http://lak12.wikispaces.com/Recordings

55.
The Wal-Martification of education? “What counts as data, how do you get it, and what does it actually mean?” “The basic question is not what can we measure? The basic question is “data narrowness” what does a good “instrumental learning” education look like? “students with no curiosity” Big questions.http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college 55http://lak12.wikispaces.com/Recordings

56.
Course completion as a proxy for learninghttp://annezelenka.com/2012/05/05/but-what-about-learning 56

57.
Course completion as a proxy for learninghttp://annezelenka.com/2012/05/05/but-what-about-learning NOTE TO SELVES: We are the HigherEd market who make it worthwhile for major vendors to design analytics focused on 57 maximising course completion PS: HEIs may feel that they are trapped by external expectations and requirements. Systems thinking required…

58.
let’s just pretend that learning analytics tookseriously the revolution going on outside the university front door…we would need to devise learning analytics for this?... 58

59.
Learning analytics for this?“We are preparing students for jobs that do not exist yet, that will use technologies that have not been invented yet, in order to solve problems that are not even problems yet.” “Shift Happens” http://shifthappens.wikispaces.com 59

60.
Learning analytics for this?“While employers continue to demand high academic standards, they also now want more. They want people who can adapt, see connections, innovate, communicate and work with others. This is true in many areas of work. The new knowledge-based economies in particular will increasingly depend on these abilities. Many businesses are paying for courses to promote creative abilities, to teach the skills and attitudes that are now essential for economic success…” All our Futures: Creativity, culture & education, May 1999 60

61.
Learning analytics for this? “Knowledge of methods alone will not suffice: there must be the desire, the will, to employ them. This desire is an affair of personal disposition.” John Dewey, 1933Dewey, J. How We Think: A Restatement of the Relation of Reflective Thinking to theEducative Process. Heath and Co, Boston, 1933 61

62.
Learning analytics for this?“The test of successful education is not the amount of knowledge that pupils take away from school, but their appetite to know and their capacity to learn.” Sir Richard Livingstone, 1941 62

63.
Learning analytics for this? The Knowledge-Agency Window co-generation Expert-led enquiry Student-led enquiry Knowledge and use Teaching as Authenticity learning design Agency Identity Repetition, Pre-scribed Knowledge Abstraction Acquisition Expert-led teaching Student-led revision Teacher agency Student agency

64.
Learning analytics for this? Creativity, Culture and Education (2009) Changing Young Lives 2012. Newcastle: CCE. http://www.creativitycultureeducation.org/ changing-young-lives-2012 64

65.
Musicality ≠ Musical Reproduction In those early days the children were taught from the start to develop their own voice, whether literally singing, or through the instrument they played. They were not taught music, but musicality. Central to this tuition were the partimenti, many pages of detailed music notes which pose many questions, but leave the pupil to find the solutions. The music is not a literal transcript, which the musician reads and reproduces. set of rules and then The partimenti establish, at the start, a pose a set of conflicts for the musician to resolve, in their own way. 65http://bit.ly/U1vkNf

66.
consider assessment for learning(not summative assessment for grading pupils, teachers, institutions or nations) 66

67.
Assessment for Learninghttp://assessment-reform-group.org 67

68.
Assessment for Learninghttp://assessment-reform-group.org 68

69.
Assessment for Learninghttp://assessment-reform-group.org To what extent could automated feedback be designed and evaluated with emotional impact in mind? 69

70.
Assessment for Learninghttp://assessment-reform-group.org Can analytics identify proxies for such advanced qualities? 70

71.
Assessment for Learninghttp://assessment-reform-group.org Do analytics provide constructive next steps? 71

72.
Assessment for Learninghttp://assessment-reform-group.org How do we provide analytics feedback that does not disempower and de- motivate struggling learners? 72

73.
analytics for… dispositions discourse social networks See SoLAR Storm: Social Learning Analytics symposium 73http://www.open.ac.uk/blogs/SocialLearnResearch/2012/06/20/social-learning-analytics-symposium

74.
Social Learning Analytics § Analytics focused on social learning theories, practices and platforms, e.g. § Discourse analytics: beyond quantitative summaries of online writing, to qualitative analysis § Social network analytics: visualizing effective social ties for collective learning § Dispositional analytics: measuring students’ readiness to engage in lifelong, lifewide learningFerguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics & Knowledge. Vancouver, 29 Apr-2 May: ACM Press. Eprint: http://oro.open.ac.uk/32910Buckingham Shum, S. and Ferguson, R., Social Learning Analytics. Educational Technology & Society (Special Issue on Learning & Knowledge Analytics, Eds. G. Siemens & D. Gašević), 15, 3, (2012), 3-26. http://www.ifets.info Open Access Eprint: http://oro.open.ac.uk/34092

75.
Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar Sheffield, UK not as sunny but if we zoom in on a peak… See you! as yesterday - still warm bye for now! Greetings from Hong Kong bye, and thank you Morning from Wiltshire, 80 sunny here! Bye all for now 60 40 20 0 9:28 9:32 10:13 11:48 12:00 12:05 12:04 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:52 11:54 12:03 -20 -40 Average Exploratory -60Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within SynchronousText Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

76.
Discourse analytics on webinar textchat Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Classified as “exploratory talk” (more substantive 100 for learning) 50 0 9:28 “non- 9:40 9:50 10:00 10:07 10:17 10:31 10:45 11:04 11:17 11:26 11:32 11:38 11:44 11:52 12:03 -50 exploratory” Averag -100Wei & He extensions to: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue within SynchronousText Chat. Proc. 1st Int. Conf. Learning Analytics & Knowledge. Feb. 27-Mar 1, 2011, Banff. ACM Press. Eprint: http://oro.open.ac.uk/28955

77.
Discourse Network Analytics = Concept Network + Social Network AnalyticsDe Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1stInternational Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

78.
KMi’s Cohere: a web deliberation platform enabling semantic social network and discourse network analytics Rebecca is playing the role of broker, connecting 2 peers’ contributions in meaningful waysDe Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1stInternational Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) http://oro.open.ac.uk/25829

79.
Analytics for C21 learning skills Different social network patterns Questioning and in different challenging may contexts may load onto Critical load onto Curiosity Learning Relationships Repeated Sharing relevant attempts to pass resources from an online test other contexts may load onto may load onto Resilience Meaning MakingBuckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and LearningAnalytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, Vancouver). Eprint: http://oro.open.ac.uk/32823

80.
LearningEmergence.net: embeddingdispositional analytics into practice + toolsEnquiryBlogger: Wordpress plugins for reflective learning journals 80

81.
Discourse analysis (Xerox Incremental Parser)Detection of salient sentences in scholarly reports,based on the rhetorical signals authors use:BACKGROUND KNOWLEDGE: NOVELTY: OPEN QUESTION:Recent studies indicate … ... new insights provide direct evidence ... … little is known …… the previously proposed … ... we suggest a new ... approach ... … role … has been elusive Current data is insufficient …… is universally accepted ... ... results define a novel role ...CONRASTING IDEAS: SIGNIFICANCE: SUMMARIZING:… unorthodox view resolves … studies ... have provided important The goal of this study ...paradoxes … advances Here, we show ...In contrast with previous Knowledge ... is crucial for ... Altogether, our results ... indicatehypotheses ... understanding... inconsistent with past findings ... valuable information ... from studiesGENERALIZING: SURPRISE:... emerging as a promising approach We have recently observed ... surprisinglyOur understanding ... has grownexponentially ... We have identified ... unusual... growing recognition of the The recent discovery ... suggests Ágnes Sándor & OLnet Project: http://olnet.org/node/512 intriguing rolesimportance ...De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-MachineAnnotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

82.
Human and machine analysis of a text for keycontributions Document 1 19 sentences annotated 22 sentences annotated 11 sentences same as human annotation Document 2 71 sentences annotated 59 sentences annotated 42 sentences same as human annotationhttp://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotationDe Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-MachineAnnotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052

83.
Closing thoughts 83

84.
“The basic question is not what can we measure? The basic question is what does a good education look like? Big questions. (Gardner Campbell) Do we value what we can measure, or measure what we really value? And just because this is tough to do, doesn’t mean we don’t do it. (Guy Claxton, BBC Radio 4 Education Debate, Nov. 2012) 84

85.
Our analytics promote values, pedagogy and assessment regimes. Are we clear which master our analytics serve? Are we happy to be judged by them? simon.buckinghamshum.nethttp:// sbskmi http://twitter.com/ 85