2016-05-30 Venia Legendi (CEITER): Luis Pablo Prieto
1. Orchestration and
Learning Analytics
for Educational Innovation
Luis P. Prieto
Venia Legendi for a Senior Research Fellow position
Tallinn University, 30 May 2016
2. 1. Orchestration and Learning
Analytics: A lecture
2. A vision to apply them in
CEITER (and beyond)
2
5. Orchestration is…
“the process of
productively coordinating
supportive interventions
across multiple learning
activities occurring at
multiple social levels”
(Dillenbourg, Järvelä & Fischer, 2009)
5
11. Application to edutech design
• GLUE!-PS: From an open architecture to solve the
fragmentation of learning design authoring tools…
• … to a tool to support orchestration of existing
VLEs and external tools (e.g., runtime changes, etc.)
(Prieto et al., 2014) 11
12. What does orchestration entail?
The ‘5+3 Aspects’ framework
(Prieto, Holenko-Dlab, Gutiérrez, Abdulwahed & Balid, 2011) 12
13. Application to edutech evaluation
• Does GLUEPS-AR support the orchestration of
mobile AR-based learning activities?
(Muñoz-Cristóbal et al., 2015)
13
16. Applications to teacher
professional development
Observational
studies
Successful
routines/
practices/
patterns
Professional
development
workshops
(Prieto et al., 2013)
16
18. Modelling teacher orchestration:
diving deeper with eye-tracking
• Study teacher cognitive
(orchestration) load
• Class-level interactions are
higher load
• Reading faces is higher load
• Novice teachers have clearer
load trends than experts
• …
(Prieto, Wen, Caballero, Sharma & Dillenbourg, 2014)
(Prieto, Sharma, Wen & Dillenbourg, 2015)
(Prieto, Sharma & Dillenbourg, 2015)
18
19. Learning Analytics
• Main goal: Aiding educators in understanding and
improving teaching and learning processes
• Main difference with Educational Data Mining
(EDM): human in the loop
• Aimed at interventions/supportive actions
• Cycle of data gathering, analysis, feedback/visualization
• Hence, LA can be seen as a very useful tool for
orchestration
• Awareness/Assessment aspect in ‘5+3’ framework
(Siemens & Long, 2011)
19
(Clow, 2012)
20. Orchestration + LA =
Teaching Analytics
• First attempt at automating the
observational modelling of orchestration
graphs using multimodal analysis
• Audio, video, eyetracking, accelerometers, EEG
• Exploration of basic, general-purpose
features and algorithms
(Prieto, Sharma, Rodríguez-Triana & Dillenbourg, 2016) 20
22. Orchestration + Learning Analytics
= Teaching Analytics
• First attempt at automating the observational
modelling of orchestration graphs using
multimodal analysis
• Audio, video, eyetracking, accelerometers, EEG
• Exploration of basic, general-purpose features and
algorithms
• Results:
• Predicting teacher activity accuracy: 65%
• Predicting social plane accuracy: 90%
• Audio-video channels most useful
• Random forest and GBM as best algorithms
(Prieto, Sharma, Rodríguez-Triana & Dillenbourg, 2016) 22
23. 2. A vision to apply them
in CEITER (and beyond)
23
25. Previously, on this
Venia Legendi…
• Orchestration as a (complex) metaphor for
teaching practice
• Focus on an innovation’s potential for adoption
within ecosystem of authentic educational setting
• Applications to educational technologies as well as
the analysis of (adoption) practice
• Learning Analytics (LA)
• “Orchestration-flavored” take on EDM?
• Multimodal LA to model what happens outside the
box (face-to-face)
25
Image from https://www.flickr.com/photos/65092514@N08/18679295525
27. An example orchestration/LA
research project
• Prolearning: simple app to foster PD conversations
based on everyday data gathering by teachers
• Students are asked simple questions about their
experience in terms of school-emphasized practices
& teachers have to predict student response
• http://prolearning.realto.ch
• Successes:
• Teachers used it in ~70% of all their lessons for 2 weeks
• Recorded evidence of changes in student experience
• Takes 2 mins: “I can’t see how it can be more efficient”
27
28. What didn’t work
Teacher interviews:
“… any time you’re teaching
there is a hundred variables
that you have to account
for”
“[I would use it] if you can
demonstrate that these data
are reliable…”
28
29. Unsolved challenges
1. Trust, privacy, agency and other ethical factors
• Traditional top-down innovations quickly get subverted
2. The quest for added value
• Hard: Evidence of benefits for student learning
• Soft: Help with existing “chores”, social value…
3. Still does not scale well!
• Measure blended learning, but with ecological validity?
• Slow: cycles of analysis and reflection are loooooong
• Can we do better than final course assessment?
• Fail to create new practices that go with the technology
• Routines are how we deal with our complex environments
29
30. The CEITER project
• From project description:
• Teaching 21st century skills
• Innovative methods & learning environments tailored to
learner’s development/needs/capabilities
• Improving the research staff capacity & new generation of
researchers
• Use Learning Analytics
• From Tobias’s talk (06.04.2016):
• Tech/Tools cannot be separated from teaching and learning
practices
• Educational innovation is taken up at different social entities
• Multiple levels of analysis (institutional, PD, learner…)
30
31. Can we change Estonian
education from the
“ivory tower”?
Probably not…
31
32. Fresh perspective
on Learning
Analytics
Ecosystem of
tools for evidence-
based orchestration
CEITER
Agency &
Ethics
Clear added
value
Adoption
& scale1 2 3
Blended LA in
distributed
LEs + ‘just
enough’
multimodal LA
Focus on
assessing
learning
Focus on
everyday
evidence
Capture
successful
practices
Teacher
training and
community
of practice
Open, privacy-
conscious
architecture +
specific tools
Integrate
existing
tools and
practices
Lightweight
assessment
techniques
Evidence-
based
training
Orchestr
ation-
aware
tools
32
34. Modus operandi
• Design-based research, mixed methods,
participatory…
• Lean startup model: start qualitative, slowly build
quantitative indicators, minimal prototypes
• Cross-functional teams/projects
• Eat your own dog food!
• Promote international exchanges (students, faculty)
• Focus on EduTech entrepreneurship to attract
students & researchers
34
35. A path for (part of) CEITER
35
Phase 1
1st LA
prototypes
Application
in PD
(teachers &
researchers)CEITER as
a 1st
evidence-
based CoP
Basic practice
capture
Phase 2
1st
orchestrati
on tools
Application
in teacher
education
Pre-service
teachers
eb-CoP
Focused
capture, multi-
classroom
To phase 3
36. Why me?
Contributions to wide variety of
orchestration-related research
communities
• From CSCL to LD to HCI to LA…
Experience in international
projects & collaboration
Quanti & quali methods
Used to Enjoys inter-disciplinary
work
Motivated by innovation
practice, teacher PD, …
FP
6
FP
7LLP
36
37. … but not only me!
• Other CEITER team members:
• Existing expertise in lifelong & workplace learning
• Infrastructure expert to communicate different data
sources and databases
• Psychology profile: to determine the “building blocks” of
learning and how to (micro-)assess them
• Pedagogical approach profile: to model existing
practices/tools and propose new ones
• Other researchers at U. Tallinn (or T.U. Tallinn)
• Partnerships with local HCI, Signal processing, Machine
learning, Sensors, (Estonian) voice recognition… experts
37
38. … but not only me! (II)
• My existing network of relevant contacts, e.g.:
• Orchestration: P. Dillenbourg (CH), P. Tchounikine (FR)
• Pedagogies, e.g., CSCL: Y. Dimitriadis (ES), D. Persico (IT)
• Multimodal Learning Analytics: X. Ochoa (EC), S.
D’Mello (US)
• Large-scale school innovation: C.K. Looi (SNG), J.
Roschelle (US)
• U. Tallinn’s own network of national & international
contacts
• Incl. Estonian teacher/school networks, policy-makers
38
39. Beyond CEITER
• This vision can be too ambitious for CEITER itself
• Once we have initial results, use them to “pitch”
consortium and proposals
• EU calls:
• Evidence-based policy aspect: SwafS-21-2017
• Explore prototypes’ market potential: SMEInst-12-2016-2017
• If things go well, ERC-STG-2018/2019…
• Other funding for non-EU countries, e.g.:
• US: Partnerships for International Research and Education (NSF
PIRE)
• Singapore (parallel for systemic change): advisors/expert for
national project (visit planned 2017)
39
40. This is just a modular vision…
CEITERBlended LA in
distributed
LEs + ‘just
enough’
multimodal LA
Teacher
training and
community
of practice
Open, privacy-
conscious
architecture +
specific tools
Lightweight
assessment
techniques
40
41. … within an ecosystem of interests
Blended LA in
distributed
LEs + ‘just
enough’
multimodal LA
Teacher
training and
community
of practice
Open, privacy-
conscious
architecture +
specific tools
Lightweight
assessment
techniques
Agile &
entrepreneurship
methods in
research
Scientific writing
& communication
support
Researcher
communities
of practice
MOOCs
Machine
learning
Learning
design
Creative
writing
Tangible
& paper
UIs
Data
literacy
Data
science
CSCL
Dissemination
to the public
41
