Teacher communities: learning design support, social mechanisms, and case studies Davinia_Hernandez-Leo @JRC_EU_Seville_2019 JCR Seville, 11-12 April 2019Joint Workshop WG 2 & WG 4: Exploring the interplay between Human Learning and Machine Learning - The Citizen Science Perspective I will present an overview of the educational technologies research conducted by the TIDE research group of the Information and Communication Technologies Department at Universitat Pompeu Fabra in Barcelona (http://www.upf.edu/web/tide @TIDE_UPF). The overview will be articulated around the perspective, central to TIDE work, of supporting teachers and teacher communities (e.g., a school) in the design of the best possible (technology-enhanced) learning activities considering their students and their contexts. Main research contributions that will be presented include a community platform for integrated learning design (ILDE), including multiple authoring tools (e.g. edCrumble for blended learning, PyramidApp for collaborative learning) and the use of data analytics at different levels (learning, design, community) to facilitate meaningful social interactions between teachers (e.g. supporting community inquiry, learning redesign). The presentation will include results of European, Spanish and Catalan projects (METIS, RESET, CoT) and our initial work in recently started projects (SmartLET, Illuminated, Spotlighters). Davinia Hernández-Leo is Associate Professor and Serra Hunter Fellow in the Department of Information and Communications Technologies at Universitat Pompeu Fabra, Barcelona (UPF), the Head of the Interactive and Distributed Technologies for Education group (TIDE), Vice-Dean of the UPF Engineering School and the Director of its Unit for teaching quality and innovation. She obtained her PhD at University of Valladolid (2007), has been a visiting scholar at the Open University of the Netherlands (2006), Virginia Tech (2012) and the University of Sydney (2015). Davinia's research lies at the intersection of network and computer applications, human-computer interaction, and learning sciences, with a special focus on technologies for learning design, computer-supported collaborative learning (CSCL), analytics, architectures and devices for learning. She is Vice-President of the European Association of Technology-Enhanced Learning, Chair of the IEEE ICICLE SIG on learning technology standards, and a member of the editorial board of the IEEE Transactions of Learning Technologies. http://www.upf.edu/web/tide

1. Teacher communities: learning design support,
social mechanisms, and case studies
JCR Seville, 11-12 April 2019
Joint Workshop WG 2 & WG 4: Exploring the interplay between
Human Learning and Machine Learning - The Citizen Science Perspective
Davinia Hernández-Leo
TIDE, Universitat Pompeu Fabra, Barcelona
twitter:
@TIDE_UPF
website
:
http://www.upf.edu/web/tide

2. https://www.upf.edu/web/tide/
TIDE research group
2

3. How we started… back in 2009…
Teacher communities
3
What is a teacher community?
How can technology support teacher communities?
Is a teacher community a “citizen science” scenario?

4. Case 1: “Integrated Biomedice” courses
n Context of the Biology and Medicine studies at UPF, Barcelona
n Courses where students are expected to integrate the knowledge
and skills developed in previous subjects
n Teams of teachers from different disciplines
(Bio-chemistry, Anatomy, Genetics, Pharmacology, …) need to work
together in the design of problem-based learning solutions

5. Case 2: “Biologia en context”
n Community of 20 high schools in Catalonia
n Create, co-create and share problem-based learning designs that
promote the situated learning of Biology topics

6. Challenges, technology support, partial solutions
6
Authoring
Sharing
Sharing educational design ideas
Cooperation between teachers in the creation of new designs
Repositories
Social networks
Editors

7. Teacher communities
7
Authoring + social network + repository
(Hernández-Leo et al., 2011)

8. Teacher communities, some lessons learnt
8
• Different patterns in navigation, by knowledge available or by users
• The importance of knowledge and social awareness
• The importance of “live content” (vs. a repository)
• The role of learning design and learning design editors, multiple pedagogies
(Hernández-Leo, et al. 2014)

9. Support teachers in designing the “best possible”
conditions for their students to learn, documenting them,
making their design ideas explicit and sharable.
(Conole, 2012; Agostinho, 2011; Laurillard, 2013;
Mor, Craft & Hernández-Leo, 2013)
Learning Design
9
Facilitates collective processes, inquiry processes
Teaching as a design science

10. Learning Design
ACAD framework: Design -> Runtime
(Carvahlo & Goodyear, 2015)
ACAD framework
10

11. The case of macro-scripting
Free collaboration does not necessarily lead to fruitful
learning. Macro-scripts structure the collaborative learning
flow (activity sequence, roles, group formation, resource
allocation) to trigger potentially effective social interactions.
(Dillenbourg & Tchounikine, 2007; Kobbe et al., 2007)
Learning Design in Collaborative Learning
11

12. Collaborative learning flow patterns
12
Learning Design in Collaborative Learning
(Hernández-Leo et al., 2007)

13. 13
Learning Design in Collaborative Learning
(Hernández-Leo et al., 2006; Villasclaras & Hernández-Leo et al., 2013)

14. n Flexibility challenges, especially in large classrooms
— Flow creation and control mechanisms for elasticity and dynamism
— PyramidApp
14
Learning Design in Collaborative Learning
https://www.upf.edu/web/tide/tools
Manathunga, K., & Hernández‐Leo, D. (2018). Authoring and enactment of mobile pyramid‐based
collaborative learning activities. British Journal of Educational Technology, 49(2), 262-275.

15. n Integrated AI techniques, adaptive vs. adaptable behaviour
n Orchestration dashboard, monitoring and regulation by the teacher
— Group formation
— Time
— Simulated learners and “teaching assistant”
15
Collaborative Learning enactment, PyramidApp
(Amarasinghe et al., accepted; Amarasinghe et al., submitted)

16. 16
Learning design tools
n Tools for conceptualization, authoring, implementation
n Generic vs. specific tools wrt. pedagogical approach

17. 17
Learning design tools
n Co-design with teachers
n E.g. design for blended learning (PBL, flipped classroom, …)

18. 18
Learning design and data analytics: from teacher communities to CSCL scripts
Learning design tools
n Co-design with teachers
n Emphasis in design for blended learning (PBL, flipped classroom, …)
ILDE2/edCrumble
Albo L, Hernández-Leo D. edCrumble:
designing for learning with data analytics.
EC-TEL 2018. Best Demo Award.

19. 19
Integrated Learning Design Environment
n Integrated solution to support the full lifecycle of (co-)designing,
sharing, and deploying ICT-based learning activities
Hernández-Leo, D., et al. (2018). An Integrated Environment for Learning Design. Frontiers in ICT, 5, 9

20. Integrated Learning Design Environment (ILDE)
20
Individual Space Social Space

21. Integrated Learning Design Environment
n Used in several different educational contexts
(higher education, vocational education, adult education)
— Well accepted in all its functions
— Fits the needs of different types of communities
— No difference in acceptance
— Flexible design processes
— Limited social awareness and inquiry process
n Data used to evaluate and understand processes…
21

22. Integrated Learning Design Environment
n Data used to evaluate and understand processes…
n How can data be used to support learning design and
learning design communities?
22

23. Data Analytics in teaching & learning
n Learning Analytics (LA): “the measurement, collection, analysis, reporting of
data about learners and their contexts, for porpuses of understanding and
optimizing learning and the environments in which it occurs” SoLAR (Ferguson,
2012)
n Social Learning Analytics (Buckingham-Shum & Ferguson 2012) & Community
analytics and visualizations (Klamma, 2013; Vasileva and Sun, 2007), including
Social Network Analysis (SNA) (de Laat & Schreurs, 2013)
23

24. ANALYTICS LAYERS FOR LEARNING DESIGN FRAMEWORK
Understanding and supporting learning design with analytics
24
Community of practitioners
and related stakeholders
Learning Design (LD) tools
Learners experiencing
learning designs
Community Analytics
metrics and patterns of design activity
Design Analytics
metrics of pedagogical decisions
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
Hernández-Leo, D., et al. (2019) Analytics for learning design: A layered framework and tools, British Journal of Educational Technology.
https://doi.org/10.1111/bjet.12645

25. Understanding and supporting learning design with analytics
25

26. Understanding and supporting learning design with analytics
n Available data
n ILDE2, extension with data analytics features
— https://www.upf.edu/web/tide/tools/ilde2
n CoT project, 2 schools
n Master program for teacher training
n MOOC for teachers
n Teacher training workshops
26

27. Analytics for learning design
Community of practitioners
and related stakeholders
Community Analytics
metrics and patterns of design activity
n Understanding community behaviour
n Triggering orientation and inspiration
(e.g. awareness, provides a context for your own activity)
n Data includes:
— what tools are being used,
— what types of designs are created, tags,
— who are the (co-)authors, edits, views,
— versioning of designs,
— social ratings …
Michos, K., Hernández-Leo, D., (2016)
Understanding Collective Behavior of Learning
Design Communities. EC-TEL 2016.

28. Analytics for learning design
Community of practitioners
and related stakeholders
Community Analytics
metrics and patterns of design activity
n Understanding community behaviour
n Triggering orientation and inspiration
n Data includes:
— what tools are being used,
— what types of designs are created, tags,
— who are the (co-)authors, edits, views,
— versioning of designs,
— social ratings …
Duplicates ….

29. Analytics for learning design
Community of practitioners
and related stakeholders
Community Analytics
metrics and patterns of design activity
n Understanding community behaviour
n Triggering orientation and inspiration
n Data includes:
— what tools are being used,
— what types of designs are created, tags,
— who are the (co-)authors, edits, views,
— versioning of designs,
— social ratings …
Michos, K., Hernández-Leo, D., (2018) Supporting awareness in communities of
learning design practice, Computers in Human Behavior, 85, 255-270.

30. Development of a community analytics dashboard
n Designs tab, Tools tab, Members tab
30

31. n Blending the dashboard with learning design tasks
n With/Without dashboard use
n Differences in community members´ participation and perceptions
Community dashboard evaluation
31

32. n Dashboard use increased social interactions by means of comments and profile views
n Positive user experience (UMUX questionnaire) for understanding community participation
n Alignment with tasks for learning design
Community dashboard evaluation
32
Variables
Group
N
Ranks
U
Z
p
r
profiles views
High dashboard use
25
31.26
138.50
-3.238
.001
-0.46
Low dashboard use
24
18.27
comments
High dashboard use
25
30.70
157.50
-2.877
.004
-0.41
Low dashboard use
24
19.06
created designs
High dashboard use
25
30.12
172.00
-2.570
.010
-0.36
Low dashboard use
24
19.67
re-used designs
High dashboard use
25
30.08
173.00
-2.547
.011
-0.36
Low dashboard use
24
19.71
edits
High dashboard use
25
29.60
185.00
-2.335
.020
-0.33
Low dashboard use
24
20.21

33. Learning Design (LD) tools
Design Analytics
metrics of pedagogical decisions
n Categorise learning designs
n Support awareness and reflection
about design decisions
n Data includes:
— learning objectives,
— type of tasks tasks,
— the social planes (individual, collaborative),
— resources and tools associated with the task,
— expected length of time for tasks, …
Analytics for learning design
Albo L, Hernández-Leo D. edCrumble: designing for learning
with data analytics. EC-TEL 2018. Best Demo Award.

34. Learning Design (LD) tools
Design Analytics
metrics of pedagogical decisions
n Categorise learning designs
n Support awareness and reflection
about design decisions
n Interaction with the
Community Analytics layer
Analytics for learning design
Albo L, Hernández-Leo D. edCrumble: designing for learning with
data analytics. EC-TEL 2018. Best Demo Award.

35. Learners experiencing
learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
n Accumulated evidence of design’s impact
n Support awareness and reflection
about effects of designs, redesign
n Data includes:
— snapshot data capturing relevant milestones
(e.g., access to resources or completion of tasks)
— data about the process (e.g., participation in
discussions)
— performance (e.g., grades)
— students’ and teachers’ opinions…
Analytics for learning design
Michos, K., Manathunga, K., Hernández-Leo, D. (2016). Connecting pattern-based
learning designs with analytics: The case of the PyramidApp., CLAD - EC-TEL 2016

36. Learners experiencing
learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
n Accumulated evidence of design’s impact
n Support awareness and reflection
about effects of designs, redesign
n Data includes:
— snapshot data capturing relevant milestones
(e.g., access to resources or completion of tasks)
— data about the process (e.g., participation in
discussions)
— performance (e.g., grades)
— students’ and teachers’ opinions…
Analytics for learning design
Michos, K., Fernández, A., Hernández-Leo, D., & Calvo, R.
(2017). Ld-Feedback App: Connecting Learning Designs with
Students’ and Teachers’ Perceived Experiences. EC-TEL.

37. Learners experiencing
learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
n Accumulated evidence of design’s impact
n Support awareness and reflection
about effects of designs, redesign
n Interaction with Design Analytics
and Community Analytics
— Teacher inquiry, Community inquiry
Analytics for learning design
Learning design and data analytics: from teacher communities to CSCL scripts
Michos, K., & Hernández-Leo, D., Albó, L. (2018). Teacher-led
inquiry in technology-supported school communities. British
Journal of Educational Technology 49(6), 1077-1095.

38. Teacher-led Inquiry for Learning dEsigns (TILE) tool
38

39. Teacher produced artifacts & Learning Analytics
39
n Learning Analytics reports
n Teacher produced artifacts
n Reflections on the enactment

40. Some results
n Awareness and regulation of design activity (before, during, and after enactment)
n Connection between initial design and data-informed reflections
n Documentation for later review
n Sharing learning design problems with solutions, with contextual information
n Teacher roles as designers is guided by practicality and time constraints
n In-depth discussions about teaching and learning strategies with the use of student generated
data and TILE
40

41. Collective teacher inquiry with data analytics
41
Cultural Historical
Activity Theory
Teacher Inquiry
Technology &
Data Analytics
Collective process Inquiry
process
Technological support
How do teachers
engage with and
perceive data-
intensive collective
inquiry processes?

42. Collective inquiry - Three case studies
n Teacher comments show evidence of
activated pedagogical knowledge
n Positive perceptions about collective
inquiry-> sharing problems, creating
spaces for pedagogy
n Collective teacher participation
decreases through the time ->
asynchronous work, time constraints,
community moderation
42
MOOC case
Two School cases

43. Conclusions
n What is a teacher community? How can technology support teacher
communities?
n Is a teacher community a “citizen science” scenario?
43
n Teacher communities: learning design support, social mechanisms, and case
studies
n Data-driven support
n Three analytics layers: Community, Design and Learning analytics
n And interplays!
n Teacher inquiry, learning redesign, awareness, inspiration, community inquiry

44. Current, future work
n New communities (e.g., Maker activities; PBL & SDG, collaboration with UNESCOCat)
n Identifying patterns in data, towards recommenders
n Multimodal learning analytics
n Actionable dashboards (Pyramid App)
n Neuroscience and learning design support (Illuminated, Spotlighters)
n Privacy, regulations, ethical implications…
— Beardsley, M., Santos, P., Hernández-Leo, D., Michos, K. (2019). Ethics in educational technology research:
informing participants in data sharing risks. British Journal of Educational Technology. 10.1111/bjet.12781
n New project COURAGE (Volkswagen Stiftung) about educating around social media
threads, an AI companion.
44

45. Thank you!
twitter:
@TIDE_UPF
website
:
http://www.upf.edu/web/tide
METIS, RESET, CoT, MdM, SmartLET, Illuminated, Spotlighers