Presentation on holistic improvement and learning analytics using hierarchical proess modellling at the Australian Learning Analytics Summer School 2014

1. ANALYTICS FOR HOLISTIC
EDUCATIONAL IMPROVEMENT:
UTS CRICOS PROVIDER CODE: 00099F
cic.uts.edu.au

2. THE STORY IN A SLIDE
• Organisations are complex and ‘messy’
• There are different ‘ways of knowing’ through which humans seek
knowledge and represent that knowledge through data.
• Using data to derive actionable insights for leadership decisioning
must take account of such complexity and be aligned to purpose &
timely.
• Case study – hierarchical process modelling is a way of structuring
data for leadership decisioning
• The perimeta dashboard – what it looks and feels like
• Actionable insghts for leadership decisioning
• What next? Key issues?
cic.uts.edu.au

3. DIFFERENT FORMS OF
RATIONALITY
http://www.youtube.com/watch?v=jBl6ALNh18Q
Jurgen Habermas
b 1929
Knowledge seeking
human interests
Empirical analytical
Hermeneutical
Emancipatory

4. cic.uts.edu.au
Strategic Rationality
region/state/national/international
Hermeneutical Rationality
institution-wide contextual
Empancipatory Rationality
individual learner progress
Policy
Makers
Researchers
Leaders/
Teachers
Learning Analytics UNESCO Policy Briefing — Simon Buckingham Shum
4
DATA FOR LEARNING ANALYTICS
Learners

5. Measurement models for learning journeys: using different types of data
and different quality criteria
Narrative; self-report;
interpretation.
Self report
questionnaires,
observation, self-assessment.
Tests of knowledge,
depths of
understanding,
MCQs,
visualisations.
Community of
Peers, public
presentations,
evaluations
Hierarchical
Process modelling
Identity
purpose
motivation
Personal
qualities
Knowledge
structuring
Performing
Authenticity
Trustworthiness
Credibility
Coherence
Reliability
Validity
Replicability
Usefulness

6. BACKGROUND
The Education Charter
The Echo Project
Three Oasis Academies

7. WHAT TO DO?
"Here is Edward Bear, coming
downstairs, bump, bump, bump, on
the back of his head, behind
Christopher Robin. It is, as far as he
knows, the only way of coming
downstairs, but sometimes he
feels that there really is another
way, if only he could stop bumping
for a moment and think of it"
Alan Alexander Milne
The opening paragraph of “Winnie-the-Pooh”, 1926

8. Systems thinking – managing complexity
Developing resilience – becoming learning organisations
Responding to emergence – unpredictable outcomes
Taking responsibility for our own development
Engaging all stakeholders in a networked
improvement community
Using rich data for actionable insights

9. KEY QUESTIONS
How can we measure the full range of what we value in
schools?
How can we collect and manage evidence to
understand the big picture as well as the detail?
How can we lead and manage in conditions of
complexity?
How can we used data for real-time leadership
decisioning as well as for evaluating performance

10. Macro Layers and feedback loops……
Learning is a core process at multi-levels in learning
organisations
Community
Learning
Leaders
Learning
Teachers
Learning
Students
Learning
Blockley, D. (2010) The Importance of Being Process, Civil Engineering and Environmental Systems, 27(3)189-199

11. THE COMPLEX PROBLEM OF CREATING
LEARNING COMMUNITIES
cic.uts.edu.au

12. IMAGINE ……

13. 13
Define purpose
Analyze system
Design Model
Design measurement
parameters for
evaluating success
Collect data
according to
measurement model
over time
Implement
prototypes and
Enter into HPM
model and analyse
performance
Explore solutions as
prototypes
Feedback to system
leaders
evaluate
Hierarchical
Process Modeling
for a Holistic
Approach to
System
Improvement
P
A S
D

14. What sort of design should
we use to model our
learning communities?

15. HIERARCHICAL PROCESS MODELLING

16. Purpose = How (who what where when)
cic.uts.edu.au

17. Outcome
1.1 Establishing and sustaining a group of high achieving learning communities
that enables everyone to realise their full potential and refuses to put limits on
achievement
2. Outputs
2.1 Developing the learning
of students so that they
realise their full potential
2.2 Developing the
learning of teachers
so that they realise
their full potential
2.3 Developing the
learning of leaders
so that they lead the
learning of teachers
and students
effectively
2.4 Engaging parents/
carers effectively in the
learning activities of the
Academy and in
supporting the learning of
their children
3. Inputs
3.1 Feeling valued and included in our community
3.2 Feeling part of a community that focuses its
activities on learning and achievement
3.3 Knowing that I am becoming an effective
learner
3.4 Knowing that we are being helped to reach
our full potential
3.5 Maintaining a good rate of progress at key
transitions, for example, primary to secondary
school
3.6 Achieving results that meet my aspirations
and expectations
3.7 Feeling that my successes are recognised
3.8 Contributing actively to a community which
focuses on service to others
3.9 Contributing actively to a community where
shared leadership is promoted
3.10 Providing evidence that training and
development opportunities have helped me to
improve my classroom practice and/or my
effectiveness as a leader and manager
3.11 Contributing actively to a community that
learns together, shares what works best and
knows what it needs to do to improve
3.12 Feeling included and involved with my
children’s education at the Academy and knowing
that there is an open door for contact when I
need it
3.13 Being effective role models as leaders of
learning
Sources
of
evidence
Students
Teachers
Leaders
Parents/ carers
PURPOSE
How data

18. A Hierarchical Model: Defined by Purpose

19. What type of data do we
need to collect to measure
the core processes which
fulfil our purpose?

20. 20
Define purpose
Analyze system
Design Model
Design measurement
parameters for
evaluating success
Collect data
according to
measurement model
over time
Implement
prototypes and
Enter into HPM
model and analyse
performance
Explore solutions as
prototypes
Feedback to system
leaders
evaluate
Hierarchical
Process Modeling
for a Holistic
Approach to
System
Improvement
P
A S
D

21. CAPTURING DATA FOR CONTINUAL
IMPROVEMENT
cic.uts.edu.au
Surveys for Open Learning Analytics (SOLA )
Platform hosting apps to support self-directed change
DATA WAREHOUSING FOR SYSTEM WIDE RESEARCH
An integrated platform for systemic improvement
LEGACY
SYSTEMS
BLACKBOAR
D
HIERARCHICAL
PROCESS MODELLING
KNOWLEDGE
STRUCTURING TOOLS
SOCIAL LEARNING
ENVIRONMENTS

22. Stakeholders Type of data Focus
1 Y7 students Numerical scores per dimension Learning power
2 Y8 & 10
Students
Numerical values for each question students as learners and
values & citizenship
3 Y9 students Stories rated by researchers Personal transformation
4 Y11 leavers Numerical values for each question Aspirations, trajectories
5 Parents/carers Numerical values for each question Participation
6 Teachers Numerical values for each question Impact of professional
learning on student
outcomes
7 Leaders Numerical values for each question Leaders as learners
8 KPI s Numerical values of performance Behavioural
indicators
9 Y11 Numerical values of performance Attainment

23. Taking responsibility for my own learning over time through defining my purposes, understanding and
managing my feelings, knowing how I go about learning & planning my learning journey carefully.
Mindful Agency
Sense making
Using my intuition &
imagination to generate new
ideas & knowledge. Taking
risks & playing with ideas
and artefacts to arrive at
new solutions.
Creativity
Having the optimism & hope that I can
learn & achieve over time. Having a
growth mindset; believing I can generate
my own new knowledge for what I need
to achieve
Hope and
optimism
Being able to work with others, to
collaborate and co-generate new
ideas and artefacts. Being able to
listen and contribute productively
to a team.
Being part of a learning community at
work, at home, in education & in my
social networks. Knowing I have social
resources to draw on when I need them
Belonging Curiosity
Collaboration
Making connections
between what I already
know & new information &
experience. Making
meaning by linking my
story, my new learning &
my purpose.
Wanting to get beneath the
surface & find out more.
Always wondering why and
how.
1
An emotional orientation of being open & ready to invest in learning, having flexible self-belief, willing to
persist & manage any self-doubt. A necessary pre-requisite for developing resilience in learning
Dependent Openness to learning Closed

24. THE NATURE OF UNCERTAINTY
Fuzziness Incompleteness Randomness

25. .. COMBINED WITH PERFORMANCE MEASURES THAT
COPE EXPLICITLY WITH UNCERTAIN EVIDENCE
Evidence that A
is successful Lack of Evidence
Evidence that A
is not successful
0 1
Performing well with little uncertainty
Performing poorly with some uncertainty
We don’t really know what is going on!

26. From Likert type scales to measures of
uncertainty

27. Hall et al’s (2004) mapping from linguistic descriptions of ‘performance’ and
‘confidence in judgement of performance’ to interval values of performance

28. So what does our
Perimeta model tell us
about Parents/carers ..…
an example

29. 29
Define purpose
Analyze system
Design Model
Design measurement
parameters for
evaluating success
Collect data
according to
measurement model
over time
Implement
prototypes and
Enter into HPM
model and analyse
performance
Explore solutions as
prototypes
Feedback to system
leaders
evaluate
Hierarchical
Process Modeling
for a Holistic
Approach to
System
Improvement
P
A S
D

30. Results overall for
specified area of
performance
cic.uts.edu.au
PERIMETA DASHBOARD FEATURES
Input data – one node for each
Respondent and each Question
Results compiled for
each Question
Results compiled for
each Gender
Results compiled for
each Academy

31. PERIMETA MODELS FROM DIFFERENT
EVIDENCE STRANDS
cic.uts.edu.au
S7: Learning
of leaders
(SLT)
S4: Learning
of students in
Year 11
S6: Learning
of teachers
S5: Quality of
relationship w.
parents/ carers

33. cic.uts.edu.au
PERIMETA MODEL RESULTS

34. DRILLING DOWN INTO THE EVIDENCE TO
UNDERSTAND KEY ISSUES
cic.uts.edu.au
Outcome Establishing and sustaining learning communities
Outputs Developing the learning of students so that they realise their full potential
Inputs Feeling valued and
included in our
community
Feeling part of a
community focused on
learning & achievement
Statement POS
I was helped and supported when I struggled 0.98
My teachers taught me well 0.90
The Academy made good use of technologies 0.90
I was valued and included 0.89
I was often challenged to do better 0.89
There were great facilities 0.89
I felt part of a learning community 0.89
I developed responsibility towards others 0.89
I had a wide range of opportunities for learning 0.89
I felt safe 0.89
Knowing that I am
becoming an effective
learner
Knowing that we are
being helped to reach
our full potential
Statement UNC
I supported activities that would improve things
0.38
for other people
I led activities 0.34
I am looking forward to the move to Sixth Form,
0.33
FE, training or work
My parents/carers felt involved 0.15
Everyone worked very hard to make it a great
0.15
place to learn and to do well
I enjoyed my time 0.15
I could not have gone to a better school 0.14
I was offered opportunities for leadership 0.14
I never felt that anyone was putting limits on
0.13
what I could achieve
My teachers accurately predicted my GCSEs 0.13
Statement NEG
I am looking forward to the move to Sixth
0.18
Form, Further Education, training or work
I led activities during my time at the Academy 0.17
No others above 10%

35. THE PERIMETA TOOLKIT SUPPORTS
COLLABORATIVE ACTION TO
IMPROVE LEARNING
Focusing on
PERFORMANCE vs
broad outcomes
Understanding
COMPLEX
PROCESSES
Weighing up
uncertainty of
EVIDENCE
Informing robust
CHOICES
Enabling robust
& collaborative
ACTION
KEY FEATURE:
“Italian Flag”
evidence model &
rich uncertainty
calculus
KEY FEATURE:
Excel interface for
linked data and
predictive models
KEY
FEATURE:
Hierarchy of
interactions
focused on
outcomes

36. So how do we respond to
what Perimeta is telling
us as a learning
community?

37. 37
Define purpose
Analyze system
Design Model
Design measurement
parameters for
evaluating success
Collect data
according to
measurement model
over time
Implement
prototypes and
Enter into HPM
model and analyse
performance
Explore solutions as
prototypes
Feedback to system
leaders
evaluate
Hierarchical
Process Modeling
for a Holistic
Approach to
System
Improvement
P
A S
D

38. Professional Enquiry through Rapid Prototyping
to address key challenges highlighted by
Perimeta….
Plan
Do
Act Study
…..within a disciplined evaluation framework

39. Measurable System Improvement: New Knowledge and Know
How
P D
S A
P D
S A
P D
S A
P D
S A
P D
S A
Experiential knowing, hunches, ideas
P D
S A
P D
S A
P D
S A
P D
S A
P D
S A
P D
S A
P D
S A
Bryk, A. Gomez L. Gunrow A. (2010)
Getting Ideas into Action: Building
Networked
Improvement Communities in Education,
Carnegie Foundation for the
Advancement of Teacher Education,
Carnegie, Stanford.

40. How can we use
technology to enhance
our ability to develop
organisational learning
through Perimeta

41. cic.uts.edu.au
AN ACADEMY WITH SYSTEMIC
ANALYTICS
Strategic Progress
Learner
Dispositions
Academy X
Teacher
Dispositions
Stories of Change
& Evidence
Reflective Enquiry
Blogs

42. cic.uts.edu.au
AN ACADEMY NETWORK WITH
SYSTEMIC ANALYTICS
42
Evidence Hub
Data
Data
Data
Data
Analytics
Integration

43. SYSTEMS LEARNING & LEADERSHIP EVIDENCE
HUB
HTTP://SYSLL.EVIDENCE-HUB.NET
cic.uts.edu.au
Share
and
debate
evidence…
About…

44. cic.uts.edu.au
SYSTEMS LEARNING & LEADERSHIP
EVIDENCE HUB
HTTP://SYSLL.EVIDENCE-HUB.NET
44
Replay Oasis/Univ Bristol/Open Uni event: http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning

45. cic.uts.edu.au
LEARNING ANALYTICS HUB (3.0)
Online Tools designed to measure core
processes and to
stimulate self-directed change
Learning Analytics
System Simulations
Recommendation Engines
Social Learning Environments
Data Sets + Data Streams
Buckingham Shum, S. & Deakin Crick, R. (2012) Learning Dispositions and Transferable Competencies: Pedagogy, Modelling, and Learning Analytics,
Proceedings of LAK12: 2nd International Conference on Learning Analytics & Knowledge, April 29 - May 2, 2012, Vancouver, BC. ACM: New York. ISBN
978-1-4503-1111-3.