Learning Analytics: moving from potentially, to genuinely useful. Topics covered • Developing the NTU Student Dashboard • Research into the accuracy of the Dashboard • Student & staff feedback • Moving from potential to really useful

1. Learning Analytics: moving from potentially, to genuinely useful
ABLE Project 2015-1-BE-EPPKA3-PI-FORWARD

2. Topics covered
• Developing the NTU Student Dashboard
• Research into the accuracy of the Dashboard
• Student & staff feedback
• Moving from potential to really useful

3. What is the NTU Student Dashboard?

4. What does the Dashboard do?
NTU
Student
Dashboard
Student biographical
info, e.g. enrolment
status
Evidence of student
engagement
• Door swipes
(where appropriate)
• Library books
• NOW use
• Dropbox
submissions
• In 2015-16
• Attendance data
• Access to e-
books and
journals through
Shibboleth
authentication
Staff
view
Student
view
Compares student
engagement across
the cohort & gives
rating
Can make
comments
in free text
box
Raises
alerts!!

5. Why NTU is interested in learning analytics?
• Reduce barriers to being known by a tutor
• Space for tutors to make notes & plan
• Students as agents
• Can see own grades, and compare self to
peers
• Feedback
• Tool for academic use
• Potentially promotes virtuous behaviour
• Alerts tutors if students are high risk for
leaving early
• Students can compare their engagement
with their peers
Attainment
Progression
Belonging
Strategic
information
• Insights into groups at risk
• Potentially design and delivery of
courses

6. Learning Analytics Cycle (Clow, 2012)

7. The NTU Student Dashboard
• The Dashboard monitors students’ engagement with their course
– Door swipes, Library use, log ins to the VLE & submission through NOW Drop
box
• It compares this data to a profile from previous years and assigns
the four ratings:
– High
– Good
– Partial
– Low
– Not fully enrolled (for students who are only eligible to enroll at the start of the
year, or who have withdrawn)
• Students and staff can see exactly the same view
– Staff have a few additional management screens
• Tutors can also make notes in the notes section
• Staff are also sent an alert if there is no engagement for two weeks
during term time

8. Developmental Cycle
• Pilot Phase (2013-14)
– 4 courses, 40 staff & 500 1st year students
– Willing participants – very positive staff feedback, limited student awareness
• Phase 1 (2014-15)
– Launched in 8 of the 9 Schools, briefings for all Schools
– Developed formal governance structure and tied into academic committee
structure
– Lots of problem-solving
– Ethical policy written
• Phase 2 (Sept 2015-February 2016)
– New data sources added
– Launched in all Schools
– Assessment view
• Phase 3 (now)
– Personalisation, embedding into University systems

9. Landing page

11. Engagement for
past 5 days
Explanation of
engagement
ratings
Advice about
what to do to
improve
engagement

13. Helping tutors out
BTEC Qual
Does the student have a
BTEC as part of their entry
qualification?
Engagement alerts
Engagement alerts are generated after 2
weeks of no engagement during term time
& indication of student at risk

14. Induction trial
Designed to give
lecturers information for
early tutorials and to help
students reflect back
later in the year

15. • This data is drawn
automatically from
the attendance app
• The wheel shows
overall attendance
• It is therefore likely
that there will
appear to be a
discrepancy
between weekly
and overall
attendance,
particularly later in
the year
• As the data is
presented the next
day, it is important
to keep attendance
data up to date

16. Electronic Resources

17. Assessment & Feedback View
This view will only show assessments and feedback
submitted through NOW

18. Notes & Referrals
• From the start of the
year, all staff will be
able to refer students
to the Academic
Librarians from the
New Referral drop
down menu
• It will open up a new
window and will
require some
inputting, but we aim
to make this a more
integrated process
• NBS staff will be able
to refer students to
Student Support
Services as a pilot.
We will review and
extend this based on
feedback.

19. Research into the Dashboard’s accuracy and use

20. Increase in staff log ins
2014/15 - 2015/16
1,528
16910
2,056
24473
0
5,000
10,000
15,000
20,000
25,000
Number of staff who logged in Total number of staff log-ins
2014-15 2015-16
2014/15 – average 11 log-ins per staff member
2016/16 – average 12 log-ins per staff member

21. Increase in student log ins
2014/15 - 2015/16
15734
133613
25893
340018
0
50000
100000
150000
200000
250000
300000
350000
Number of students who logged in Total number of student log-ins
2014-15 2015-16
2014/15 – average 8 log-ins per student
2016/16 – average 13 log-ins per student

22. Feedback from the Student Transition Survey
• The transition survey was conducted with first year students (Feb/March 2016)
• 91% reported using the Dashboard at least once (90% in 2015)
Have you logged into the NTU student Dashboard?
When using the Dashboard, how often have you explored the following?
Base: 515 (2016), 469 (2015)
4%
5%
14%
15%
13%
24%
5%
6%
26%
24%
35%
36%
12%
13%
33%
34%
34%
36%
79%
77%
28%
28%
19%
5%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Spoke to someone providing specialist help (for
example student support services/ library) as a result
of looking at information on the Dashboard
Spoke to your tutor as a result of looking at
information on the Dashboard
Changed your behaviour to raise or maintain your
engagement score (for example made sure that you
swiped to go into a building)
Compared your engagement score with other students
on your course
Increased the amount of time you spend studying
Checked your own engagement score
Very Often Often Sometimes Never

23. Research into the
Dashboard’s accuracy
2014-15 data

24. Relationship between average engagement & progression
1st year students only Sept 2014 –June 2015
Low engagement Partial engagement Good engagement High engagement
Other (n=48) 1.1% 0.8% 0.5% 0.1%
Withdrawn (n=368) 26.0% 5.0% 3.2% 1.4%
Transfer (n=54) 0.7% 1.0% 0.3% 0.2%
Repeating (n=348) 9.4% 5.9% 2.8% 1.3%
Academic failure 47.3% 5.0% 0.7% 0.8%
Progressed 15.5% 82.3% 92.6% 96.2%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Progression status by enegagement rating (year 1 full time UG students)
Progressed Academic failure Repeating (n=348) Transfer (n=54) Withdrawn (n=368) Other (n=48)

25. Validity of ‘no engagement’ alerts

26. 28-21 days 21-14 days 14-7 days 7-1 days 1 day
4 4 4 4 4
3 4 4 4 4
3 4 4 4 4
4 3 4 4 4
3 3 4 4 4
4 4 3 4 4
4 4 3 4 4
3 3 3 4 4
3 3 3 4 4
2 2 3 4 4
2 2 3 4 4
3 4 4
2 2 3 3 4
2 3 3 4
2 2 2 3 4
2 3 4
1 3 4
2 3 2 2 4
4 4 4 4 3
3 4 3 4 3
3 3 3 4 3
2 3 3 4 3
2 2 4 3
4 4 4 3 3
3 4 4 3 3
4 3 4 3 3
3 3 4 3 3
3 3 4 3 3
2 3 4 3 3
2 3 4 3 3
1 3 4 3 3
2 4 3 3
4 4 3 3 3
4 4 3 3 3
4 4 3 3 3
3 4 3 3 3
3 4 3 3 3
3 4 3 3 3
4 3 3 3 3
4 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
2 3 3 3 3
3 2 3 3 3
2 2 3 3 3
2 2 3 3 3
2 2 3 3 3
3 1 3 3 3
4 3 2 3 3
4 3 2 3 3
3 3 2 3 3
3 3 2 3 3
3 3 2 3 3
3 3 2 3 3
2 3 2 3 3
4 2 2 3 3
3 2 2 3 3
3 2 2 3 3
3 2 2 3 3
2 2 2 3 3
2 2 2 3 3
2 2 2 3 3
2 3 3
2 3 3
1 1 1 3 3
3 4 3 2 3
3 3 3 2 3
3 3 3 2 3
4 2 3 2 3
2 2 3 2 3
2 2 3 2 3
2 4 2 2 3
2 3 2 2 3
4 2 2 2 3
3 2 2 2 3
2 2 2 2 3
2 2 2 2 3
2 1 2 2 3
1 1 2 2 3
2 2 3
2 2 3
3 4 3 4 2
3 3 3 3 2
3 3 3 3 2
3 2 3 3 2
2 2 3 3 2
2 3 3 2
1 1 3 3 2
3 3 2
4 3 2 3 2
3 2 3 2
4 2 2 3 2
3 2 2 3 2
2 2 2 3 2
2 2 2 3 2
2 2 3 2
2 1 2 3 2
2 3 2
4 4 4 2 2
2 4 4 2 2
3 3 4 2 2
3 3 4 2 2
4 2 4 2 2
3 2 4 2 2
3 2 4 2 2
2 2 4 2 2
2 2 4 2 2
2 4 2 2
4 4 3 2 2
3 4 3 2 2
3 4 3 2 2
4 3 3 2 2
3 3 3 2 2
3 3 3 2 2
3 3 3 2 2
3 3 3 2 2
3 3 3 2 2
3 3 3 2 2
3 3 3 2 2
2 3 3 2 2
2 3 3 2 2
1 3 3 2 2
3 2 3 2 2
3 2 3 2 2
3 2 3 2 2
3 2 3 2 2
Timeframe before withdrawal
28-21 days 21-14 days 14-7 days 7-1 days 1 day
2 2 3 2 2
2 2 3 2 2
2 2 3 2 2
2 2 3 2 2
2 2 3 2 2
2 2 3 2 2
2 2 3 2 2
2 2 3 2 2
2 2 3 2 2
2 3 2 2
2 3 2 2
2 3 2 2
1 1 3 2 2
4 4 2 2 2
3 4 2 2 2
3 4 2 2 2
3 4 2 2 2
2 4 2 2 2
2 4 2 2 2
2 4 2 2 2
4 3 2 2 2
4 3 2 2 2
3 3 2 2 2
3 3 2 2 2
3 3 2 2 2
3 3 2 2 2
3 3 2 2 2
3 3 2 2 2
3 3 2 2 2
3 3 2 2 2
2 3 2 2 2
2 3 2 2 2
2 3 2 2 2
2 3 2 2 2
2 3 2 2 2
2 3 2 2 2
3 2 2 2
4 2 2 2 2
3 2 2 2 2
3 2 2 2 2
3 2 2 2 2
3 2 2 2 2
3 2 2 2 2
3 2 2 2 2
3 2 2 2 2
3 2 2 2 2
3 2 2 2 2
3 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
2 2 2 2 2
1 2 2 2 2
1 2 2 2 2
1 2 2 2 2
1 2 2 2 2
2 2 2 2
2 2 2 2
2 1 2 2 2
2 1 2 2 2
2 1 2 2 2
2 1 2 2 2
2 1 2 2 2
2 1 2 2 2
2 1 2 2 2
1 1 2 2 2
1 1 2 2 2
1 1 2 2 2
1 1 2 2 2
1 2 2 2
2 2 2
2 2 2
2 2 2
2 2 2
2 2 2
4 3 1 2 2
4 3 1 2 2
2 2 1 2 2
2 2 1 2 2
2 2 1 2 2
2 2 1 2 2
2 2 1 2 2
2 2 1 2 2
2 1 1 2 2
1 1 1 2 2
1 1 1 2 2
1 1 1 2 2
1 1 2 2
1 1 2 2
2 2
2 2
2 2
2 2 2 1 2
2 1 2 1 2
1 2 1 1 2
3 1 1 1 2
2 1 1 1 2
2 1 1 1 2
1 1 1 1 2
1 1 1 1 2
1 1 1 1 2
1 1 1 1 2
1 1 1 1 2
2 1 3 3 1
Timeframe before withdrawal
28-21 days 21-14 days 14-7 days 7-1 days 1 day
3 3 3 2 1
3 3 2 2 1
3 2 2 2 1
2 2 2 2 1
2 2 2 2 1
2 2 2 2 1
2 2 2 2 1
2 2 2 2 1
2 2 2 2 1
2 2 2 2 1
2 2 2 2 1
1 2 2 2 1
2 2 2 1
2 1 2 2 1
2 1 2 2 1
2 2 1
2 2 1 2 1
2 2 1 2 1
2 2 1 2 1
1 2 1 2 1
2 1 1 2 1
2 1 1 2 1
2 1 1 2 1
1 1 1 2 1
2 1
3 3 4 1 1
3 3 1 1
2 2 3 1 1
2 2 3 1 1
4 4 2 1 1
2 3 2 1 1
3 2 2 1 1
3 2 2 1 1
2 2 2 1 1
2 2 2 1 1
2 2 2 1 1
2 2 2 1 1
2 2 2 1 1
2 2 2 1 1
2 2 2 1 1
2 2 2 1 1
1 2 2 1 1
1 2 2 1 1
1 2 2 1 1
1 2 2 1 1
1 2 2 1 1
2 1 2 1 1
1 1 2 1 1
1 1 2 1 1
3 3 1 1 1
3 2 1 1 1
3 2 1 1 1
2 2 1 1 1
2 2 1 1 1
2 2 1 1 1
2 2 1 1 1
2 2 1 1 1
2 2 1 1 1
2 2 1 1 1
2 2 1 1 1
2 2 1 1 1
2 2 1 1 1
1 2 1 1 1
1 2 1 1 1
2 1 1 1 1
2 1 1 1 1
2 1 1 1 1
2 1 1 1 1
2 1 1 1 1
2 1 1 1 1
2 1 1 1 1
2 1 1 1 1
2 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1
1 1 1
1 1 1
1 1
Timeframe before withdrawal
Key: Low Engagement Partial Engagement Good Engagement High Engagement Student withdrew before timeframe
Student engagement in the month prior to
departure
NB Each row represents one student in the month prior to departure

27. Dashboard log ins by engagement classification
(mode for days enrolled) 1st yr., FT, UG students

28. Relationship between engagement at key
times of year and progression

29. Staff Feedback

30. Staff feedback
How useful have you found the Dashboard?
Not at all useful
4%
Not very useful
14%
Undecided
21%
Useful
48%
Very useful
13%
* n=160 from all staff with access to the Dashboard less the 20 who had not used the dashboard = 140 responses

31. Yes about 1-2
times
Yes about
monthly
Yes about weekly
Yes whenever I’ve
had an alert email
very useful 2 1 11 4
quite useful 7 26 27 8
undecided 13 11 1 4
Not very useful 7 9 2 2
Not at all useful 1 2 2
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%AxisTitle
Relationship between Dashboard use and
perceptions of usefulness (n=140) Survey
conducted summer 2015

32. Lessons from the Dashboard Implementation

33. Ethics
• Student perceptions about any data kept on them
• Data protection awareness paradox
• Limitations, particularly sharing
• Sophisticated stereotyping
• Does this lead to better student outcomes or more stressed students?
• Staff concerns – hidden agenda/ loss of autonomy
• We have written guidance about how we will use the Dashboard
– (pg 9)
• Have an ethics group and ethics research group
• Need more work on student communications
• Open University has good (if lengthy) guidance

34. What have we achieved?
• We have a working learning analytics resource
• Following a successful pilot, it has been implemented across the
whole University
– Continue to be low level teething problems
• Start dates
• Identifying personal tutors
• We are clear that the analytics works
– More work needed on part-time student engagement
• Positive feedback from staff & students

35. The nub of the issue for me
• Learning analytics is only as useful as the actions it instigates
For the academic to
jump into the bath
tub, they need
time, training,
motivation, to
have easy access
to data, a room to
talk to the student
or space to email
them.
Just knowing that
they need to jump
is only the first link
in the chain

36. The sector already has enough data to act,
but appears not to have done so
ECU
report
from
2010 –
evidence of
attainment gap
comfortably 10 years
old – 3 complete
generations of
University students

37. IT infrastructure, management processes & quality
New data sources for the learning analytics model
Further Dashboard developments: NECs, design, use of notes etc.
Communications
Integrating Dashboard into institutional working practices
Stuff we haven’t considered yet

38. Some of our key lessons
• It’s a team game
• Philosophical opposition appears stronger amongst staff, not
students
– Although academics can see the benefits more readily as they appear to have
terrible access to student data
• Whatever you’ve budgeted for communications, double it
• There are two products: the dashboard and the analytics
– The quick wins are as much about the dashboard as they are about the
analytics
• The miraculous quickly becomes the mundane
– Trivial irritants become big problems
• This is disruptive technology
– The dashboard highlights existing problems with systems
– Embedding/ competing/ duplicating

39. Three questions
• I think that in order to be genuinely useful there are three questions
you need to ask of learning analytics
• Can your system actually spot those students who need help?
– Can it spot them early enough?
• Do you have a system to engage with students you have concerns
about?
– Do students respond to it?
• Can learning analytics support interventions that lead to sustained
changes in student behaviour?