This paper reports on the efficacy of a mobile learning intervention that combined ‘push notifications’ and game principles within a timed quiz app. An institutional interdisciplinary case study was conducted which compared rates of student retention and academic performance with their usage of a purpose-designed learning app. Leading up to lectures the app ‘pushed’ daily quizzes to students’ personal mobile devices and then rewarded them with feedback, points, badges and a position on a leaderboard. It was found that since the introduction of the app there was an increase in student retention rate of 12.23%, an increase in academic performance of 7.03% and a significant positive correlation of .40 between students’ scoring highly on the app and achieving higher academic grades. Conclusions are made in regards to what these findings mean for the future research into higher education learning enabled via mobile app technologies. More broadly, we discuss the implications of our findings in regards to the key higher education stakeholders: universities, educators and students.
Push notifications, digital badges & leaderboards: Evaluating the impact of Quitch mobile application on learning
1. CRICOS 00111D TOID 3059
Push notifications, digital badges &
leaderboards: Evaluating the impact of
Quitch mobile application on learning
Dr Grainne Oates FLB, Senior Lecturer, Accounting/Finance
Dr Katya Pechenkina LTU, Research Fellow
Dan Laurence LTU (formerly), Learning Designer
Dr Daniel Eldridge FSET, Lecturer in Chemistry
Prof Dan Hunter FBL, Foundation Dean, Swinburne Law
Presented at Transforming Learning Conference
Swinburne University of Technology
13-14 September 2016
Melbourne Australia
http://transformconference.com/ #SwinTLC
2. Outline
• Mobile apps for learning
• Quitch: Rationale & elements
• Quitch: Development process
• Measuring impact
• Findings
• Further steps
3. Mobile apps for learning
• Challenges: low retention & high failure rates
• Students likely to access content via mobile
phone (or a similar device) not desktop computer
• Push notifications
• Student expect immediate constructive feedback
• Self-testing has capacity to improve outcomes
• Multitude of mobile apps - rarely tailored to
lecture content
4. Quitch: Rationale & elements
• Engagement loops
• Regular ‘push-notifications’
• Dynamic timed quiz
• Points & badges
• Leaderboards
• Progression trees
• Videos & forums
A solution should:
• Be easy for educators to use
• Have real time analytics for early intervention &
adaptive learning
8. Quitch: Analytics
Possibilities:
• Reengage ‘at risk’ students
• Test if lecture content is comprehended
• Adapt course material dynamically
• Provide regular personalised feedback
9. Measuring impact
• Is there a relationship between introducing a
mobile app into the classroom and student
retention rates?
• Is there a relationship between the use of
the app and academic performance?
• Is student performance within the app
related to academic performance?
• Do students respond positively to the
introduction of the app?
10. Measuring impact: Methods
• Two cohorts taking part in the Quitch trial
(S2, 2015)
Accounting (462 enrolled, 265 app users)
Science (249 enrolled, 129 app users)
• Total sample: N=711, with 55% using the
app (394) + prior cohorts for comparison
• Data collected (with student consent):
Final grades compared to the pre-app cohort
App uptake & engagement analytics
Small-scale qualitative online survey (N=8)
Student feedback from end-of-year surveys
11. Findings
Is there a relationship between introducing a
mobile app into the classroom and student
retention rates?
• The app introduction was associated with a
positive improvement in student retention
(calculated at 12.23%) compared to the pre-
app semester
12. Findings
Is there a relationship between the use of the
app and academic performance?
• App users averaged a percentage mark of
65.19% compared to students who did not
use the app (who averaged a percentage
mark of 58.16%).
• App users on average achieved marks that
were 7.03% higher than students who did not
use the app
13. Findings
Is student performance within the app related to
academic performance?
• For app user there was a positive correlation
of .40 between performing well on the app
and achieving higher academic grades
14. Findings
Do students respond positively to the introduction of the
app?
The app “reinforced the content” & “kept [students]
involved and invested” by appealing to their competitive
nature
Leaderboards & badges: allowed students to see how they
placed compared to their peers. However: it is “dangerous
to start ranking people” & some students felt “exposed”
Immediate feedback was desired in addition to right/wrong
answer indicator
15. Further steps
• Correlation ≠ causation
• In-depth studies of student &
educator experiences needed
• Implications for universities,
educators & students
• Tailored content, timed
content-aligned quizzes,
immediate feedback, rankings,
flexibility
16. Thank you
Presented by Dr Katya Pechenkina
Research Fellow
The Office of the Senior DVC & Provost
Learning Transformations
Email: epechenkina@swin.edu.au
Twitter: @katya_pechenk
