SlideShare una empresa de Scribd logo
1 de 29
Research into Practice
Building and implementing
learning analytics atTribal
Chris Ballard, Data Scientist
Building and implementing learning analytics
1. Start at the very beginning
2. How research and practice differ
3. Building a learning analytics platform
4. Implementing learning analytics
5. Summary
A leading provider of technology-enabled management solutions
for the international education, learning and training markets
Higher
Education
Vocational
Learning
Schools /
K-12
Research universities
Employment-focused
universities
Government agencies
Further education colleges
Training providers
and employers
Government agencies
Schools
School groups
State and district
government agencies
AboutTribal
3
Objectives
• Predict student academic
performance to optimise success
• Predict students at risk of non-
continuation
• Build on research into link
betweenVLE activity and
academic success
• Scale data processing
• Understand risk factors and
compare to cohorts
3 years of matched student and activity
data used to build predictive models
Staff can use student, engagement and
academic data to understand how they
affect student outcomes.
Information accessible in one place on
easy to understand dashboards.
Integrated withTribal SITS:Vision and
staff e:vision portal
Consultation with academic staff on
presentation and design
Accuracy of module academic
performance predictions* 79%
*Using module academic history and demographic factors
R&D project overview
Student Insight
Current projects
Providing learning analytics for 160,000
students across a state-wide vocational and
further education provider inAustralia
Student Insight being implemented
as part of the JISC UK Effective
Learning Analytics programme
What do we mean by practice?
From research to practice
Research
• Domain knowledge
• Interpretation
• In depth understanding
• Testing an approach
Practice
• Integrated into everyday life
• Interpret easily
• Take action
• Implementing an approach
Domain and people
• What is the problem?
• Identify the users and stakeholders
• Data owners
• Are research results sufficient?
• Design
• Project cost
Technical
• Research limitations
• Architecture
• Data munging
• Automating manual processes
• Data suitability
• Robustness of technical platform
Building a learning analytics platform
Key design decisions
1. Transparency – knowing why a student is a risk
2. Flexibility – viewing learning analytics which relates to an
institution's curriculum and organisation
3. Efficiency – ease of use, implementation and interpretation
Information relevant for different users
Aggregating warning indicators
Aggregating warning indicators
Transparency – individual risk
Ensemble
Decision
Combination
Enrolment
Academic
Performance
Engagement
Historic module results
VLE Event Data
Library Event Data
Attendance
VLE
Library
Attendance
%
%
%
%
%
%
%
Demographics
Risk
Prediction
%
Module
History
AssessmentsFormative Assessments
Datasets
Ensemble learning
Student
Weight
Weight
Weight
Weight
Weight
Weight
Transparency – risk factors
Providing flexibility to an institution
Reflecting differences between courses
Student activity data is not consistent across all courses/modules
1. Standardise data so it is comparative across all courses and
modules
2. Build different models for each course or module
Need to be careful that you have sufficient data for the model to
generalise to new data.
Provide opportunity for intervention
Identify student at risk
Log intervention details
Assess intervention
effectiveness
Learning analytics
Allocates intervention to
support team
Assign SLA
SLA based alerts
Monitor intervention
progress
Student support teams
Embed into business process
Need to consider how learning analytics becomes embedded into
the day to day working life of academic and support staff.
 Notifications – analytics becomes proactive; support different
types of notification
 Integration – accessible from existing tools and services through
single sign on
Implementing learning analytics
CRISP-DM – Cross Industry Standard Process for Data Mining
https://the-modeling-agency.com/crisp-dm.pdf
CRISP-DM process
Data understanding
Understanding which features are important
Example: end month of unit for successful and failed units
Data preparation
Creating comparative features
Example:Total proportion of hours worked on failed units
Modelling and evaluation
Understanding whether model is under- or over- fitting
Example: Learning curve for Random Forest model
Evaluation
 Define business focused success criteria
 Define model focused success criteria
 Define what baseline performance is acceptable
 Consider a model cost benefit analysis that takes into account
intervention cost
Actual
Withdrawn Enrolled
Predicted
Withdrawn benefit cost
Enrolled cost benefit
Summary
Design
 Embed learning analytics into business process
 Ensure that analytics can be interpreted easily by staff
 Intervention processes that are clearly articulated
 Measure intervention effectiveness
Implementation
 Use a standard project approach such as CRISP-DM
 Evaluate data in the context of the business problem and
process
 Define what success means, including acceptable accuracy and
how it needs to be measured
Thank you
Chris Ballard
@chrisaballard
chris.ballard@tribalgroup.com
www.tribalgroup.com

Más contenido relacionado

La actualidad más candente

Overview of CETL programme evaluation
Overview of CETL programme evaluationOverview of CETL programme evaluation
Overview of CETL programme evaluationpetertknight
 
Analytics in Action - Education
Analytics in Action - EducationAnalytics in Action - Education
Analytics in Action - EducationLee Schlenker
 
Mainstreaming e-data collection in CIAT programs in Africa
Mainstreaming e-data collection in CIAT programs in AfricaMainstreaming e-data collection in CIAT programs in Africa
Mainstreaming e-data collection in CIAT programs in AfricaCIAT
 
Using Naviance Data to Drive College & Career Readiness
Using Naviance Data to Drive College & Career ReadinessUsing Naviance Data to Drive College & Career Readiness
Using Naviance Data to Drive College & Career ReadinessNaviance
 
Summary of EdTech Symposium EdTech Decision Making in Higher Ed for eLearn
Summary of EdTech Symposium EdTech Decision Making in Higher Ed for eLearnSummary of EdTech Symposium EdTech Decision Making in Higher Ed for eLearn
Summary of EdTech Symposium EdTech Decision Making in Higher Ed for eLearnDr. Kristin Palmer
 
Joel Njuguna Resume January 6 2016
Joel Njuguna Resume January 6 2016Joel Njuguna Resume January 6 2016
Joel Njuguna Resume January 6 2016Joel Njuguna
 
Aa group overview final
Aa group overview finalAa group overview final
Aa group overview finaleliwr
 
2001 wisconsin school boards association issues in building data and inform...
2001 wisconsin school boards association   issues in building data and inform...2001 wisconsin school boards association   issues in building data and inform...
2001 wisconsin school boards association issues in building data and inform...Christopher Thorn
 
Factors Affecting the Sustained Use of Process Models
Factors Affecting the Sustained Use of Process ModelsFactors Affecting the Sustained Use of Process Models
Factors Affecting the Sustained Use of Process ModelsMarlon Dumas
 
Virtual Property Manager: Providing a Simulated Learning Environment in a New...
Virtual Property Manager: Providing a Simulated Learning Environment in a New...Virtual Property Manager: Providing a Simulated Learning Environment in a New...
Virtual Property Manager: Providing a Simulated Learning Environment in a New...Andy Carswell
 
Spring 2011 Signature Experience Projects Kick-Off
Spring 2011 Signature Experience Projects Kick-OffSpring 2011 Signature Experience Projects Kick-Off
Spring 2011 Signature Experience Projects Kick-OffTXWes_PMO
 
Resume yixiong shang
Resume yixiong shangResume yixiong shang
Resume yixiong shangYixiongShang
 

La actualidad más candente (20)

Overview of CETL programme evaluation
Overview of CETL programme evaluationOverview of CETL programme evaluation
Overview of CETL programme evaluation
 
Analytics in Action - Education
Analytics in Action - EducationAnalytics in Action - Education
Analytics in Action - Education
 
Mainstreaming e-data collection in CIAT programs in Africa
Mainstreaming e-data collection in CIAT programs in AfricaMainstreaming e-data collection in CIAT programs in Africa
Mainstreaming e-data collection in CIAT programs in Africa
 
Academic Support and Technology
Academic Support and TechnologyAcademic Support and Technology
Academic Support and Technology
 
Facets of Educational Technology
Facets of Educational TechnologyFacets of Educational Technology
Facets of Educational Technology
 
Using Naviance Data to Drive College & Career Readiness
Using Naviance Data to Drive College & Career ReadinessUsing Naviance Data to Drive College & Career Readiness
Using Naviance Data to Drive College & Career Readiness
 
Summary of EdTech Symposium EdTech Decision Making in Higher Ed for eLearn
Summary of EdTech Symposium EdTech Decision Making in Higher Ed for eLearnSummary of EdTech Symposium EdTech Decision Making in Higher Ed for eLearn
Summary of EdTech Symposium EdTech Decision Making in Higher Ed for eLearn
 
Joel Njuguna Resume January 6 2016
Joel Njuguna Resume January 6 2016Joel Njuguna Resume January 6 2016
Joel Njuguna Resume January 6 2016
 
Aa group overview final
Aa group overview finalAa group overview final
Aa group overview final
 
Data informed decision making - Yaz El Hakim
Data informed decision making - Yaz El HakimData informed decision making - Yaz El Hakim
Data informed decision making - Yaz El Hakim
 
2001 wisconsin school boards association issues in building data and inform...
2001 wisconsin school boards association   issues in building data and inform...2001 wisconsin school boards association   issues in building data and inform...
2001 wisconsin school boards association issues in building data and inform...
 
Ba education
Ba educationBa education
Ba education
 
Factors Affecting the Sustained Use of Process Models
Factors Affecting the Sustained Use of Process ModelsFactors Affecting the Sustained Use of Process Models
Factors Affecting the Sustained Use of Process Models
 
Learning Analytics
Learning AnalyticsLearning Analytics
Learning Analytics
 
Virtual Property Manager: Providing a Simulated Learning Environment in a New...
Virtual Property Manager: Providing a Simulated Learning Environment in a New...Virtual Property Manager: Providing a Simulated Learning Environment in a New...
Virtual Property Manager: Providing a Simulated Learning Environment in a New...
 
Ed psy 510 5th class 2014
Ed psy 510 5th class 2014Ed psy 510 5th class 2014
Ed psy 510 5th class 2014
 
NISO Apr 29 Virtual Conference: Assessing Game-Based Library Initiatives Kyle...
NISO Apr 29 Virtual Conference: Assessing Game-Based Library Initiatives Kyle...NISO Apr 29 Virtual Conference: Assessing Game-Based Library Initiatives Kyle...
NISO Apr 29 Virtual Conference: Assessing Game-Based Library Initiatives Kyle...
 
NISO Apr 29 Virtual Conference: ‘Good Enough’: Applying a Holistic Approach f...
NISO Apr 29 Virtual Conference: ‘Good Enough’: Applying a Holistic Approach f...NISO Apr 29 Virtual Conference: ‘Good Enough’: Applying a Holistic Approach f...
NISO Apr 29 Virtual Conference: ‘Good Enough’: Applying a Holistic Approach f...
 
Spring 2011 Signature Experience Projects Kick-Off
Spring 2011 Signature Experience Projects Kick-OffSpring 2011 Signature Experience Projects Kick-Off
Spring 2011 Signature Experience Projects Kick-Off
 
Resume yixiong shang
Resume yixiong shangResume yixiong shang
Resume yixiong shang
 

Similar a Implementing Learning Analytics

Botor_project_research_methodology_2016
Botor_project_research_methodology_2016Botor_project_research_methodology_2016
Botor_project_research_methodology_2016Shayne Botor
 
Exploring learning analytics
Exploring learning analyticsExploring learning analytics
Exploring learning analyticsJisc
 
Jisc learning analytics service overview Aug 2018
Jisc learning analytics service overview Aug 2018Jisc learning analytics service overview Aug 2018
Jisc learning analytics service overview Aug 2018Paul Bailey
 
E12+Analytics+Workshop+ppt.pptx
E12+Analytics+Workshop+ppt.pptxE12+Analytics+Workshop+ppt.pptx
E12+Analytics+Workshop+ppt.pptxchatbot9
 
Overview of Effective Learning Analytics Using data and analytics to support ...
Overview of Effective Learning Analytics Using data and analytics to support ...Overview of Effective Learning Analytics Using data and analytics to support ...
Overview of Effective Learning Analytics Using data and analytics to support ...Bart Rienties
 
Learning Analytics
Learning AnalyticsLearning Analytics
Learning AnalyticsJames Little
 
How can a data scientist expert solve real world problems?
How can a data scientist expert solve real world problems? How can a data scientist expert solve real world problems?
How can a data scientist expert solve real world problems? priyanka rajput
 
Jisc learning analytics scotland HEIDS
Jisc learning analytics scotland HEIDSJisc learning analytics scotland HEIDS
Jisc learning analytics scotland HEIDSPaul Bailey
 
Predictive Analytics in Education Context
Predictive Analytics in Education ContextPredictive Analytics in Education Context
Predictive Analytics in Education ContextIJMTST Journal
 
Advanced Analytics Certificate Flyer
Advanced Analytics Certificate FlyerAdvanced Analytics Certificate Flyer
Advanced Analytics Certificate Flyermisanthropic777
 
ABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyEd Foster
 
Wbl maturity toolkit meeting v1
Wbl maturity toolkit meeting v1Wbl maturity toolkit meeting v1
Wbl maturity toolkit meeting v1balham
 
Munassir etec647 e presentation
Munassir etec647 e presentationMunassir etec647 e presentation
Munassir etec647 e presentationMunassir Alhamami
 
Lightning Talks (select full talk per category)
Lightning Talks (select full talk per category)Lightning Talks (select full talk per category)
Lightning Talks (select full talk per category)Embedding Employability
 
FACT2 Learning Analytics Task Group (LATG) SCOA briefing
FACT2 Learning Analytics Task Group (LATG) SCOA briefingFACT2 Learning Analytics Task Group (LATG) SCOA briefing
FACT2 Learning Analytics Task Group (LATG) SCOA briefinggketcham
 
LERU Presentation - March 2017
LERU Presentation - March 2017LERU Presentation - March 2017
LERU Presentation - March 2017Anne-Marie Scott
 
EDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINALEDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINALEllen Wagner
 
Learning Analytics for Learning
Learning Analytics for LearningLearning Analytics for Learning
Learning Analytics for LearningWolfgang Greller
 
A Literature Survey on Student Profile Management System
A Literature Survey on Student Profile Management SystemA Literature Survey on Student Profile Management System
A Literature Survey on Student Profile Management SystemIRJET Journal
 
Discuss the requirements for developing a Security Education, Tranin.pdf
Discuss the requirements for developing a Security Education, Tranin.pdfDiscuss the requirements for developing a Security Education, Tranin.pdf
Discuss the requirements for developing a Security Education, Tranin.pdffabmallkochi
 

Similar a Implementing Learning Analytics (20)

Botor_project_research_methodology_2016
Botor_project_research_methodology_2016Botor_project_research_methodology_2016
Botor_project_research_methodology_2016
 
Exploring learning analytics
Exploring learning analyticsExploring learning analytics
Exploring learning analytics
 
Jisc learning analytics service overview Aug 2018
Jisc learning analytics service overview Aug 2018Jisc learning analytics service overview Aug 2018
Jisc learning analytics service overview Aug 2018
 
E12+Analytics+Workshop+ppt.pptx
E12+Analytics+Workshop+ppt.pptxE12+Analytics+Workshop+ppt.pptx
E12+Analytics+Workshop+ppt.pptx
 
Overview of Effective Learning Analytics Using data and analytics to support ...
Overview of Effective Learning Analytics Using data and analytics to support ...Overview of Effective Learning Analytics Using data and analytics to support ...
Overview of Effective Learning Analytics Using data and analytics to support ...
 
Learning Analytics
Learning AnalyticsLearning Analytics
Learning Analytics
 
How can a data scientist expert solve real world problems?
How can a data scientist expert solve real world problems? How can a data scientist expert solve real world problems?
How can a data scientist expert solve real world problems?
 
Jisc learning analytics scotland HEIDS
Jisc learning analytics scotland HEIDSJisc learning analytics scotland HEIDS
Jisc learning analytics scotland HEIDS
 
Predictive Analytics in Education Context
Predictive Analytics in Education ContextPredictive Analytics in Education Context
Predictive Analytics in Education Context
 
Advanced Analytics Certificate Flyer
Advanced Analytics Certificate FlyerAdvanced Analytics Certificate Flyer
Advanced Analytics Certificate Flyer
 
ABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of Derby
 
Wbl maturity toolkit meeting v1
Wbl maturity toolkit meeting v1Wbl maturity toolkit meeting v1
Wbl maturity toolkit meeting v1
 
Munassir etec647 e presentation
Munassir etec647 e presentationMunassir etec647 e presentation
Munassir etec647 e presentation
 
Lightning Talks (select full talk per category)
Lightning Talks (select full talk per category)Lightning Talks (select full talk per category)
Lightning Talks (select full talk per category)
 
FACT2 Learning Analytics Task Group (LATG) SCOA briefing
FACT2 Learning Analytics Task Group (LATG) SCOA briefingFACT2 Learning Analytics Task Group (LATG) SCOA briefing
FACT2 Learning Analytics Task Group (LATG) SCOA briefing
 
LERU Presentation - March 2017
LERU Presentation - March 2017LERU Presentation - March 2017
LERU Presentation - March 2017
 
EDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINALEDUCA Leveraging Analytics FINAL
EDUCA Leveraging Analytics FINAL
 
Learning Analytics for Learning
Learning Analytics for LearningLearning Analytics for Learning
Learning Analytics for Learning
 
A Literature Survey on Student Profile Management System
A Literature Survey on Student Profile Management SystemA Literature Survey on Student Profile Management System
A Literature Survey on Student Profile Management System
 
Discuss the requirements for developing a Security Education, Tranin.pdf
Discuss the requirements for developing a Security Education, Tranin.pdfDiscuss the requirements for developing a Security Education, Tranin.pdf
Discuss the requirements for developing a Security Education, Tranin.pdf
 

Más de LACE Project

SHEILA workshop at EC-TEL 2018
SHEILA workshop at EC-TEL 2018SHEILA workshop at EC-TEL 2018
SHEILA workshop at EC-TEL 2018LACE Project
 
SHEILA Results – Conference 5 June 2018
SHEILA Results – Conference 5 June 2018SHEILA Results – Conference 5 June 2018
SHEILA Results – Conference 5 June 2018LACE Project
 
SHEILA LAK18 Presentation
SHEILA LAK18 PresentationSHEILA LAK18 Presentation
SHEILA LAK18 PresentationLACE Project
 
SHEILA Project – LAK18 Workshop Slides
SHEILA Project – LAK18 Workshop SlidesSHEILA Project – LAK18 Workshop Slides
SHEILA Project – LAK18 Workshop SlidesLACE Project
 
SHEILA project LAK17 workshop slides
SHEILA project LAK17 workshop slidesSHEILA project LAK17 workshop slides
SHEILA project LAK17 workshop slidesLACE Project
 
SHEILA Project - Workshop Slides Online Educa Berlin 2016
SHEILA Project - Workshop Slides Online Educa Berlin 2016SHEILA Project - Workshop Slides Online Educa Berlin 2016
SHEILA Project - Workshop Slides Online Educa Berlin 2016LACE Project
 
Overview of the LAEP learning analytics project
Overview of the LAEP learning analytics projectOverview of the LAEP learning analytics project
Overview of the LAEP learning analytics projectLACE Project
 
EU-funded learning analytics projects
EU-funded learning analytics projectsEU-funded learning analytics projects
EU-funded learning analytics projectsLACE Project
 
Expert presentations: LAEP / LACE Amsterdam workshop
Expert presentations: LAEP / LACE Amsterdam workshopExpert presentations: LAEP / LACE Amsterdam workshop
Expert presentations: LAEP / LACE Amsterdam workshopLACE Project
 
Leveraging the full value of learning analytics: The power of learning design
Leveraging the full value of learning analytics: The power of learning designLeveraging the full value of learning analytics: The power of learning design
Leveraging the full value of learning analytics: The power of learning designLACE Project
 
Learning analytics LACE SoLAR Flare 2015
Learning analytics LACE SoLAR Flare 2015Learning analytics LACE SoLAR Flare 2015
Learning analytics LACE SoLAR Flare 2015LACE Project
 
Scalable Learning Analytics and Interoperability – an assessment of potential...
Scalable Learning Analytics and Interoperability – an assessment of potential...Scalable Learning Analytics and Interoperability – an assessment of potential...
Scalable Learning Analytics and Interoperability – an assessment of potential...LACE Project
 
Assessment Analytics - EUNIS 2015 E-Learning Task Force Workshop
Assessment Analytics - EUNIS 2015 E-Learning Task Force WorkshopAssessment Analytics - EUNIS 2015 E-Learning Task Force Workshop
Assessment Analytics - EUNIS 2015 E-Learning Task Force WorkshopLACE Project
 
LACE Spring Briefing - WatchMe project overview
LACE Spring Briefing - WatchMe project overviewLACE Spring Briefing - WatchMe project overview
LACE Spring Briefing - WatchMe project overviewLACE Project
 
LACE Spring Briefing - Lea's Box Project overview
LACE Spring Briefing - Lea's Box Project overviewLACE Spring Briefing - Lea's Box Project overview
LACE Spring Briefing - Lea's Box Project overviewLACE Project
 
LACE Spring Briefing - PELARS project overview
LACE Spring Briefing - PELARS project overview LACE Spring Briefing - PELARS project overview
LACE Spring Briefing - PELARS project overview LACE Project
 
LACE Spring Briefing - the LACE project
LACE Spring Briefing - the LACE project LACE Spring Briefing - the LACE project
LACE Spring Briefing - the LACE project LACE Project
 
LACE Spring Briefing - Learning analytics are more than measurement
LACE Spring Briefing - Learning analytics are more than measurementLACE Spring Briefing - Learning analytics are more than measurement
LACE Spring Briefing - Learning analytics are more than measurementLACE Project
 
Themes in Learning Analytics - A Critical View
Themes in Learning Analytics - A Critical ViewThemes in Learning Analytics - A Critical View
Themes in Learning Analytics - A Critical ViewLACE Project
 
European Perspectives on Learning Analytics: LAK15 LACE panel
European Perspectives on Learning Analytics: LAK15 LACE panelEuropean Perspectives on Learning Analytics: LAK15 LACE panel
European Perspectives on Learning Analytics: LAK15 LACE panelLACE Project
 

Más de LACE Project (20)

SHEILA workshop at EC-TEL 2018
SHEILA workshop at EC-TEL 2018SHEILA workshop at EC-TEL 2018
SHEILA workshop at EC-TEL 2018
 
SHEILA Results – Conference 5 June 2018
SHEILA Results – Conference 5 June 2018SHEILA Results – Conference 5 June 2018
SHEILA Results – Conference 5 June 2018
 
SHEILA LAK18 Presentation
SHEILA LAK18 PresentationSHEILA LAK18 Presentation
SHEILA LAK18 Presentation
 
SHEILA Project – LAK18 Workshop Slides
SHEILA Project – LAK18 Workshop SlidesSHEILA Project – LAK18 Workshop Slides
SHEILA Project – LAK18 Workshop Slides
 
SHEILA project LAK17 workshop slides
SHEILA project LAK17 workshop slidesSHEILA project LAK17 workshop slides
SHEILA project LAK17 workshop slides
 
SHEILA Project - Workshop Slides Online Educa Berlin 2016
SHEILA Project - Workshop Slides Online Educa Berlin 2016SHEILA Project - Workshop Slides Online Educa Berlin 2016
SHEILA Project - Workshop Slides Online Educa Berlin 2016
 
Overview of the LAEP learning analytics project
Overview of the LAEP learning analytics projectOverview of the LAEP learning analytics project
Overview of the LAEP learning analytics project
 
EU-funded learning analytics projects
EU-funded learning analytics projectsEU-funded learning analytics projects
EU-funded learning analytics projects
 
Expert presentations: LAEP / LACE Amsterdam workshop
Expert presentations: LAEP / LACE Amsterdam workshopExpert presentations: LAEP / LACE Amsterdam workshop
Expert presentations: LAEP / LACE Amsterdam workshop
 
Leveraging the full value of learning analytics: The power of learning design
Leveraging the full value of learning analytics: The power of learning designLeveraging the full value of learning analytics: The power of learning design
Leveraging the full value of learning analytics: The power of learning design
 
Learning analytics LACE SoLAR Flare 2015
Learning analytics LACE SoLAR Flare 2015Learning analytics LACE SoLAR Flare 2015
Learning analytics LACE SoLAR Flare 2015
 
Scalable Learning Analytics and Interoperability – an assessment of potential...
Scalable Learning Analytics and Interoperability – an assessment of potential...Scalable Learning Analytics and Interoperability – an assessment of potential...
Scalable Learning Analytics and Interoperability – an assessment of potential...
 
Assessment Analytics - EUNIS 2015 E-Learning Task Force Workshop
Assessment Analytics - EUNIS 2015 E-Learning Task Force WorkshopAssessment Analytics - EUNIS 2015 E-Learning Task Force Workshop
Assessment Analytics - EUNIS 2015 E-Learning Task Force Workshop
 
LACE Spring Briefing - WatchMe project overview
LACE Spring Briefing - WatchMe project overviewLACE Spring Briefing - WatchMe project overview
LACE Spring Briefing - WatchMe project overview
 
LACE Spring Briefing - Lea's Box Project overview
LACE Spring Briefing - Lea's Box Project overviewLACE Spring Briefing - Lea's Box Project overview
LACE Spring Briefing - Lea's Box Project overview
 
LACE Spring Briefing - PELARS project overview
LACE Spring Briefing - PELARS project overview LACE Spring Briefing - PELARS project overview
LACE Spring Briefing - PELARS project overview
 
LACE Spring Briefing - the LACE project
LACE Spring Briefing - the LACE project LACE Spring Briefing - the LACE project
LACE Spring Briefing - the LACE project
 
LACE Spring Briefing - Learning analytics are more than measurement
LACE Spring Briefing - Learning analytics are more than measurementLACE Spring Briefing - Learning analytics are more than measurement
LACE Spring Briefing - Learning analytics are more than measurement
 
Themes in Learning Analytics - A Critical View
Themes in Learning Analytics - A Critical ViewThemes in Learning Analytics - A Critical View
Themes in Learning Analytics - A Critical View
 
European Perspectives on Learning Analytics: LAK15 LACE panel
European Perspectives on Learning Analytics: LAK15 LACE panelEuropean Perspectives on Learning Analytics: LAK15 LACE panel
European Perspectives on Learning Analytics: LAK15 LACE panel
 

Último

Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Sapana Sha
 
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...Sapna Thakur
 
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptxSOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptxiammrhaywood
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsTechSoup
 
Paris 2024 Olympic Geographies - an activity
Paris 2024 Olympic Geographies - an activityParis 2024 Olympic Geographies - an activity
Paris 2024 Olympic Geographies - an activityGeoBlogs
 
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Krashi Coaching
 
Student login on Anyboli platform.helpin
Student login on Anyboli platform.helpinStudent login on Anyboli platform.helpin
Student login on Anyboli platform.helpinRaunakKeshri1
 
Separation of Lanthanides/ Lanthanides and Actinides
Separation of Lanthanides/ Lanthanides and ActinidesSeparation of Lanthanides/ Lanthanides and Actinides
Separation of Lanthanides/ Lanthanides and ActinidesFatimaKhan178732
 
Interactive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communicationInteractive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communicationnomboosow
 
Accessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactAccessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactdawncurless
 
Beyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global ImpactBeyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global ImpactPECB
 
Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...
Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...
Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...EduSkills OECD
 
Grant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingGrant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingTechSoup
 
Sanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfSanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfsanyamsingh5019
 
Russian Call Girls in Andheri Airport Mumbai WhatsApp 9167673311 💞 Full Nigh...
Russian Call Girls in Andheri Airport Mumbai WhatsApp  9167673311 💞 Full Nigh...Russian Call Girls in Andheri Airport Mumbai WhatsApp  9167673311 💞 Full Nigh...
Russian Call Girls in Andheri Airport Mumbai WhatsApp 9167673311 💞 Full Nigh...Pooja Nehwal
 
Arihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdfArihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdfchloefrazer622
 
Nutritional Needs Presentation - HLTH 104
Nutritional Needs Presentation - HLTH 104Nutritional Needs Presentation - HLTH 104
Nutritional Needs Presentation - HLTH 104misteraugie
 
Measures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeMeasures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeThiyagu K
 
A Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformA Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformChameera Dedduwage
 

Último (20)

Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111Call Girls in Dwarka Mor Delhi Contact Us 9654467111
Call Girls in Dwarka Mor Delhi Contact Us 9654467111
 
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
BAG TECHNIQUE Bag technique-a tool making use of public health bag through wh...
 
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptxSOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
SOCIAL AND HISTORICAL CONTEXT - LFTVD.pptx
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The Basics
 
Paris 2024 Olympic Geographies - an activity
Paris 2024 Olympic Geographies - an activityParis 2024 Olympic Geographies - an activity
Paris 2024 Olympic Geographies - an activity
 
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
Kisan Call Centre - To harness potential of ICT in Agriculture by answer farm...
 
Student login on Anyboli platform.helpin
Student login on Anyboli platform.helpinStudent login on Anyboli platform.helpin
Student login on Anyboli platform.helpin
 
Separation of Lanthanides/ Lanthanides and Actinides
Separation of Lanthanides/ Lanthanides and ActinidesSeparation of Lanthanides/ Lanthanides and Actinides
Separation of Lanthanides/ Lanthanides and Actinides
 
Interactive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communicationInteractive Powerpoint_How to Master effective communication
Interactive Powerpoint_How to Master effective communication
 
Accessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactAccessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impact
 
Beyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global ImpactBeyond the EU: DORA and NIS 2 Directive's Global Impact
Beyond the EU: DORA and NIS 2 Directive's Global Impact
 
Advance Mobile Application Development class 07
Advance Mobile Application Development class 07Advance Mobile Application Development class 07
Advance Mobile Application Development class 07
 
Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...
Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...
Presentation by Andreas Schleicher Tackling the School Absenteeism Crisis 30 ...
 
Grant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingGrant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy Consulting
 
Sanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdfSanyam Choudhary Chemistry practical.pdf
Sanyam Choudhary Chemistry practical.pdf
 
Russian Call Girls in Andheri Airport Mumbai WhatsApp 9167673311 💞 Full Nigh...
Russian Call Girls in Andheri Airport Mumbai WhatsApp  9167673311 💞 Full Nigh...Russian Call Girls in Andheri Airport Mumbai WhatsApp  9167673311 💞 Full Nigh...
Russian Call Girls in Andheri Airport Mumbai WhatsApp 9167673311 💞 Full Nigh...
 
Arihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdfArihant handbook biology for class 11 .pdf
Arihant handbook biology for class 11 .pdf
 
Nutritional Needs Presentation - HLTH 104
Nutritional Needs Presentation - HLTH 104Nutritional Needs Presentation - HLTH 104
Nutritional Needs Presentation - HLTH 104
 
Measures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and ModeMeasures of Central Tendency: Mean, Median and Mode
Measures of Central Tendency: Mean, Median and Mode
 
A Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy ReformA Critique of the Proposed National Education Policy Reform
A Critique of the Proposed National Education Policy Reform
 

Implementing Learning Analytics

  • 1. Research into Practice Building and implementing learning analytics atTribal Chris Ballard, Data Scientist
  • 2. Building and implementing learning analytics 1. Start at the very beginning 2. How research and practice differ 3. Building a learning analytics platform 4. Implementing learning analytics 5. Summary
  • 3. A leading provider of technology-enabled management solutions for the international education, learning and training markets Higher Education Vocational Learning Schools / K-12 Research universities Employment-focused universities Government agencies Further education colleges Training providers and employers Government agencies Schools School groups State and district government agencies AboutTribal 3
  • 4. Objectives • Predict student academic performance to optimise success • Predict students at risk of non- continuation • Build on research into link betweenVLE activity and academic success • Scale data processing • Understand risk factors and compare to cohorts 3 years of matched student and activity data used to build predictive models Staff can use student, engagement and academic data to understand how they affect student outcomes. Information accessible in one place on easy to understand dashboards. Integrated withTribal SITS:Vision and staff e:vision portal Consultation with academic staff on presentation and design Accuracy of module academic performance predictions* 79% *Using module academic history and demographic factors R&D project overview
  • 6. Current projects Providing learning analytics for 160,000 students across a state-wide vocational and further education provider inAustralia Student Insight being implemented as part of the JISC UK Effective Learning Analytics programme
  • 7. What do we mean by practice?
  • 8. From research to practice Research • Domain knowledge • Interpretation • In depth understanding • Testing an approach Practice • Integrated into everyday life • Interpret easily • Take action • Implementing an approach
  • 9. Domain and people • What is the problem? • Identify the users and stakeholders • Data owners • Are research results sufficient? • Design • Project cost
  • 10. Technical • Research limitations • Architecture • Data munging • Automating manual processes • Data suitability • Robustness of technical platform
  • 11. Building a learning analytics platform Key design decisions 1. Transparency – knowing why a student is a risk 2. Flexibility – viewing learning analytics which relates to an institution's curriculum and organisation 3. Efficiency – ease of use, implementation and interpretation
  • 12. Information relevant for different users
  • 16. Ensemble Decision Combination Enrolment Academic Performance Engagement Historic module results VLE Event Data Library Event Data Attendance VLE Library Attendance % % % % % % % Demographics Risk Prediction % Module History AssessmentsFormative Assessments Datasets Ensemble learning Student Weight Weight Weight Weight Weight Weight
  • 18. Providing flexibility to an institution
  • 19. Reflecting differences between courses Student activity data is not consistent across all courses/modules 1. Standardise data so it is comparative across all courses and modules 2. Build different models for each course or module Need to be careful that you have sufficient data for the model to generalise to new data.
  • 20. Provide opportunity for intervention Identify student at risk Log intervention details Assess intervention effectiveness Learning analytics Allocates intervention to support team Assign SLA SLA based alerts Monitor intervention progress Student support teams
  • 21. Embed into business process Need to consider how learning analytics becomes embedded into the day to day working life of academic and support staff.  Notifications – analytics becomes proactive; support different types of notification  Integration – accessible from existing tools and services through single sign on
  • 22. Implementing learning analytics CRISP-DM – Cross Industry Standard Process for Data Mining https://the-modeling-agency.com/crisp-dm.pdf
  • 24. Data understanding Understanding which features are important Example: end month of unit for successful and failed units
  • 25. Data preparation Creating comparative features Example:Total proportion of hours worked on failed units
  • 26. Modelling and evaluation Understanding whether model is under- or over- fitting Example: Learning curve for Random Forest model
  • 27. Evaluation  Define business focused success criteria  Define model focused success criteria  Define what baseline performance is acceptable  Consider a model cost benefit analysis that takes into account intervention cost Actual Withdrawn Enrolled Predicted Withdrawn benefit cost Enrolled cost benefit
  • 28. Summary Design  Embed learning analytics into business process  Ensure that analytics can be interpreted easily by staff  Intervention processes that are clearly articulated  Measure intervention effectiveness Implementation  Use a standard project approach such as CRISP-DM  Evaluate data in the context of the business problem and process  Define what success means, including acceptable accuracy and how it needs to be measured

Notas del editor

  1. Tribal provides management solutions to the international education, training and learning markets. Our tools allow education providers such as universities, colleges and local government to manage student admin and learning processes such as recruitment, admissions, finance, timetabling and course portfolios. Student management systems – for example 70% of UK universities use our Higher Education Student Management system called SITS:Vision. Involved in large scale education technology implementations – e.g. we have recently implemented one of our student management systems across all schools and campuses in New South Wales, Australia.
  2. Student Insight has been developed through a close working partnership between the University of Wolverhampton and Tribal. The original objectives of this partnership were to identify how we could build on initial research carried out by the university and build a solution that could enable Wolverhampton to benefit from improved use of student data. The outcome of this partnership is the Student Insight product that has been developed as a configurable solution that can now be adopted by other institutions. Consultation with academic staff at the university has enabled us to design the system in such a way so that it is flexible and can be tailored to meet the unique needs of each institution.
  3. Example of violin practice from my childhood…
  4. What do we mean by practice? Applying something that has been proven by research to be effective Moving from a research technique to application of that technique Research is focused on testing a technique or proving a theory; practice is focused on applying the results of that research so that we can benefit from it. To put it into perspective, we can compare research to a spreadsheet and practice to mobile apps. The latter "just works" and we can interpret what it is saying easily, and integrate it within our lives. The former requires in depth understanding, evaluation, interpretation and domain knowledge. We can't just stop and use it at a bus stop. But to go from one to the other takes effort and time to translate it from one area to the other.
  5. Knowing what problem we are using the results of the research to solve. This has to be a real problem for which we have real data (or could collect data). Something that people are willing to use (and in the case of a product, pay for!) Identify the users and what their problems are - we might have a model that can identify student's at risk, but how do staff want to see this presented? Do they want to see individuals, or are they more interested in overall patterns to help future planning? Who are the stakeholders? Different from the users. Data owners. Senior leadership team - input onto university's strategy. Important that their needs are represented. Asking ourselves whether the research results are sufficient in order to be applied in the real world, whether further refinement is necessary, or whether what we have done needs to be put on the shelf. Very often, we will identify further improvements, but these may not necessarily stop us from moving to application. These may come gradual refinements to something that is being used. Design decisions - how does our technique need to change to be used in the real world? User interface; interpreting results of some analysis; involves understanding of objective; Might want to pilot different approaches to see what works - A/B testing commonly employed by Facebook etc. - look this up).
  6. Technical decisions - often something written for research is not suitable for using in the real world. E.g. scalability, limitations of original research in order to keep the research focus narrow, hard coding, handling flexibility Data munging - often data used in research will have been collected from multiple sources and have been manipulated to resolve data quality issues, sampled etc. We will need to identify how to convert the manual approaches which were sufficient during research to automate them. Integration may present a real practical problem - often data owners want to hold onto their data! Data monitoring - a model is only as good as the data you use, therefore we need to be sure that data is loaded into the system correctly and there are no data quality issues.
  7. How did we move from research to a platform that could be used by different institutions? Key design decisions were:   Transparency - ensuring you know why a student is at risk Flexibility - allowing an institution to see analytics in a way that relates to their curriculum and organisation Efficiency - allowing the product to be implemented quickly, reducing implementation cost Here are some examples of key decisions that we made during the development of the product.
  8. Worked with the University of Wolverhampton to identify the main users. Identified three main classes - Course Director, Module Director, Personal Tutor. Wanted to be able to monitor the students they are responsible for quickly and easily, from an aggregated perspective and down to individuals. Realised that we needed to do this generically as other institutions may have different requirements - different user roles and different institutional structures. Ultimately this changed how we approached the technical design of the system but also what features the system provides. We therefore designed an institution structure that could represent these different structures allowing the platform to adapt to different situations. Security can then be applied to different student groups within that structure. Discuss prediction aggregation.
  9. Seeing an aggregated view of student risk as well as individuals - staff said they wanted to see how much groups of students may be at risk, not just individuals. Predictions are aggregated across the institution structure automatically to provide this information.
  10. Seeing an aggregated view of student risk as well as individuals - staff said they wanted to see how much groups of students may be at risk, not just individuals. Predictions are aggregated across the institution structure automatically to provide this information.
  11. We didn't want our models to be a "black box" where no-one could understand what they do, or why they identify a student at risk. An important consideration when intervening with a student is understanding what the data is saying and why the system has flagged a student. The initial research suggested a way forward and we evaluated a number of ways to solve this problem as part of the early "second phase" of R&D of the product. There are two examples of this: Ensemble learning - multiple predictions, single overall decision. Influence chart allowing comparisons across different data sources. Helps at an individual student level.
  12. Group influence chart - ability to see what is going on from a more strategic level. What factors influence student outcomes? Helps at a more strategic level when designing intervention measures.
  13. Flexible data - institutions have different characteristics and students with different needs and backgrounds. Therefore it is important that the data you bring into the system reflects the needs of both the institution and your students. Your data requirements may change over time - new data may become available and you may want to test its efficacy in student early warning prediction. During research, data is necessarily hard coded and fixed, and, as a result, any models built from that data are more static and relate to the structure of the data that has been used. We built a flexible modelling approach that allows you to bring any data into the system, and map it and view what that data looks like. You can then test models to verify the usefulness of the data to student early warning prediction. Configuration by an institution - an institution can configure the application once it has been setup.
  14. Institutions are complex because rather than one overall consistent business process, often different faculties, departments, courses or even modules have different approaches to the delivery of their curriculum. The best example of this is in the use of the VLE for the delivery of course materials - some modules may not make use of the VLE at all, or may have a different approach to how the content is delivered. This will change the VLE usage patterns in the data and this needs to be taken into account when using student activity data sources such as those from the VLE. This represents one of the key areas that needs to be considered when implementing learning analytics - designing an approach that takes into account the different needs of students taking different courses and how that is reflected in the data. A conventional approach is to normalise the raw data to reflect this - ensuring that the predictive features can be compared across different course or modules. An alternative approach is to build separate models for different courses. However an impact of this may be that your training data for an individual course/module becomes too limited to generalise well to new data. Both approaches need to be compared in your context to identify which one works best for you. We are planning on building a tool in Student Insight that allows you to perform this comparison and automatically segment models by course using the institution structure hierarchy.
  15. Intervention - unless the research is focused on evaluating the impact of interventions taken, one of the key areas which separates practice from research is in the action which is taken as a result of an early warning of risk. Staff should be able to determine whether an intervention is necessary based on the analytics and information they can see about the student and their context. They need a process to determine what types of intervention exist, record the intervention and then track the progress the student is making following that measures that have been taken. A key starting point is having clear institutional guidance about the intervention measures that are available, and in what circumstances they should be applied. With Student Insight, we have built in integration with our student support product that allows an intervention to be manually applied and assigned to the correct student support team for action and communication with the student. Institutions differ in their approach to how this is handled and therefore it is important that the workflows can be configured to reflect that.
  16. Notifications - you need to consider how learning analytics can become embedded into the day to day working life of staff. There may be barriers to widespread adoption if staff need to go somewhere to look up information. Although predictive analytics is by nature proactive, if staff need to go and look for early warning indicators, then it will start to be used retrospectively. Notifications are one method to overcome this where staff perhaps receive a communication via email highlighting students who are at risk. They can then take action and arrange meetings directly from this communication or choose to view more information about the student. In addition, which may need to raise the profile of students at risk who have not been dealt with, perhaps highlighting those who have not been reviewed to ensure that they do not slip through the net.
  17. When implementing learning analytics, institutions need to follow a project approach that ensures that key decisions have been taken and the project has been fully evaluated at different steps. This will increase the likelihood of project success and the embedding of learning analytics into the day to day activities of the institution.   When implementing Student Insight with an institution, we use an established data mining project methodology called "CRISP-DM" - Cross Industry Standard Process for Data Mining. CRISP-DM was first conceived in 1996 and a review in 2009 called CRISP-DM the "de facto standard for developing data mining and knowledge discovery projects."   The process model provides an overview of the lifecycle of a data mining project. Here we use the terminology "data mining", but it equally applies well to any analytics project. It breaks the analytics process down into six discrete phases. It is important to note that the order that the phases are approached is not constrained, as the result of each phase will dictate which phase needs to be performed next.   The diagram illustrates analytics as a cyclical process. Lessons learned during deployment and use of a model provide inputs to further iterations of the process and provide inputs to more in depth understanding of the business problem to be solved.
  18. Business understanding   Understand what you are trying to accomplish, what is the problem that we are trying to use analytics to solve? This is a key step because although there are commonalities between how institutions are currently looking to deploy analytics, they are sufficiently different to have different slants on the same problem. For example, reducing student attrition may be a key issue, but there may be differences between the causes, effects and students for whom it is a particular issue. For example, although our objective may be to reduce attrition, it may be better to focus on student success for some cohorts of students, rather than specifically identifying students at risk of dropping out.   If we don't do this step we might identify the wrong objectives, or possibly spend a lot of time analysing data to get the right answers to the wrong problems in the first place.   In addition to identifying the objectives for the project, you need to agree what will be considered success, and therefore establish some success criteria. Such criteria may cover different areas, and relate to tangible improvements against which we want to target the use of analytics. For example in the case of retention, we may want to increase retention by a specified amount. If we decide on criteria such as this, we will need to carefully plan whether we wish to try and attribute any improvement to the implementation of analytics, taking into account other factors.   JISC Discovery Phase accomplishes most of the tasks required in the Business Understanding phase of a project.
  19. Data understanding   Having good quality data available is at the heart of successful implementation of learning analytics. Although there are a large variety of algorithms, having good data with well thought out predictive attributes is the most important step is learning analytics implementation. Indeed, it has been said that 80% of total time spent on an analytics project is not building models, it is working with the data. So this is an important phase. The data understanding phase starts with the identification of potential data sources, collection of that data, assessment of data quality and initial analysis to help with understanding further. It is important that this is carried out in cooperation with data owners across the institution, as well as those who understand the link between a business process and how that process is reflected in the data itself. So, during a Tribal project, we identify the relevant people across the institution who need to be involved in this process.  
  20. Data preparation   Once the data understanding phase has been completed, the raw data collected in the data understanding phase will need to be processed and prepared ready for modelling. This may involve resolving data quality issues, creating aggregated summaries or merging multiple data sources together. Both business and data understanding feed into this stage - it may involve transforming the data to reflect the business process we are modelling. A common issue that you will encounter when working with student data is dealing with time dependencies, where data updates made at the time an event occurs affect data that acts as inputs to our model. For example, current modules a student is studying may be automatically recorded as a fail once the student has withdrawn. If we're not careful then failure can become a very good predictor of likelihood of dropout. It may be, but we need to control for these quirks in the way that the data is recorded otherwise the model that we build will not reflect the real world.
  21. Modelling   Once data has been prepared, we're ready to build a model against the data. This will involve selecting appropriate algorithms and comparing the performance of models built using each algorithm. Parameters for each algorithm will need to be chosen so as to optimise the performance of a particular model. One of the most important aspects of this process is identifying whether our model is underfitting or overfitting the data. There can be a number of reasons for this, but one of the most important factors is that more complex models will tend to overfit the data and the converse is true in the case of underfitting. Complexity can come in different forms, for example models with a large number of predictive features in comparison to the number of training examples will tend to be more complex, and thus overfit. In each case, it will mean that our model does not generalise well to instances it has not seen before, and may not give us optimal performance.   When implementing a model in Student Insight at Tribal, one of the main techniques we use are learning and fitting curves which can be used to diagnose whether a model is under or over fitting.   [Illustrate our modelling process]   We have also built in functionality into Student Insight to allow a model to be optimised automatically whilst it is being trained.
  22. Evaluation   In the case of predictive analytics, another type of success criteria may relate to goals as to the accuracy of models which have been built. In this situation, it is not sufficient to arbitrarily choose a baseline accuracy figure, but we need to decide what is going to be a sufficient baseline accuracy for our model. Accuracy of a predictive model is measured in different ways, according to our goal and the nature of the data we have available. We may decide that having a model which makes as few false positive predictions as possible is our goal, at the expense of the overall number of positive predictions made. Conversely, we may decide that we wish to make a large number of positive predictions in order to capture as many students at risk as possible. Choosing the appropriate balance for these figures can be tricky and needs to be based around an assessment of business objectives and intervention cost. Often, a cost benefit analysis can be helpful.