Dr. Gábor Kismihók's presentation at Textkernel's Conference Intelligent Machines and the Future of Recruitment on 2 June 2016. Learning analytics is an emerging discipline in education, aiming at analysing (big) educational data in order to improve learning processes. In this talk, Dr. Gábor Kismihók will give an overview about the main challenges of this field, with a special emphasis on bridging the education - labour market divide.

1. jobknowledge.eu
Labour Market Driven Learning Analytics
Dr. Gábor Kismihók
Senior Researcher
University of Amsterdam
Amsterdam Business School
g.kismihok@uva.nl
@kismihok
www.eduworks-network.eu
www.jobknowledge.eu

2. jobknowledge.eu
Background
Multidisciplinary (HRM-OB/Data
Science/Knowledge Management/
Education)
Focus:
• Data science and HRM/Lifelong
Learning
• Learning Analytics
• Adaptive, personalised assessment
• Self regulated learning
• Job Analysis/Job Knowledge
• Employability of individuals
• HR practicies in the 21st century

3. jobknowledge.eu
Learning Analytics
Learning analytics is “the measurement, collection, analysis, and
reporting of data about learners and their contexts, for purposes of
understanding and optimizing learning and the environments in which
it occurs” (SOLAR 2012).
4 levels of LA
• Describe
• Diagnose
• Predict
• Recommend
Expertise: Educational scientist, computer scientist, data scientist,
managers, teachers, students, labour market representatives

4. jobknowledge.eu
Hot topics
Person
• Learner profiles
• Personalisation
• Individual
feedback/benchmarking
• Teaching analytics
Organisation
• Student retention
• Curriculum design
• Workplace and
professional learning
• Institutional readiness
Technology
• Tools and interventions
• Visualisation
• MOOCs
• Predictive modelling
• Learning environments
Pedagogy
•Learning design
•Personalisation
•Feedback
•Blended learning
Ethics
•Data ownership
•Data management
•Transparency of algorithms
•Quality of
recommendations

5. jobknowledge.eu
What type of data?
• Performance
• Grades
• Assignments (text)
• Behavioral
• Clicks
• Content views
• Social media
• Physiological
• Pulse
• Brain activity
• Labour Market data
• Vacancy data
• Economic indicators/surveys

6. jobknowledge.eu
Examples
Purdue University (US) – Course Signals
• Predicting student drop outs based on LMS activity
• Teachers and students are notified and personal intervention is planned
• 21% retention rate improvement (Kimberley and Pistili 2012)
OU Analyse
• Predicts students at risk
• Predictive models on the basis of VLE and demographic data
• Also explains the reason, recommends activities
• Open dataset
Social Networks Adapting Pedagogical Practice (SNAPP)
• real-time social network analysis and data visualisation of forum discussion
activity
• identification of isolated students, non-functioning groups or groups need to work
together

7. jobknowledge.eu
Examples
Predictive Analytics Reporting (PAR) Framework (US)
• Non-profit provider of analytics-as-a-service
• Central analytics service for HE institutions
• Cross institutional analyses
Kahn Academy Analytics (US/Global)
• Learning content is mapped to skills
• Learning content is offered on the basis of effort, engagement, difficulty,
etc…

8. jobknowledge.eu
Labour market oriented
learning trajectories
Match learners’ pathways
to those of alumni
Help incoming students
find a long term focus
during their university
education
Mirror alumni data
to current students
based on
desired/acquired
occupations

9. jobknowledge.eu
Goal setting pilot
• Goal setting improves performance
• Create a goal setting interface for students to manage and track
(learning) goals
• Focus of research:
• Goal commitment/Shared goals
• Matching goals to behavioral and performance data
• Apply advanced analytics

10. jobknowledge.eu
Goal App Features
• Set goals (Specific, Measurable, Attainable, Relevant, Time Based)
• Set sub-goals
• Option to make goals private or public
• Feedback on goals
• Can view and commit public goals
• Tag goals
• Learning records
• Dashboard
• Reminders
First findings:
• Difficult to think about goals
• Goals should be generated
on the basis of labour market data
(vacancies)
virgo.ic.uva.nl:3000

11. jobknowledge.eu
Learning Analytics is a great opportunity to shorten
the gap between education and the labour market

12. jobknowledge.eu
Challenges
xkcd.com

13. jobknowledge.eu
Challenge 1
Changing nature of the society and
the Labour Market

14. jobknowledge.eu
Jobs (society) are changing
What are the relevant job information types, what is a job in the 21
century?
• increased idiosyncratic nature of work and the crafting of jobs by job holders
• 161 job information types in 50 studies - Volume and semantics
Shared economy
New selection and recruitment methods
Many vacant jobs simply do not show up on the web
Role of vacancy announcements in the future
How to target the output of Education?

15. jobknowledge.eu
Challenge 2
Landing analytics at organisations
xkcd.com

16. jobknowledge.eu
Stakeholder analysis
(Szörényi, 2014)
LA stakeholder map at the UvA

17. jobknowledge.eu
Positioning a learning analytics project
• Centralized vs. decentralized
• Research vs. practice
• Imposed vs. desired
• Technology vs. Pedagogy
• Make or buy (resource oriented)
• Adoption of best practices or local
identification thereof
There is a lot of ambiguity and fear, but little
experience with LA

18. jobknowledge.eu
Sandbox – safe environment to play
https://www.flickr.com/photos/mzn37/

19. jobknowledge.eu
Challenge 3
Terminology

20. jobknowledge.eu
Definitions of constructs (like skills) are very
context sensitive
Data generation is not unified, terminology is weak (in the hands of HR
managers and their objectives)
Definition is influenced by
• The data producer (person or machine)
• Country, region
• Organisation
• Occupation
• Language
Universal taxonomy, ontology

21. jobknowledge.eu
Challenge 4
Data and algorithms

22. jobknowledge.eu
Call for Transparency
• Trust is a big issue
• Data/algorithms are often not public - Black box society
Web-data:
• Traceability is an issue
• Scraping policies of data providers are not always visible
• Paid websites are rarely scraped

23. jobknowledge.eu
Educational data
• Many data silos
• Who owns what data?
• Highly political issue
• Organizational resistance
• Gatekeepers resistance
• Complex infrastructure

24. jobknowledge.eu
Challenge 5
Ethics

25. jobknowledge.eu
Ethics and Privacy
• One of the greatest barrier
• JISC reports 86 ethical, legal and logistical issues
https://analytics.jiscinvolve.org/wp/2015/03/03/a-taxonomy-of-ethical-legal-
and-logistical-issues-of-learning-analytics-v1-0/
• Algorithms/codes are often hidden – Black Box society
• Personal/sensitive data, lasting effects of recommendations
• How this data will be used on learners?
• What is the objective of the data collection? (not known at the time
the data is collected)

26. jobknowledge.eu
What is more important?
Personal development or privacy?

27. jobknowledge.eu
Educational data
Is it ethical to use these data to
improve the learning process?

28. jobknowledge.eu
Educational data
Is it ethical NOT to use these data
to improve the learning process?

29. jobknowledge.eu
Summary
Power is in numbers
Bigger data, better matching, better insights, better research
LA is a new area
Many opportunities for innovative ideas and services
More evidence (research) needed what works and what doesn’t
Inductive vs Deductive
Need to document failures (not only success) properly
Technology is not a bottleneck
Organizational awareness is growing - LA is on the agenda in many stakeholder
groups
Legal and ethical concerns are critical

30. jobknowledge.eu
Further information
• LACE Evidence HUB http://evidence.laceproject.eu/
• LEAP Inventory http://cloudworks.ac.uk/cloudscape/view/2959
• SOLAR Community https://solaresearch.org/
• Learning Analytics and Knowledge Conference (LAK)
• Learning Analytics Summer Institute (LASI)
• Journal of Learning Analytics http://learning-analytics.info/

31. jobknowledge.eu
Participate!
Facilitate discussions among key stakeholders

32. jobknowledge.eu
Invitations
• Eduworks community
Newsletter: http://www.eduworks-network.eu/
Fb: https://www.facebook.com/eduworksnetwork
• Eduworks events, stakeholder meetings
http://www.eduworks-network.eu/upcoming-events
• Eduworks Dutch Stakeholder meeting (October 2016)
• Contribute to our Sandbox!

33. jobknowledge.eu