VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017.
VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017.
Portfolio - just some things I'm working on - July 2021
Similar a VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017.
Similar a VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017. (20)
Transaction Management in Database Management System
VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017.
1. Fighting level 3:
From the LA framework to LA practice on
the micro-level
Hendrik Drachsler, @hdrachsler
LASI Spain, JULY 4-5, 2017
CC-by: https://www.flickr.com/photos/juandelpozo/6110435588/
2. 3
WhoAmI
Full Professor of
Educational Technologies and
Learning Analytics
School of Computer Science
Goethe University Frankfurt
German Institute for international
Educational Research (DIPF)
Frankfurt am Main, Germany
Welten Institute
Research Centre for Learning,
Teaching and Technology
Open University, Heerlen, The
Netherlands
3. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 3 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?
4. 4
Greller, W. & Drachsler, H. (2012). Turning Learning into Numbers. Toward a Generic Framework for
Learning Analytics. Journal of Educational Technology & Society.
http://ifets.info/journals/15_3/4.pdf
Learning Analytics Framework
5.
6. Let‘s not forget, LA is for teachers and
learners
Learner’s and teachers are most
of the located on the micro-level
with their needs and skills
7. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 7 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?
8. Sophistican model
Siemens, G., Dawson, S., & Lynch, G. (2014). Improving the Quality and Productivity of
the Higher Education Sector – Policy and Strategy for Systems-Level Deployment of
Learning Analytics. Canberra, Australia: Office of Learning and Teaching, Australian
Government. Retrieved from http://solaresearch.org/Policy_Strategy_Analytics.pdf
LA Sophistication Model
1
2
3
4
5
Who of you arrived at
level 3 with their
organisation?
How long you are already
in level 1 and 2?
9. Maturing, Roll-Out, Uptake?
VERY URGENT for uptake of LA
• Ethics & Privacy (Dragan keynote)
• Stakeholder needs & Competences (Hendrik)
• LA supported learning design (Hendrik)
LESS URGENT for uptake of LA (but lots of fun)
• Infrastructure
• Data (Katrien keynote)
• Dashboards (Katrien keynote)
10. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 10 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?
11. STOTA Dashboards
1. Verbert, et al., (2013) Learning analytics
dashboard applications. American
Behavioral Scientist 57(10), 1500
2. Verbert, et al. (2014) Learning
dashboards: an overview and future
research opportunities. Personal and
Ubiquitous Computing 18(6), 1499
3. Yoo, Y., Lee, H., Jo, I.H., Park, Y.:
Educational dashboards for smart
learning: Review of case studies. In:
Emerging Issues in Smart Learning, pp.
145. Springer (2015)
4. Schwendimann, et al., Perceiving learning
at a glance: A systematic literature review
of learning dashboard research. IEEE
Transactions on Learning Technologies
(2016)
5. Bodily, R., Verbert, K.: Trends and issues in
student-facing learning analytics
reporting systems research. In: Proc. of
LAK'17. pp. 309{318. ACM (2017)
For the question of uptake we have been
investigating 26 out of 95 papers that
have grounding in learning sciences:
According to which educational concepts
are learning analytics dashboards
designed?
12. Educational concepts mentioned in LA dashboard
papers
Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of
learning analytics dashboards in the educational practise. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate.
18
10
1
7
5
12
13. Competences mentioned in LA dashboard papers
Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of
learning analytics dashboards in the educational practise. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate.
34
8
12
5
13
15. Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of
learning analytics dashboards in the educational practise. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate.
Reference frames mentioned in LA
dashboard papers
24
16
10
16. • Design of LA Dashboard are technology driven (only 26 out
of 95 papers use edu. concepts)
• Mostly to support awareness and reflection
• According to McAlpine & Weston, refection is a mechanism
to facilitate learning & teaching not an goal in itself.
• More than half of the dashboards rely on SRL but none of
the tool really support the cycle:
fore-thought -> planning -> monitoring -> self-evaluation.
• Social framing is more common than achievement framing
According to which educational concepts are
learning analytics dashboards designed?
Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of
learning analytics dashboards in the educational practise. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate.
17. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 18 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?
18. Learning Analytics supported
Learning Design
Three main opportunities:
1. Using LA indicators for on demand measures
of the learning design
2. Intervening during the runtime of a course
due to LA indicators
3. Increasing students learning outcomes and
interventions through earlier interventions
with LA indicators
19. Opportunities and challenges for using LA in LD
Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., Drachsler., H. (2017). Opportunities and
challenges in using Learning Analytics in Learning Design. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.
20. Opportunities and challenges for using LA in LD
Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., Drachsler., H. (2017). Opportunities and
challenges in using Learning Analytics in Learning Design. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.
21. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 22 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?
22. Learning Analytics Readiness Instrument (LARI)
Oster, M., Lonn, S., D. Pistilli, M., and Brown., M., G. (2016). The learning analytics
readiness instrument. LAK '16. ACM, DOI: https://doi.org/10.1145/2883851.2883925A
23. Evaluation of the effect of LA
• If you want to measure uptake
of LA you need an instrument to
explore progress and adoption
• It needs to be a light way
instrument as students and
teachers don’t fill overwhelming
long surveys again and again.
24. We were looking for something quick
and robust like ...
... the SUS scale.
But than deidicated
for LA.
25. Scheffel, M., Drachsler, H.,
Stoyanov, S., and Specht, M.
(2014). Quality Indicators
for Learning Analytics.
Educational Technology &
Society, 17 (4):117–132.
Scheffel, M., Drachlser, H.,
Specht, M. (2015). Developing
an evaluation framework of
quality indicators for learning
analytics. LAK ’15, NY, USA.
ACM.
Scheffel, M., Drachsler, H., de
Kraker, J., Kreijns, K.,
Slootmaker, A., Specht, M.
(2017). Widget, widget on the
wall, am I performing well at
all? IEEE Transactions on
Learning Technologies, 10
(1):42–52.
Evaluation Framework for Learning Analytics (EFLA)
26. Evaluation Framework for Learning Analytics (EFLA)
Scheffel, M., Drachsler, H., Kreijns, K., de Kraker, J., Specht, M. (2017). Widget, widget as you lead, I am
performing well indeed!: Using results from an exploratory offline study to inform an empirical online
study about a learning analytics widget in a collaborative learning environment, LAK ’17, NY, USA. ACM.
Scheffel, M., Drachsler, H., Toisoul, C., Ternier, S., Specht, M. (2017). The Proof of the Pudding: Examining
Validity and Reliability of the Evaluation Framework for Learning Analytics. 12th European Conference on
Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.
27. Evaluation Framework for Learning Analytics (EFLA)
Quick evaluation of LA
tools in micro-settings.
We are looking for
partners to try EFLA and
further develop it.
A great outcome would be
N numbers of LA case
studies that applied EFLA
and share their
experiences.
Can we work towards an
EFLA score like the SUS
score?http://www.laceproject.eu/evaluation-
framework-for-la/
28. @HDrachsler, #
_NL, Zeist, Netherlands
Slide 29 / 29 June 2014
1. LA Framework
in a nutshell
2. Fighting
level 3
Lecture structure
5. How to
evaluate LA
on the work
floor?
3. Why are our
dashboards not
used?
4. How to
connect LA
and LD?
CONCLUSIONS
for Uptake of LA
LA is about
learning!
LA is about
learning!
29. The problem for uptake of LA
and mastering level 3 is ...
Educational
Science
Computer
Science
Learning
Analytics
MIDDLE SPACE
30. The solution, LA as a real middle space
Educational
Science
Computer
Science
Learning
Analytics
MIDDLE SPACE
Who of you has a computer science and who has a more educational background?
=> HUH, this is great than we can nicely penetrated LA in the middle space between Informatics, Analytics and Educational Science
Who of you is new (so less than a year) to LA research field?
a. To bad you will not understand to much of this talk.
b. Very good so all of you know what I‘m referring to in my title?
The picture of the road as symbol to getting to Level 3 and finally rollout and use what we are working on since almost 7 years or longer.
Who of you know the LA framework?
Although we have implemented various dashboards, policies, tools, have a national SIG in NL and plenty of attention we are still more prototyping than transforming the university into a data driven organisation -> WHY? Any ideas from the audience?
Maybe this is the answer? Maybe LA is not that much the middle space between Educational technologies and Computer Science that it should be.
What needs to be done to get more uptake, roll-out, maturing of LA in the educational everyday life?
What have you done to achieve the uptake?
Check programme again !!!!
Not taking away Katrien‘s keynote on Dashboards, so just the quick overview
Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.L.: Learning analytics dashboard applications. American Behavioral Scientist 57(10), 1500{1509 (2013)
Verbert, K., Govaerts, S., Duval, E., Santos, J.L., Van Assche, F., Parra, G., Klerkx, J.: Learning dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing 18(6), 1499{1514 (2014)
Yoo, Y., Lee, H., Jo, I.H., Park, Y.: Educational dashboards for smart learning: Review of case studies. In: Emerging Issues in Smart Learning, pp. 145{155. Springer (2015)
Bodily, R., Verbert, K.: Trends and issues in student-facing learning analytics
reporting systems research. In: Proc. of LAK'17. pp. 309{318. ACM (2017)3 .Design and evluation of 10 edu dashboards
Schwendimann, B., Rodriguez-Triana, M., Vozniuk, A., Prieto, L., Boroujeni, M., Holzer, A., Gillet, D., Dillenbourg, P.: Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies (2016) 55 dashboards (context, purpose, indicators & technologies used, evaluation, futuer work)
Very little research about the effects of LA Dashboards, none of them investigated the connection to learning sciences or use of educational concepts.
For the question of uptake we have been investigating 26 out of 95 papers that have grounding in learning sciences: According to which educational concepts are learning analytics dashboards designed?
Six clusters presentation of 17 educational theories, models, concepts identified in the LA Dashboard papers
Competencies and the goal that are inteded to affect each competence
Competences target by the dashboards in relation to educational concepts that were used as theoretical basis for their development.
Strong emphasis on Self regulated learning, through Cognitivism. SRL is a concept that requires forethought, performance, and self-reflection.Less used are instructional design, and for formative assessment as means of reflection.
Influence learners metacognitive skills. Surprisingly, very few dashboards aim to support learners on the cognitive level i.e. acquiring knowledge and improve performance
Wise, A., Learners need a references frame to interpret their data. We analysed this in the studied papers by looking at how information was contextualised on the dashboard.
Two types:
Social Comparison (most papers)
Goal Achievement
The competence level target by the dashbaord in realtion to the three reference frames identified: social (S: red), achievement (A: blue), and progress (P: yellow)
With constantly being exposed to motivational triggers that rely on social comparison ‚being better than your peers‘ becomes the norm in what defines an successful leaner? We argue that learning and education should be about mastering knowledge and therefore social comparison should be used carefully in the design of dashboards.
Attention should be given to different needs of learners with different performance levels and motivating factors.
These findings of LA Dashboards emphasis the need of desining LA dashboards as a tool embedded into learning design, and might be a reason for low uptake of LA in stakeholder groups.
Scheffel, M., Drachsler, H., Toisoul, C., Ternier, S., Specht, M. (2017). The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics. 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.
Does any one know what this is ?
Average scores of EFLA for students (left) and teachers (right)
Average scores of EFLA for students (left) and teachers (right)
LA is about learning and teaching
Our dashboards are not that much designed for learning theories and models
There are challenges for LA in LD that need to be addressed in each organisation as learning is strongly context based.
In order to measure Uptake we need a light evaluation instrument like EFLA
If we really want to achieve level 3 of the LA sophistication model, LA needs to ‚really‘ become the middle layer between educational science and computer science. So far on educational dashboard side as well as connection of LA to Learning Design, LA is still to far away from educational practice and even promoting in some papers contra productive incentive to promote ‚better‘ learning like focusing on social comparison rather than goal achievement.
We need to work closer with educational practise, take into account innovative learning and teaching concepts rather than visualising and comparing what is easy to do by the data we have.
If we really want to achieve level 3 of the LA sophistication model, LA needs to ‚really‘ become the middle layer between educational science and computer science. So far on educational dashboard side as well as connection of LA to Learning Design, LA is still to far away from educational practice and even promoting in some papers contra productive incentive to promote ‚better‘ learning like focusing on social comparison rather than goal achievement.
We need to work closer with educational practise, take into account innovative learning and teaching concepts rather than visualising and comparing what is easy to do by the data we have.