We describe our work on serious games, game-like simulations, gamification and learning analytics. We describe some of the experiments done and our approach to do a more scientific approach in the serious games domain. We present some of the results from H2020 RAGE and BEACONING european projects

1. Gamification in medical training:
from content and procedures to
game-like applications
Baltasar Fernandez-Manjon
balta@fdi.ucm.es , @BaltaFM
e-UCM Research Group , www.e-ucm.es
Gamilearn 2017, Tenerife, June 6
Realising an Applied Gaming Eco-System

2. Only the medical domain?
•Not only!
•What we have learned in the medical
domain can be applied to many other
domains

3. Serious Games
• Any use of digital games with purposes other than entertainment
(Michael & Chen, 2006)
• The use of digital games for educational purposes
• Other terms frequently used:
• Educational games,
• Applied games,
• Game-like Simulations,
• Games with Purpose,
• Simulations
• Edutaiment
3

4. Gamification, serious games, game-like
applications?
https://www.gamified.uk/2016/04/21/simulation-breaks-free-game-thinking/

5. Not clear borders …
http://codingconduct.cc/9-5-Theses-on-Gamification
Where is simulation here?
Where game-like simulation?

6. Serious games
Games are used in different fields such as
military, medicine, science…
They provide several benefits:
engaging, goal-oriented.
Serious games main purpose is not
to entertain but to
- learn
- change attitude or behavior
- create awareness of an issue
https://www.americasarmy.com/
http://www.aislados.es/
http://play.centerforgamescience.org/treefrog/

7. Serious games learning goals
http://www.celt.iastate.edu/teaching-resources/effective-practice/revised-blooms-taxonomy/

8. Gaming for good
• gaming approaches applied to solving complex problems in a
collaborative way
8
Play to Cure™: Genes in Space - a mobile game in which players collaborate to analyse real genetic data
(Cancer Research UK, n.d.)
http://centerforgamescience.org/portfolio/foldit/
http://www.cancerresearchuk.org/support-
us/play-to-cure-genes-in-space

9. Serious Games challenges
• Identification of more large success cases
• including scientific and formal evaluation
• taking into account current technical infrastructure
• Scalability and maintainability
• reducing initial cost and TCO
• Taking into account curriculum and involving
educators
• Sharing the actual game (open code)
• Including data-based evaluation into games
• Learning analytics

10. Use of games in residency programs in USA
(2010)
Akl EA, Gunukula S, Mustafa R, et al. Support for and aspects of use of educational games in family medicine
and internal medicine residency programs in the US: a survey. BMC medical education. 2010;10:26.

11. Alinier G. A typology of educationally focused medical simulation tools. Medical teacher. 2007;29(8):e243–50.
Available at: http://www.ncbi.nlm.nih.gov/pubmed/18236268.

12. Effects of different types of simulations
Arnold, J. J., Johnson, L. M., Tucker, S. J., Chesak, S. S., & Dierkhising, R. A.
(2013). Comparison of Three Simulation-Based Teaching Methodologies for
Emergency Response. Clinical Simulation in Nursing, 9(3), e85–e93.
“to compare the effects of 3 simulation methodologies
(low-fidelity, computer-based, and full-scale) on the
outcomes of emergency response knowledge,
confidence, satisfaction and self-confidence with
learning, and performance.”
“the statistical findings did not support the hypothesis
that RNs who receive full-scale simulation training will
score higher in knowledge, confidence, and performance
than those receiving computer-based simulation or low-
fidelity simulation”
Note: only 28 participants

13. A Serious Games for (almost) any task?
http://www.slideshare.net/dings/would-the-real-mary-poppins-please-stand-up-49259549/35-and_snap_the_jobs_a

14. Can we apply serious games?
Margolis JL, Nussbaum M, Rodriguez P, Rosas R. Methodology for evaluating a novel education technology: a case study of
handheld video games in Chile. Computers & Education. 2006;46(2):174–191.

15. Do games actually works?
In many cases, we are not fully sure
about it
- Usually, no full formal evaluation has been carried out
- Limited number of users
- Formal evaluation could be as expensive as creating the
game (or even more expensive)

16. Example: First Aid - CPR game
• Collaboration with Centro de Tecnologias Educativas de Aragon, Spain
• Identify a cardiac arrest and teach how to do a cardiopulmonary
resuscitation
• Oriented to middle and high school students
• Includes how to use a automatic defribilator
• Tested in 4 schools with 340 students
Marchiori EJ, Ferrer G, Fernández-Manjón B, Povar Marco J, Suberviola González JF, Giménez Valverde A.
Video-game instruction in basic life support maneuvers. Emergencias. 2012;24:433-7.
Available at http://first-aid-game.e-ucm.es

19. Formal Evaluation
Simultaneous sessions with same time
One with game
Other with 2 emergency medical doctors with a medical
mannequin and an automatic defibrillator

20. First aid - CPR game Results
Pre-test and post-test for evaluating knowlege
game

21. Example: Surgical Safety Checklist
21
http://www.who.int/patientsafety/safesurgery/en/index.html

22. 2

23. 2
When implementation leaders did not explain why or show how
the checklist should be used, staff neither understood the
rationale behind implementation nor were they adequately
prepared to use the checklist, leading to frustration, disinterest,
and eventual abandonment despite a hospital-wide mandate
Conley, et at (2011). Effective surgical safety checklist implementation. Journal of the American
College of Surgeons, 212(5), 873–9

24. We created the Checklist Game
• Raise awareness about the checklist
• Learn how to apply it properly and consequences of not applying it
• Let practice it application in a free-risk environment
In cooperation with UCM (Surgical Department), Hospital Doce de Octubre, LCS-
MGH
http://sourceforge.net/projects/e-adventure/files/games/checklist/

25. Game description
• Role selection
• Checklist coordinator
• Case based game
• Two cases with features that covers most common checklist errors

26. Game mechanics
• Procedure freely explored
• Three stages

27. Game elements and Feedback
• Randoms events or team not cooperating
• Feedback

28. Checklist game evaluation
• Formative evaluation at UCM and MGH/Harvard
• Final evaluation at 5 hospitals in Madrid (Clinico, 12 de Octubre, Santa
Cristina, Puerta de Hierro)
• Some good initial results

29. A quasi-experimental study (pre-post)
• 90 profesional from OR: 21 anestesiologyst, 31 nurses, 35 surgeons
• Three constructs evaluated
• Utilitily of the check-list perceived by the professionals
• 11 Likert 7 values question (alfa de Cronbach: 0,927)
• Procedural knowledge about the correct administration of the check-
list
• 10 test questions
• Perception of the effort of the correct application of the check-list
• 3 Likert 7 values question (alfa de Cronbach: 0,73)

30. Results of the check-list evaluation
• Increase of:
• –Utilitily of the check-list perceived by the professionals (p<0.000)
• Procedural knowledge about the correct administration of the
check-list (C) (p=0.018).
• •Decrease of :
• –Perception of the effort of the correct application of the check-
list (PE) (0=0.004).
Pedro Ruiz López, Ángel del Blanco, Javier Torrente, Manuel Giner; Baltasar Fernández Manjon. (2014).
Impacto de un videojuego del check-list quirúrgico en el conocimiento y sensibilización de los profesionales.
Comunicación oral XXXII Congreso SECA, Madrid, 15, 16 y 17 octubre 2014.

31. Improving
collaboration between
different experts

32. Methodology for game development
Torrente et at (2014) Development of Game-Like Simulations for Procedural Knowledge in Heathcare Education. IEEE Transactions on
Learning Tecnologies. 7(1), 69-82
USERS

33. EGDA Methodology
• Learning procedural knowledge in healthcare
• Game-like simulations created from
• description of medical procedures
• representative teaching cases
• Close collaboration of game designer with medical experts
• Minimal requirement for medical experts
• Rapid prototyping (using eAdventure)
• Early validation by experts

34. eAdventure game platform
Authoring environment for the production of point-and-click
adventure games & immersive learning simulations

35. Formal evaluation pre-post
• Formal evaluation of games is very
complex and expensive
• Pre-test
• Post-test
• Very few games have been formally
probed to be effective
• Similar results with
Learning Analytics than
with pre-post test?
35

36. Serious games issues
Usually serious games effectiveness is measured through pre-post tests
But actual learning takes place
during in-game interactions
How to measure game effectiveness?
Games usually have a black box model (only score)
No information about what is happening inside the game while
the user is playing

37. Analytics and Game Learning Analytics
Game Learning Analytics
(GLA) for Serious Games:
- collect, analyze and
visualize data from learners’
interactions
Can GLA be systematized?
➔ In entertaining games: Game
Analytics (GA)
➔ In learning systems:Learning
Analytics (LA)

38. First step: Data tracking
But data collection for analytics lacks of standards.
New standard interactions model
developed and implemented in
Experience API (xAPI) with ADL
(Ángel Serrano et al, 2017).
The model allows tracking of all
in-game interactions as xAPI traces
(e.g. level started or completed,
interactions with NPC or game items,
options selected, score increased)
https://www.adlnet.gov/xapi/

39. Data tracking with xAPI for SG
xAPI model for serious
games developed by e-
UCM research group in
collaboration with ADL
https://www.adlnet.gov/serious-games-cop

40. H2020 EU RAGE Project simplifies the
creation of SGs via ready-to-use assets
➔ game tracker and analytics server
Traces in xAPI are sent to the
RAGE Analytics server for their analysis.
➔ general game-independent analysis & visualizations
provided
➔ possible to configure game-dependent analysis
Next steps: Data analysis and visualization

41. Data analysis: game (in)dependent
Systematization of GLA: set of game-independent analysis
provided for specific stakeholders:
- teachers: players’ progress, players’ errors
- developers: times of completion, videos seen/skipped
- students: final results, errors made
As long as traces follow xAPI format, these analysis do not
require further configuration!
Also possible to configure game-dependent analysis and
visualizations for specific games and game characteristics.

42. RAGE Analytics Dashboards
EDUCON 2017

43. RAGE Analytics Alerts and warnings
From the game dependent
analytics specific
Alerts and Warnings can be
generated
e.g. teachers gain insight and
real-time control of their
classes when deploying
games

44. GLA architecture and technologies

45. Current work: uAdventure
Previous game engine eAdventure (in Java).
• Helps to create educational
point & click adventure games
• Users do not need to program
Platform updated to uAdventure (in Unity).
Full integration of game learning analytics
into uAdventure authoring tool
No extra effort required to integrate default analytics into uAdventure
games!

46. GLA full life-cicle

47. H2020 Beaconing project
• BEACONING stands for ‘Breaking Educational Barriers with
Contextualised, Pervasive and Gameful Learning’
• Global goal is learning ‘anytime anywhere’
• Exploitation of technologies for contextual pervasive games and use of
gamification techniques
• Problem based approach to learning
• Enriching the Learning Analytics data model with the contextual,
geolocalized and accessibility information

48. BEACONING GLA Pilot: Experiment description
• 227 students
• 1, 2, 3 and 4 year of ESO and 1 year of
BACHILLERATO
• From 12 to 17 years old
Each student completed:
1.Pre test (including game habits)
2.Complete gameplay (three game levels)
3.Post test (including game opinion)
• Game rebuilt with uAdventure
• Included analytics using RAGE tracker
based on xAPI specification.

49. Players clusters according to game habits
High correlation between
game habits and sex:
- frequent players and
male (triangle)
- less frequent players
and female(circle)

50. Learning compared to original experiment
• Original experiment
with the game
• Original experiment,
control group
• Current experiment
(from 8 to 9.8 out of 15)
Lower learning but still
significative!
Replicability of Results

51. Predicting post test score
1.With pre test information + game traces
Greater importance of:
- score in pre test
- game habits
- interactions with game elements

52. Predicting post test score
2.Only with game traces
Greater importance of:
- interactions with game elements
- scores in game levels
Long-term goal: predict score solely with in-game actions and,
therefore, avoid the pre test.

53. And players remember what they learnt!
10 students repeated the
experiment 3 weeks
after.
Lower learning than in the
previous experiment, but
their pre test score was
significantly higher
(recall).

54. And they also liked it!
Commonly used words in post test free
text comments:
❖ game
❖ liked
❖ good
❖ learnt
❖ fun
❖ teach

55. Case study: Downtown
• Serious Game designed and develop
to teach young people with Down
Syndrome to move around the city
using the subway
• Status: Designed and developed.
Analysis pending
• Audience: People between 15 and 40 y/o with
Down syndrome
• Platform: PC (Unity3D)

56. Case Study: Downtown
• From user requirements to a game
design and its observables
• Currently being evaluated with 50
people with cognitive dissabilities
56

57. Next steps: Cyberbullying

58. Desing based on research studies
Seminario eMadrid sobre Serious games 2017-02-24 58

59. Social Networks Risks
Seminario eMadrid sobre Serious games 2017-02-24 59
Implications of the social networks

60. Serious Games challenges
• Identification of more large success cases
• including scientific and formal evaluation
• taking into account current technical infrastructure
• Scalability and maintainability
• reducing initial cost and TCO
• Taking into account curriculum and involving
educators
• Sharing the actual game (open code)
• Including data-based evaluation into games
• Learning analytics

61. 61
Thank You!
Gracias!
¿Questions?
• Mail: balta@fdi.ucm.es
• Twitter: @BaltaFM
• GScholar: https://scholar.google.es/citations?user=eNJxjcwAAAAJ&hl=en&oi=ao
• ResearchGate: www.researchgate.net/profile/Baltasar_Fernandez-Manjon
• Slideshare: http://www.slideshare.net/BaltasarFernandezManjon