Presentation of full paper "Using Simva to evaluate serious games and collect game learning analytics data" at LASI-Spain 2019, held in Vigo 27-28 June 2019.

1. Using Simva to evaluate serious games
and collect game learning analytics data
Cristina Alonso-Fernández, Iván J. Pérez-Colado, Antonio Calvo-Morata,
Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón
LASI-SPAIN 2019
Vigo (Spain), June 27, 2019
e-UCM Research Group
www.e-ucm.es

2. Outline
● Introduction
● Simva
● Applications of Simva
○ #1: Conectado
○ #2: 15 Object Test
○ #3: First Aid Game
● Discussion & Conclusions

3. Introduction
Serious games increased application in multiple fields.
● increase knowledge
● raise awareness
● change attitude/behaviors
Pre-post experiments: common way to evaluate knowledge
acquisition from the gameplay
Game Learning Analytics (GLA) allows to obtain further
information from players’ in-game interactions

4. Simva
Tool to simplify the scientific validation of
serious games.
It manages:
➔ questionnaires
➔ classes of students
➔ Game Learning Analytics interaction data
Deals with privacy and anonymity.
Game
Learning
Analytics
data

5. Simva
Collects pre-post questionnaires and Game Learning Analytics interaction
data.
➔ LimeSurvey
➔ Questionnaires status:
(started, finished)
➔ Traces collection
➔ Control of steps
completion

6. Simva
Pseudo-anonymization technique:
➔ 4-letter random tokens
➔ No personal information
entered into the system
➔ Teachers can relate
students-tokens outside Simva

7. Simva

8. #1 Conectado: a SG to raise awareness
Serious game designed to raise awareness about
bullying and cyberbullying
➔ 12-17 years old
➔ Tool for teachers to start a discussion based on
the gameplay common experience
➔ +1000 students, +200 teachers & educational
science students
Ref: Antonio Calvo-Morata, Dan-Cristian Rotaru, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2018):
Validation of a Cyberbullying Serious Game Using Game Analytics. IEEE Transactions on Learning Technologies, DOI: 10.1109/TLT.2018.2879354

9. #1 Evaluation of Conectado
Simva used to evaluate Conectado via pre-post experiments
● Session managers distributed access tokens
● Conectado checks Simva configuration:
○ pre-post experiments available
○ user (token) has access to questionnaires
○ complete pre-test, then start game
● GLA data collected
➢ Results from experiments proved that Conectado indeed increases
awareness

10. #2 15-Object Test for active aging
Visual task to evaluate the slowing of
cognitive processing in Parkinson’s disease.
Experiments to compare:
➔ Two versions (A and B) of the task
➔ Paper vs computer based versions
Participants randomly assigned to 4
experimental conditions.
Ref: Dan-Cristian Rotaru, Sara García-Herranz, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón, Mª Carmen Díaz-Mardomingo (2018):
Using Game Technology to Automatize Neuropsychological Tests and Research in Active Aging. GOODTECHS 2018 - 4th EAI International Conference on Smart Objects and Technologies for Social Good.

11. #2 Comparing 15-Object Test versions
Simva used to compare two versions:
● Pre-post computer-based tests
handled by Simva
● Additional metadata information of
game version used
➢ Results from experiments showed that A and B are equivalent, and that
computer-based test yields similar results as the traditional paper-based

12. #3 First Aid Game to teach first-aid techniques
Game-like simulation to teach first-aid
techniques.
➔ 12-16 years old
➔ Previously validated in a pre-post
paper-based experiment with a
control group
➔ Game proven to teach knowledge
Ref: Cristina Alonso-Fernández, Ana Rus Cano, Antonio Calvo-Morata, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Lessons learned
applying learning analytics to assess serious games. Computers in Human Behavior. DOI: 10.1016/j.chb.2019.05.036.

13. #3 First Aid Game recall with GLA
New set of experiments with +300
students and collecting GLA data
Additional recall experiment a few
weeks later.
Simva used to maintain all data
together via re-used tokens (kept by
teachers)
➢ Results from experiments showed that students indeed recall knowledge
learned playing First Aid Game

14. Discussion
Simva simplifies serious games validation and players assessment
● Ensuring users privacy
○ Pseudo-anonymization technique
○ No personal information collected
● Collecting different data sources
○ All data per participant linked together
○ Simplifies later analysis
● Moving from pre-post to GLA data
○ Further insight obtained from gameplay interactions
○ Using standard data format (e.g. xAPI-SG)

15. Conclusions
● Pre-post experiments still the common way to
evaluate serious games
● Simva aims to simplify these costly experiments
○ Managing players groups, access & progress
○ Dealing with privacy
○ Linking questionnaires & in-game interactions
○ Simplifying collection, storage and analysis
○ Providing metadata features
Further Simva reference: Iván Pérez-Colado, Antonio Calvo-Morata, Cristina Alonso-Fernández, Manuel Freire, Iván Martínez-Ortiz, Baltasar Fernández-Manjón (2019):
Simva: Simplifying the scientific validation of serious games. 19th IEEE International Conference on Advanced Learning Technologies (ICALT)

16. Conclusions
Future research with game learning analytics data:
● Need to increase sample size
● Recommended to use standard formats
● Define data to be collected early in game design
See more details in the literature review:
Cristina Alonso-Fernández, Antonio Calvo-Morata, Manuel Freire, Iván
Martínez-Ortiz, Baltasar Fernández-Manjón (2019): Applications of data
science to game learning analytics data: a systematic literature review.
Computers & Education (in press)

17. Thank you!
Any questions?
Cristina Alonso-Fernández
@calonsofdez
calonsofernandez@ucm.es