Integrating Established Learning Theories Into Online Programs Learning Outco...
Learning Assessment With Serious Games
1. Learning Assessment
with Serious Games
Collaboratory for Interactive Learning Research (C.I.L.R.)
Dept. of Curriculum & Instruction, COEHS
Southern Illinois University Carbondale
Carbondale, Illinois, U.S.A.
2. Performance-based Assessment
Monitoring of students' progress in relationship to
identified learner outcomes.
The Office of Technology Assessment of the U.S. Congress
defines performance assessment as “any form of testing
that requires a student to create an answer or a product
that demonstrates his or her knowledge or skills.”
This differs from traditional testing methods which
require a student to select a single correct answer or to fill
in the blank.
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
3. Characteristics of Effective
Performance Assessment
Students should be active participants, not passive
“selectors of the single right answer."
Intended outcomes should be clearly identified and should
guide the design of a performance task.
Students should be expected to demonstrate mastery of
those intended outcomes when responding to all facets of
the task.
Students must demonstrate their ability to apply their
knowledge and skills to reality-based situations and
scenarios.
A clear, logical set of performance-based activities that
students are expected to follow should be evident.
A clearly presented set of criteria should be available to
help judge the degree of proficiency in a student response.
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
4. Cognitive Learning Theory
Knowledge is constructed. Learning is a process of creating
personal meaning from new information and prior knowledge.
Learning has social components. Group work is valuable.
People perform better when they know the goal, see models,
and know how their performance compares to the standard.
There is great variety in learning styles, attention spans,
memory, developmental paces, and intelligences.
It's important to know when to use know when to use
knowledge, how to adapt it, how to manage one's own learning.
Motivation, effort, and self-esteem affect learning and
performance.
Learning isn't necessarily a linear progression of discrete skills.
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
5. Definitions of Assessments
Formal and informal ways of eliciting evidence of intended student
Assessment
learning
Any assessment for which the first priority in its design and practice
Assessment for Learning (AfL)
is to serve the purpose of promoting student learning
When the evidence of learning elicited by the assessment is actually
Formative (use of) Assessment
used to adapt teaching to meet the learning needs
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
6. Casual Games
Video game or online game targeted at a mass audience
Any type of gameplay
Fit any genre.
Simple rules
Lack of commitment required -- very little time
commitment
Need no special skills to play
Comparatively low production and distribution costs for
the producer
Played on a personal computer online in web browsers
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
7. Casual games
Based on Ludology Examples:
school of game design Sudoku
principles
Tetris, Minesweeper
Quick, fun, engaging Card games
Not story-based Casino games
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
8. Serious Games
Games for Education, Instruction & Training
Serious Play (Rieber, 1996), Serious Games (Abt, 1970)
Instructors » teachers (public school) » business trainers
Learners » students » employees
Salesman of the Month A Force More Powerful
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
9. Virtual Worlds/Complex Games
If story-based, then it is based on the
Naratology school of game design principles
The story provides the context for
interaction
The story “grounds” the experience
The storyline engages the mind/personality
Player becomes the “hero” in the plot
Objects in the environment are interact-
able and usually for a reason (plot-related)
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
10. Current Focus for Game Design
Current focus (of the industry) is in game
design
What comes after design & development?
Implementation and then evaluation
Design and evaluation (assessment) is
intertwined
Assessment methodology usually stem from
design
Game designers need to think about
assessment as they design games (NOW)
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
11. Assessment of Learning
Technology + Learner Learning?
Computers
Students/Trainees
(Serious)
Video Games
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
12. Problems of Assessment
Can we (teachers) know what they (learners) really
learned?
UNINTENDED CONTENTS? INTENDED CONTENTS?
“How do we get into the Black Box ?”
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
13. Game Studies / Game Research
European-based Scientific research
GAME STUDIES GAME RESEARCH
(Netherland) Problem/scenario-based
MIT
learning
(Language/linguistics) Pioneered by military-based
Recommendation why we
training
should study games »
Data-driven conclusions
Recommendation:
because it is a rising social
Why we should study games
phenomenon, and is To effect change in
interesting to society
Learning
Phenomenological
Education
Ethnographical
Performance
Feministic, etc.
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
14. Getting into the Black Box
Qualitative Analysis
Video-taping the whole event
Short (1-2 sessions)
Medium (a few weeks)
Long (a few months… even
years)
We need to double/triple the
time for data analysis
Limited generalization
Case Studies
Ethnography
Phenomenology
Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA
15. Measuring Change
Quantitative Analysis
Educators’ method:
Attitude surveys
Measuring effects of new
technology against
traditional teaching method
Pre-test» Intervention»Post-
test
Randomized testing
Better Methods?
Require “harder” statistics!
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
16. New Research Methods
Large data sets
DATA MINING RESEARCH FRAMEWORK
Conceptual Frameworks »
Sampling at regular intervals Test Models
Design for learning/
Tens of thousands/millions
data points
Data mining design for assessment
Learning framework
Analyzing for patterns in
data Design-based learning?
Assessment framework
Clustering of data (amateur
vs. experts) Criterion-Reference (for
Associative data mining games?)
Multimedia mining Information Trails
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
18. Tracking a Moving Object [Vehicle]
Vehicle tracking is a way of monitoring the locations,
movements, status and behavior of a vehicle or fleet of
vehicles.
This is achieved through a combination of an electronic
tracking device fitted in the vehicle, a method of
returning the data to the user and PC or web-based
software.
The data is turned into information by management
What tools in conjunction with a visual display
on computerized mapping software.
reportingcan be learned?
Driver 1 on schedule, lunch break: 2 eggs, 24
chips, 7 mushrooms and fried bread
Driver 3 helped friend move house
Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA
19. Tracking Devices
INFORMATION RECEIVED FROM DATA RECEIVED = EVIDENCE OF
TRACKING DEVICE / SOFTWARE VIOLATION » NOTIFICATION
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
20. Characteristics of Tracker
The tracker is a visual moderation and logging software product,
which enables meeting participants to keep track of their meeting
goals and the necessary steps to achieving them, as well as allowing
contributions to be anchored within the meeting log immediately.
The tracker graphically depicts the course of the current meeting
and locates it within the overall context. Tangential departures from
the meeting objectives can immediately be seen.
The course of the meeting is structured efficiently and becomes
traceable. At the end of a meeting, the minutes are already completed
and can immediately be approved by all participants.
For the most important meeting situations (such as project kick-off
meeting, information meeting, decision meeting, crisis meeting,
problem analysis meeting, etc.), the tracker offers proven agenda
templates which will ensure an efficient meeting process.
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
21. Agent Tracking
Tracking is the process of locating a moving object (or
several ones) in time.
Tracking a Tropical Depression Tracking the Position of an Object via GPS
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
22. Tracking a Moving Object [GPS]
A tracking unit is a device that determine the precise
location of an animal, person, or other asset to which it is
attached and to record the position of the asset at
The recorded location data can be stored within the
regular intervals.
tracking unit, or it may be transmitted to a central location
data base, or internet-connected computer.
This allows the asset’s location to be displayed against a
map backdrop either in real-time or when analyzing the
track later, using customized software.
Ref: http://en.wikipedia.org/wiki/GPS_tracking
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
23. What Can Be Learned
Tracking a live moving agent (sea turtle) in its habitat via a
tracking device attached to the agent
Christian Sebastian LOH [csloh@siu.edu] University
•
Ref: http://www.bonairenature.com/turtles/turtletracking.html
Christian Sebastian Loh, Ph.D. Southern Illinois Carbondale, USA
24. Components of Tracking
(A) PROCESS (B) TIMELINE
Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA
25. Data Processing
Tracking the movement of Sea Turtle: Process +
Timeline
Data Flow: Data Logger » Pusher » Puller » Store
Data Mining: Data Collection » Reduction »
Visualization
Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA
26. Foraging » Trails
foraging
Animals Food Source
Path Finding
(Trails)
start
• Pirolli, P., & Card, S. K. (1999). Information foraging. Psychological Review, 106(4), 643-675.
Information Foraging
• Chi, E. H., Pirolli, P., Chen, K., & Pitkow, J. (2001, March 31-April 4). Using information scent to model user
information needs and actions on the Web. Paper presented at the SIGCHI '01, Seattle, WA.
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
27. Serious Games & Assessment
(Serious)
Video Games
? (Verification
Assessment
of Learning)
Learning
Goal
Process of
Learning
start
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
28. Serious Games & Assessment
(Serious)
Video Games
? (Verification
Assessment
of Learning)
Information Trails
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
29. Information Nodes
Conceptually: Series of planned event hooks that are
embedded within an information ecology.
Operationally: Series of planned event hooks waiting to be
trigger by players’ avatars as they traverse a virtual game
environment. Once an event is triggered, a marker
will be placed on the trail to mark the location,
time, and outcome of the event.
Christian Sebastian Loh, Ph.D. Southern Illinois University Carbondale, USA
30. Information Trails
Conceptually: A series of agent-detectable markings left by
a moving agent within an information ecology.
Operationally, it is the long track of information markings
left behind by players’ avatars as they traverse a virtual
game environment (i.e. triggered event hooks)
Christian Sebastian LOH [csloh@siu.edu] • Collaboratory for InteractiveUniversity Carbondale,
Southern Illinois Learning Research
• Southern Illinois University Carbondale
Christian Sebastian Loh, Ph.D. USA
31. Tracking an Avatar
Avatar: a representation of self in a game environment
(An extension of self that behaves just like a live being in a
habitat)
Can a tracking device be attached on to an avatar, so as to
track the moving agent?
(Just like in the case of a vehicle or a turtle?)
Data (locations, status, behavior) about the avatar can then
be transmitted out of the game system, and be received and
stored in an external data store for later analysis
(Data logger » Push » Pull » Store)
Data can then be represented graphically, where goals and
behaviors can be scored as assessment
Christian Sebastian LOH [csloh@siu.edu] • Collaboratory for InteractiveUniversity Carbondale,
Southern Illinois Learning Research
• Southern Illinois University Carbondale
Christian Sebastian Loh, Ph.D. USA
32. Serious Games & Assessment
(Serious)
Video Games (Verification
Assessment
of Learning)
Data Data Visualization
data (Data Display)
Collection
Data Reduction
Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA
33. Serious Games & Assessment
Serious Games
(with embedded
Information Nodes)
(Verification
Assessment
of Learning)
Data Collection Data Visualization
(Information Trails) data (Data Display)
Data Reduction
Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA
34. Game Development
Game Developers’ Toolkit Game Mods
(Free, New Contents)
Commercial
(Serious)
Off-The-Shelf (COTS)
Video Games Data Store
Games
Persistent Worlds
(Free Multiplayer Games)
Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA
35. Embedding Information Trails
Game Developers’ Toolkit Game Mods
(Free, New Contents)
(Serious)
Commercial
Video Games
Off-The-Shelf (COTS)
Data Store
Games
Embedding Information Trails
Serious Game Developers:
into the Game Development Process
Persistent Worlds
(Free Multiplayer Games)
Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA
36. In Situ Data Collection
Assessment of Digital Game-based
Researchers:
Learning using Information Trails
Game Mods
(Free, New Contents)
Game Developers’ Toolkit
(Serious)
Video Games
Data Store
Commercial
Off-The-Shelf (COTS)
Games
Assessment of Learning
(Based on Learners’
Persistent Worlds
Behaviors, Decisions)
(Free Multiplayer Games)
Christian Sebastian Loh, Ph.D.
Christian Sebastian LOH [csloh@siu.edu] University
Southern Illinois
• Carbondale, USA