An introductory lecture on Games User Research methods which was first given to students at Hanze University on the 9th of March 2011. This presentation was later turned into two articles on Gamasutra that can be read here: - Part 1: http://www.gamasutra.com/view/feature/169069/ - Part 2: http://www.gamasutra.com/view/feature/170332/

1. Games User Research
Ben Lewis-Evans

3. >Introduction
 Games User Research
- Fun & Awareness raising
- Emotion
- Awareness
 Methods
- Production
- Post-Production

4. >Games User Research
 Game testing traditionally
done by the QA/Test
department
- QA/Test are (usually)
experts at gaming
- The audience may not be
- QA/Test have an
investment in the game
- The audience does not
- Mainly looking for bugs

5. >Games User Research
 About the user
experience & fun
 What do you want to
know?
- Is the game fun?
- Does it raise
awareness?

6. >Fun
 What is fun?
 Well…
- Easy to use
- Challenging
- Emotional impact
- Engaging
- Compelling
- Relaxing
 It is subjective!

7. >Emotions & Feelings
High Activation
Scared - x x – Excited
Unpleasant Pleasant
Bored - x x – Relaxed
Low Activation

12. >Emotions & Feelings
Sometimes fun High Activation Usually fun
Scared - x x – Excited
Unpleasant Pleasant
Bored - x x – Relaxed
Almost never fun
Low Activation

13. >Awareness Raising
 What is this?

14. >Awareness Raising

16. >In production testing (aka
Playtesting)
 General points:
- Get representative
users (kids, 10-14)
- Make it clear that the
game is being tested,
NOT the user
- Work out what you
want to know before
you test

17. >In production testing (aka
Playtesting)
 General points cont.:
- Test as early as possible, it
is easier to fix problems
that way (then test again)
- Listen to problems, but not
necessarily solutions
- Not for balance, and bugs,
but for fun! (& raised
awareness)

18. >Methods
 Focus Groups
 Heuristic Evaluation
 Questionnaires, Surveys and
Interviews
 Observational studies
 Gameplay metrics
 Biometrics/psychophysiology
 Think out loud

19. >Focus groups
 6-10 people
 Lead by a facilitator
- Specific questions
 Try the game/discuss
potential ideas
 Talk about it

20. >Focus Groups
 Pros
- More people can = more feedback
- Gets everyone together in one place
- Follow up questions
- Good for discussing concepts
 Cons
- You need a good facilitator
- Strong voices may take over
- Too many “helpful” suggestions
- What people say is not what they do

21. >Heuristic Evaluation
 Expert evaluation (somewhat like a game
review)

22. >Heuristic Evaluation
 List of Heuristics:
• Are clear goals provided? • Is the game and the outcome fair?
• Are players rewards meaningful? • Is the game replayable?
• Does the player feel in control? • Is the AI visible, consistent, yet
• Is the game balanced? somewhat unpredictable?
• Is the first playthrough and first • Is the game too frustrating?
impression good? • Is the learning curve too steep or
• Is there a good story? too long?
• Does the game continue to progress • Emotional impact?
well? • Not too much boring repetition?
• Is the game consistent and • Can players recognise important
responsive? elements on screen?
• Is it clear why a player failed? • etc…
• Are their variable difficulty levels?
From Christina et al 2009

23. >Heuristics Evaluation
 Pros
- Smaller numbers
- Experts are experts
 Cons
- You need experts
- Which heuristics to
pick?
- Experts are experts

24. >Questionnaires, Surveys &
Interviews
 During gameplay (at or after
set moments)
 After gameplay
 Ask specifically for what
interests you (don’t forget
about the raising awareness
part)
- But also allow for some
open ended answers

26. >Questionnaires, Surveys & Interviews
 Some pre-existing questionnaires:
- Game Experience Questionnaire (GEQ)
- http://www.gamexplab.nl/
- The Computer System Usability
Questionnaire (if modified)
- http://oldwww.acm.org/perlman/question.cg
 If you use them modify them to fit your
particular game & what you want to know

27. >Questionnaires, Surveys & Interviews
 For Emotion – The Affect Grid
Russell, Weiss & Mendelsohn, 1989

28. >Questionnaires design
 Order of questions
 Use clear, concise everyday, simple language
- Avoid jargon
- Don’t be vague
 Avoid asking duplicate questions
- The same question in a different way
 Avoid questions that are phrased negatively
- “I don’t like the jumping” agree -> disagree
- “I like the jumping” agree -> disagree

29. >Examples of bad questions
 Leading
”Now that you have had fun playing our
game, which was your favorite level?”
Level 1
Level 2
Level 3
Level 4

30. >Examples of bad questions
 Double(7)barrelled:
"Should there be a reform of our justice
system placing greater emphasis on the
needs of victims, providing restitution and
compensation for them and imposing
minimum sentences and hard labour for all
serious violent offences?”
Yes No

31. >Examples of bad questions
 Loaded, ill defined and misleading:
"Should a smack as part of good parental
correction be a criminal offence in New
Zealand?"
Yes No
[The question] "could have been written by Dr Seuss – this isn't Green
Eggs and Ham, this is yes means no and no means yes, but we're all
meant to understand what the referendum means. I think it's ridiculous
myself.” - PM John Key

32. >Questionnaires design
 Type of question
- Open
- Closed

33. >Questionnaires design
 Type of question
- Open
- “What did you like about level 1?”
- “Why might a child not be able to come to
school?”
- “How old are you?”
- Closed

34. >Questionnaires design
 Closed questions
- Dichotomous scale
- Yes/No

36. >Questionnaires design
 Closed questions:
- Dichotomous scale
- Continuous scale

38. >Questionnaires design
 Closed questions:
- Dichotomous scale
- Continuous scale
- Interval scale

40. >Questionnaires design
 Interval scale
- Numeric/categorical:
1 2 3 4 5
- Likert:
1 2 3 4 5
Disagree Agree
- Semantic:
1 2 3 4 5
Poor Good

41. >Questionnaires design
 Interval scale
- Unipolar:
1 2 3 4 5
Not Exciting Very Exciting
- Bipolar:
1 2 3 4 5
Very Boring Very Exciting

42. >Questionnaires, Surveys & Interviews
 Questionnaires & Surveys
- Pros
- Consistent
- Quantifiable
- Fast
- Good for testing raised awareness
- Cons
- Can lack follow up
- Not objective
- Need a large(ish) sample

43. >Questionnaires, Surveys &
Interviews
 Interviews
- Pros
- Rich data
- Can follow up
- Cons
- Less quantifiable
- Time consuming
- Not objective

44. >Observation studies
 Watch/Record through
video
 Either with a facilitator
or without
- Facilitator must be as
hands off as possible
 Watch faces/body for
emotion
 Only write what you
actually see!

45. >Observation studies
 Pros
- Objective data
- i.e. You see what players actually DO,
not what they say they would do
- Facilitator can help if really needed
 Cons
- Time consuming to analyse video
- Training required to get the best out of
observation (especially for emotion)
- Avoid Observer Bias

46. >Gameplay Metrics
 Observation via
the game data
- Number of
incidents
- Where and when
they occurred
and with who or
what?

47. Heatmap of Kills

48. Heatmap of Deaths

50. >Gameplay Metrics Deaths
 Pros
- Objective data
- See trends
 Cons
- Time consuming
Kills
- No subjective feedback/context
- Needs larger sample sizes
- Data overload

51. >Biometrics/psychophysiology
 Measuring body signals:
- From the Brain (EEG), the
Heart (EKG), the muscles
(EMG), the eyes
(eyetracking), the skin
(EDA), etc
 The body gives clues into
cognition, and emotion

52. >EDA Game Research
 Dying is fun?
 Ravaja et al (2008)
- EDA increased for
- Opponent Killed
- Player Killed
- EMG (Zygomatic
and Orbicularis)
- Increased for
longer when
player killed

54. >Biometrics/psychophysiology
 Pros
- Gives objective quantifiable data
- Allows for continuous data
recording
 Cons
- Invasive
- Costs a lot of time & money to use
& analyse
- Problems with specificity, artefacts,
inference and validity

55. >Think out loud
 An expansion of observation
 Players play the game and
talk about what they are
thinking as they go along
- Observe gameplay, and
note down what they say &
when they say it
- Do NOT prompt them, and
do NOT correct them

56. >Think out loud
 Pros
- Gain an idea what players
are thinking & feeling
- May give unexpected
insight
- Could test raised
awareness too
 Cons
- Unnatural
- Subjective

57. >In production testing (aka Playtesting)
 Many options
 In your case I recommend:
- Observation
- Gives objective data, and may
provide insights (think out loud?)
- After play (between level) questionnaire
- Allows you to ask specific questions and
receive feedback
- Collect gameplay metrics as if you can
(Optional)

58. >In production testing (aka
Playtesting)
 Remember
- Don’t wait until the game is
almost finished
- It is easier to change things
early in the process
- Listen to what people say is
wrong/right, don’t worry too
much about what they
suggest to do to fix it
- You are the game designer

59. >Post-production testing
 Raised awareness, it is too
late to be overly concerned
about fun
 Can’t observe, take
biometrics, run focus groups,
collect gameplay metrics
(game runs offline), etc
- On the ground observation?
Costly.

60. >Questionnaires
 Pen & Paper is one
option
 Better to build them into
the game
- Between levels or
within levels
- Multiple choice
questions
- Fill in the blanks

62. >Questionnaires
 In game questions
- Require completion to continue & provide
rewards for correct answers
- Try to make them fun too
- i.e playtest them
- Have to get the data back somehow though
- Uploaded when connected to the web?
- Teachers & students can access a report?

66. >In game behavioural modelling
 Raised awareness is fine
 Actually doing behaviour is
better
 So, if you can, build it into the
game
- Require the awareness you are
raising to be used to progress
in the game
- Not just through answering
questionaires
- But by DOING

67. http://www.peta.org/interactive/games/default.aspx

68. >Going further than Awareness?
 Is there any way you can monitor behaviour
change?
- New questions on later play throughs?
- Teacher/adult report system?
- Tied to in-game achievements/rewards?

69. >Summary
 Playtest early, playtest as often as you can
- With yourselves
- But also with players
- Behavioural observation allows for good
data with small numbers for playtesting
 Build in awareness testing
- Require it to progress
- Try and make it fun too

70. Thank you
b.lewis.evans@rug.nl

71. All images used in this
presentation belong to their
respective copyright holders.
If an image belongs to you
and you wish it removed
please notify me at
b.lewis.evans@rug.nl