DevEX - reference for building teams, processes, and platforms
Real-time Sensory Substitution to Enable Players who are Blind to Play Video games using Whole-Body Gestures
1. Real-time Sensory Substitution to Enable
Players who are Blind to Play Video games
using Whole-Body Gestures
Tony Morelli, Eelke Folmer - FDG’11, Bordeaux
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
2. Gesture Based Interaction
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
3. Reliance on Visual Cues
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
4. Reliance on Visual Cues
dodge
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
5. Reliance on Visual Cues
dodge
♫
♫
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
6. visual impairment & obesity
60,000 Blind Children in US More obese
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
7. Barriers to physical activity
Rely on sighted guide Safety
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
8. Exercise games
★ Moderate to Vigorous PA (MVPA) ★Independently
★60 min of MVPA daily (CDC) ★Safer
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
9. Research Question
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
10. Gestures w/o visual cues?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
11. Gestures w/o visual cues?
?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
12. Sensory Substitution
Visual
information
Haptic Audio
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
13. Sensory Substitution
closed captioning
Visual
information
Haptic Audio
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
14. Sensory Substitution
closed captioning
Visual
information
Haptic Audio
braille speech synthesis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
15. SS with Gestures?
Wii Sports VI Tennis
serve ♫
bounce ♫ Bu
zz
return
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
16. User studies
vigorous
moderate
light
VI Tennis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
17. User studies
vigorous
moderate
light
VI Tennis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
18. User studies
vigorous
moderate
Upper Body
light
VI Tennis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
19. User studies
vigorous
moderate
Upper Body
light
VI Tennis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
20. User studies
vigorous
moderate
light
VI Tennis
Whole Body
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
21. Whole Body Exergames
Yield Vigorous PA [Biddis & Irwin 2010]
Track Gestures using Computer Vision
Problems with Sensory Substitution
»Systems are Controller less
»Access to Source code Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
22. Real Time Sensory Substitution
Le*!!
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
23. Video Analysis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
24. Instrumentation
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
25. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
26. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
27. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
28. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
29. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
30. Real time Video Analysis
<SECTION>
<XMIN>100</XMIN>
<XMAX>300</XMAX>
<YMIN>150</YMIN>
<YMAX>300</YMAX>
<MINR>230</MINR>
<MAXR>999</MAXR>
<MING>230</MING>
<MAXG>999</MAXG>
<MINB>0</MINB>
<MAXB>100</MAXB>
<ACTION
<WIIMOTE>
<ID>0</ID>
<DURATION>-1</DURATION>
</WIIMOTE>
<AUDIO>
<ID>alert.wav</ID>
<LOOP>0</LOOP>
</AUDIO>
</ACTION>
</SECTION>
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
31. Real time Video Analysis
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
32. Evaluate
How efficient is RTSS?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
35. User Study
Anonymously recruited and tested
Evaluated 2 versions of the hurlde game
»Regular (Audio + Video)
»RTSS ( Audio + Haptic)
28 sighted participants
Novice players
Switch between versions a*er 3 players
12 people played Regular <-- data loss
16 people played RTSS
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
36. Results
version N success time
(seconds)
RTSS 12 0.39
(δ=0.26) 28.67
(δ=8.41)
Regular 16 0.37
(δ=0.32) 26.76
(δ=4.76)
No Significant Difference p>.05
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
37. Jump Timing Accuracy
Significant Difference p<0.05
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
38. Observations
Game Restarts
»False Starts
‣Regular: 2 players started too early
‣RTSS: None
»Out of Range
‣Regular: None
‣RTSS: 3 players out of range
Successful jump depends on pace
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
39. Limitations of RTSS
Kinect Sports Kinect Adventures
★ Predefined gestures ★ Different gestures
★ Fixed target ★ Random targets
★ 1 haptic cue ★ (directional) haptic cues?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
40. Future Work
Techniques for keeping
players in range and
facing the camera.
Multiplayer
Detecting cues/objects
Test with Blind Kids & Measure Energy Expenditure
Integrate RTSS in Console OS
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
41. Questions
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011
42. Questions
?
Player-Game Interaction Research
University of Nevada, Reno
Thursday, June 30, 2011