Nanja Smets, Corine Horsch, Mark Neerincx and Raymond Cuijpers on "Comparing Performance and Situation Awareness in USAR Unit Tasks in a virtual and real Environment" at ISCRAM 2013 in Baden-Baden.
10th International Conference on Information Systems for Crisis Response and Management
12-15 May 2013, Baden-Baden, Germany
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Comparing Performance and Situation Awareness in USAR Unit Tasks in a virtual and real Environment
1. Comparing performance and situation
awareness in USAR unit tasks in a virtual
and real environment
Nanja Smets, Corine Horsch, Mark Neerincx and Raymond Cuijpers
3. Introduction
Urban Search And Rescue
Live scenarios are often:
Costly, high personnel and
time, demands for preparation
execution, analysis and
debriefing
Use a virtual environment!
That is why we want to
compare the outcomes of an
evaluation in a virtual versus
‘real’ environment
May 15, 2013
3
4. Introduction: NIFTi project
Human robot cooperation
End-users involved
Each year an end-user evaluation
UGV
UAV
Operators
In-field rescuers
Mission Commander
…
NIFTi (www.nifti.eu) is funded by the EU FP7 Programme, ICT Cognitive Systems, Interaction, Robotics, Project #247870
May 15, 2013
4
5. Introduction: unit tasks
Unit tasks are:
Simple exercises
Independent of location and development phase
They examine the human-robot team
Convey the capabilities that are critical for the eventual operational
task
Have different levels
Performed every year at the end-user evaluation
Benchmarking
Predictive value for performance of end-user in scenario
May 15, 2013
5
7. Research question
Evaluate to what extent the unit tasks of Mioch and colleagues (2012)
in a virtual reality can represent the results of the unit tasks in a
physical experiment.
May 15, 2013
7
Unit tests in real
environment
Unit tests in
virtual
environment?
8. Method
Task: perform unit tasks in virtual environment
12 participants, within subjects
Virtual environment (UDK, USARSim)
P3AT robot with game controller
May 15, 2013
8
9. Hypotheses
The performance and situation awareness of the human-robot system
will be different between the real environment and the virtual
environment
We expect influences of age and gaming experience on performance
and SA
May 15, 2013
9
11. Results: Situation Awareness
May 15, 2013
11
Real world Virtual environment
Distancetowall
Unit task: stop before collision
Measured distance
Difference in measured
distance and distance
estimated by participant
Questionnaire on SA:
Better overall SA in real
than in virtual
environment
12. Conclusion: SA and Performance
Difference in performance and SA?
Only small differences found in performance in slalom and a trend in
detect objects maybe people were unaware of the shape of
the robot in the virtual environment?
Medium effect size in SA for stop before collision
Reported SA better in virtual than in real environment
May 15, 2013
12
13. Results and conclusion: individual differences
We expect influences of age and gaming experience on performance
and SA
Not found because of small age range in virtual environment
experiment?
May 15, 2013
13
14. Future work
No big differences found, it could be that the resemblance is close
enough in a tele-operation unit task like this
Continue to use, develop and test unit tasks and benchmark
development of robots
Virtual environments might be interesting for training of end-users with
robots
We will continue to use virtual reality in experiments with big number
of participants
May 15, 2013
14
15. Thank you for your attention!
Any questions?
May 15, 2013
15
NIFTi (www.nifti.eu) is funded by the EU FP7 Programme, ICT Cognitive Systems, Interaction, Robotics, Project #247870
Editor's Notes
Beschrijvenalgemeenprobleem van testen in complexe/gevaarlijkeomgevingen.
Narrow hallway and stop before collisionDetect objects (within 2 minutes find as many objects as possible)Slalom
During the slalom task participants in the virtual experiment bumped on average more into cones (M=5.1, SE=0.73), than participants in the real experiment (M=3.0, SE=0.41, t(16.7)=-2.48, p=.024, r =.5), see Figure 3. The other performance measures were not significantly different between the real and virtual environment in any task. Nonetheless, a medium effect size (r=.3) was found in the detect objects tasks where participants in the virtual environment bumped more into the walls (M=1.33, SE=.26) than in the real environment (M=.67, SE=.44).
The end questionnaire regarding overall SA showed that on average, participants reported a better overall SA in the real experiment (M=4.04, SE=.25), than participants in the virtual experiment (M=3.29, SE=.24, t(18)=2.13, p=.047, r=.45). , a medium effect size (r=.34) was found in the mismatch of distance (difference between the real distance to the wall and the estimated distance to the wall) between the real and virtual experiment in the collision task. Despite the non significant difference, the mismatch was bigger in the virtual environment (M=7.37, SE=5.35) than in the real environment (M=-1.00, SE= 4.39). A negative value means that the participant estimated that he was closer to the wall than he actually was.
. When an operator controls the robot in a real USAR situation he is physically not in the same room and has to rely on sensory input from the robot, which is similar to operation in VR. Even if differences between VE and reality exist, testing robots in virtual environments could still be useful, since comparisons of different conditions in VE seems to have the same results as the identical comparison in the field
