In this research, We explore novel Augmented Virtual Teleportation (AVT) methods based on the hybrid technologies of Augmented Reality (AR), Virtual Reality (VR), 3D live scene capturing, and multimodal interaction. Natural behavioral cues (Hand gestures, eye gaze, etc.) that are used in face-to-face communication play an essential role in effective collaboration. In contrast, most Mixed Reality (MR) remote collaboration systems mainly investigated computer-generated visual cues rendered as graphic objects or text for delivering instructions. In this research, we first study natural communication cues that people use in face-to-face collaboration. We then develop a novel remote collaboration system to enable people to communicate remotely as face-to-face. The system will contain two main parts: 1) Live scene capturing to enable real-time environment reconstruction and sharing of a user’s location, 2) Multimodal input such as gaze, gesture, and physiological signals to enhance remote communication. So far we have conducted two experiments to study the collaboration between a person with an AR interface and a remote user within a VR interface using multimodal input. We found that the remote collaboration system could provide a significantly stronger sense of co-presence for both the local and remote users by combing gaze and gesture cues than using the gaze cue alone. The combined cues were also rated significantly higher than using gaze cues alone in terms of the ease of conveying spatial actions. We plan to extend this system to study the effect of incorporating physiological signals in communication, especially in co-presence and usability. There are many potential applications of this research in different areas such as training, tourism, entertainment, gaming, and others. In conclusion, this thesis aims to study the effect of incorporating multimodal input and scene capture in remote collaboration systems in terms of presence, engagement, and task efficiency. This research will produce many benefits, such as design guidelines for future AVT systems, software libraries making it easy to create AVT systems, sample data-sets from experiments conducted, research publications, and more.
Using Physiological sensing and scene reconstruction in remote collaboration
1. 1Augmented Human Lab
Using Multimodal Input in Augmented
Virtual Teleportation
04th November 2020
Prasanth SasikumarSupervisors: Mark Billinghurst, Suranga Nanayakkara, Huidong Bai
3. 3
“To contribute to enhancing the ability to engage as a team from anywhere in the world”
Motivation
4. Outline
● Overview
● Progress so far:
○ Mixed Reality Remote Collaboration, Multimodal Input and Spatial Audio
○ Mixed Reality Training System and Physiological Sensing
● Future Works:
○ Physiological sensing
○ A framework for remote collaboration and Training
● Publications/Outcomes
● Timeline
○ Schedule and Resources.
4
Outline
6. 6
Overview
"What is the effect of including multimodal
cues in remote collaboration on presence,
engagement, and task efficiency?"
Research Question
7. 7
Overview
"How does capturing and sharing a user’s
surroundings with another person improve
the sense of presence and enhance
collaboration on physical tasks? (RQ1)"
8. 8
Overview
"How does sharing multi-modal inputs
(Physiological Sensing and Non-verbal
Communication) affect Social Presence in
remote collaboration? (RQ2)"
14. 16
User Study 1
● Compare user-centric and device- centric cues.
● Sharing gaze and gesture cues from the remote expert
to the local worker.
● Evaluation: performance, copresence and user
experience.
● Recruited 10 participants
○ Completion time,
○ Networked Mind Measure of Social Presence Questionnaire
○ MEC Spatial Presence Questionnaire
○ NASA Task Load Index Questionnaire
Wearable remote fusion (Pilot Study)
Remote expert guiding local worker to complete task
15. 17
User Study 1
● Findings:
○ Combing gaze and gesture cues, provide
significantly stronger sense of co-presence.
○ For both the local and remote users than using
the gaze cue alone.
○ The combined cues were also rated significantly
higher than gaze in terms of the ease of
conveying spatial actions.
● P. Sasikumar, L. Gao, H. Bai and M. Billinghurst, "Wearable RemoteFusion: A Mixed
Reality Remote Collaboration System with Local Eye Gaze and Remote Hand Gesture
Sharing," 2019 IEEE International Symposium on Mixed and Augmented Reality
Adjunct (ISMAR-Adjunct), Beijing, China, 2019, pp. 393-394, doi: 10.1109/ISMAR-
Adjunct.2019.000-3.
16. 18
User Study 2
● Live 3D panorama reconstruction for remote expert.
● Hand gesture and eye gaze sharing
● A formal user study with 24 participants
A User Study on MR Remote Collaboration
with Eye Gaze & Hand Gesture Sharing
System Overview
Camera Calibration
17. 19
User Study 2
● Findings:
○ Combing gaze and gesture cues, provide
significantly stronger sense of co-presence.
○ For both the local and remote users than using
the gaze cue alone.
○ The combined cues were also rated significantly
higher than gaze in terms of the ease of
conveying spatial actions.
● Huidong Bai, Prasanth Sasikumar, Jing Yang, and Mark Billinghurst. 2020. A User Study
on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing. In
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI
'20). Association for Computing Machinery, New York, NY, USA, 1–13.
DOI:https://doi.org/10.1145/3313831.3376550
18. 20
User Study 3
● The Effect of Spatial Auditory and Visual Cues in Mixed
Reality Remote Collaboration
● MR remote collaboration system that shares both spatial
auditory and visual cues
● Two user studies: (a) Audio cues, (b) Audio + Visual Cues.
Spatial Situation Model
NASA TLX
Spatial Audio
19. 21
User Study 3
● Improves spatial awareness in remote collaborative
tasks.
● Spatialized remote expert’s voice and auditory beacons
enabled local workers to locate occluded objects as
small as 2cm3 with significantly stronger spatial
perception.
Spatial Audio
View frustum & hand Gesture
NASA TLX
Lego brick layout
20. Working with Industry
● Engine Maintenance and Training.
● Issue: High training cost.
● Solution: Training system using volumetric playback
22
Industry Requirement
21. 23
User Study 4
MR Training System
● Study the impact of volumetric playback in a
MR spatial training system.
● Four visual instruction cues were compared;
○ Annotation
○ Hand Gestures
○ Avatar Representation
○ Volumetric Playback
System Overview
Engine
23. 25
User Study 4
MR Training System(Results)
● Volumetric instruction cues exhibits an increase
in co-presence and system usability while
reducing mental workload and frustration.
● To be published.
Mental Load (NASA TLX)
System Usability
24. 26
Industrial Survey
● Evaluated two long zoom interviews and 8
survey responses.
● Summarized the findings and drew design
implications.
● There was no standard tool to support physical
tasks remotely.
● Incorporating safety aspects into MR systems.
● Progress monitoring along with Improving
spatial accuracy and
● Better collaborative visualization of information.
25. 27
Progress so far:
● The experience uses real time depth sensing technology
and AR/VR displays to enable participants to view and
be part of tabletop conversations with people from
different cultural backgrounds, in a playful, explorative
and powerful way.
● Mairi Gunn, Huidong Bai, and Prasanth Sasikumar. 2019. Come to the Table! Haere mai ki te tēpu! In
SIGGRAPH Asia 2019 XR (SA '19). Association for Computing Machinery, New York, NY, USA, 4–5.
DOI:https://doi.org/10.1145/3355355.3361898
XR Demo (based of training system)
26. 28
Enhancing Remote Collaboration
● To study the usability of incorporating virtual worlds with
existing telecommunication systems.
● VIP programme.
● Literature review in progress.
Effect of MR in existing teleconferencing
solutions
Prototype MR system.
30. 32
Planned User Study
● Physiological responses are robust indicators of
autonomic nervous system activity that is related to
emotion.
● Evaluating heart rate and skin conductance.
● Adapt virtual environments based on sensory feedback.
● Hypothesis : "Real Time multimodal input provides a more
immersive, personalised,engaging experience in remote
collaboration systems".
● Two planned user studies.
Physiological sensing.
System Overview
31. 33
Physiological sensing.
System Overview
● User Study 1
○ 12 to 15 participants
○ Test physiological cues for emotion-relevant information.
○ Exposed to various collaborative tasks developed with
general available contextual variables.
○ Tasks will include (a) Pick and place, (b) Mechanical
assembly/disassembly and (c) Collaborative gaming.
● User Study 2
○ Evaluate the efficiency of the system in terms of task
completion time and co-presence.
○ Evaluation: Self-Assessment Manikin (SAM) questionnaire,
System Usability Questionnaire (SUS) and SSM.
● Outcome will answer part of RQ2 and RQ3.
32. 34
Progress so far:
● Dual EEG set-up that allowed us to use hyperscanning
to simultaneously record the neural activity of both
participants.
● We found that similar levels of inter-brain synchrony can
be elicited in the real-world and VR for the same task.
● A. Barde, N. Saffaryazdi, P. Withana, N. Patel, P. Sasikumar and M. Billinghurst, "Inter-Brain
Connectivity: Comparisons between Real and Virtual Environments using Hyperscanning," 2019 IEEE
International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Beijing, China,
2019, pp. 338-339, doi: 10.1109/ISMAR-Adjunct.2019.00-17.
HyperScanning
33. 35
Progress so far:
● Participants wear electroencephalography (EEG) head-
mounted displays to create music together using a
physical drum.
● Visualization reflects the synchronicity level while at the
same time trains the participants to create music
together, enriching the experience and performance.
● Ryo Hajika, Kunal Gupta, Prasanth Sasikumar, and Yun Suen Pai. 2019. HyperDrum: Interactive
Synchronous Drumming in Virtual Reality using Everyday Objects. In SIGGRAPH Asia 2019 XR (SA '19).
Association for Computing Machinery, New York, NY, USA, 15–16.
DOI:https://doi.org/10.1145/3355355.3361894
XR Demo (HyperDrum)
36. 38
End Game!
MiSo RC.
● Multimodal Input & Scene Reconstruction based Remote Collaboration system.
● Developing the framework would enable us to:
○ 24 to 30 participants.
○ Exposed to various collaborative tasks developed with general available contextual variables.
○ Tasks will include (a) Pick and place, (b) Mechanical assembly/disassembly and (c) Collaborative gaming.
● Hypothesis:
○ "Real-time multimodal input improves user experience in remote collaboration systems compared to a traditional RC
system".
● Outcome will answer RQ3.
38. Provisional goals
1. Complete a literature review of related work in (1) Use of Shared remote collaboration in AR/VR (2) Use of gaze
and gesture input in shared AR/VR (3) Rapid Scene Construction. Compile a report based on this literature review,
to the satisfaction of the advisory committee.
2. Meet end users from a target user group (e.g. utility company) and interview them to understand their needs for
remote collaboration. Write a report about this, to the satisfaction of the advisory committee.
3. Develop a system for shared remote collaboration based on scene capture, to the satisfaction of the advisory
committee.
4. Conduct a pilot test to study the effect of 3D scene sharing in remote collaboration, to the satisfaction of the
advisory committee.
40
Provisional goals
39. Publications
1. P. Sasikumar, L. Gao, H. Bai and M. Billinghurst, "Wearable RemoteFusion: A Mixed Reality Remote Collaboration System with Local Eye Gaze and
Remote Hand Gesture Sharing [Poster]," 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Beijing,
China, 2019, pp. 393-394, doi: 10.1109/ISMAR-Adjunct.2019.000-3..
2. Huidong. Bai, Prasanth Sasikumar, Jing Yang, and Mark Billinghurst. 2020. A User Study on Mixed Reality Remote Collaboration with Eye Gaze and
Hand Gesture Sharing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing
Machinery, New York, NY, USA, 1–13. DOI:https://doi.org/10.1145/3313831.3376550.
3. Yang, J., Sasikumar, P., Bai, H. et al. The effects of spatial auditory and visual cues on mixed reality remote collaboration. J Multimodal User
Interfaces (2020). https://doi.org/10.1007/s12193-020-00331-1
4. Hajika, R., Gupta, K. ,Sasikumar, P., and Pai, Y. S. (2019). HyperDrum: Interactive Synchronous Drumming in Virtual Reality using Everyday
Objects[Best XR demo paper runner up]. In SIGGRAPH Asia 2019 XR (pp. 15-16).
5. Mairi Gunn, Huidong Bai, and Prasanth Sasikumar. 2019. Come to the Table! Haere mai ki te tēpu! [Demo] In SIGGRAPH Asia 2019 XR (SA '19).
Association for Computing Machinery, New York, NY, USA, 4–5. DOI:https://doi.org/10.1145/3355355.3361898
6. A. Barde, N. Saffaryazdi, P. Withana, N. Patel, P. Sasikumar and M. Billinghurst, "Inter-Brain Connectivity: Comparisons between Real and Virtual
Environments using Hyperscanning [Poster]," 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct),
Beijing, China, 2019, pp. 338-339, doi: 10.1109/ISMAR-Adjunct.2019.00-17.
7. Pai, Y. S, Hajika, R., Gupta, K. , and Sasikumar, P. (2019). NeuralDrum: Perceiving Brain Synchronicity in XR Drumming accepted as a work for
SIGGRAPH Asia 2020 Technical Communications program. This will be presented on 20th November 2020.
41
Publications
41. 43Augmented Human Lab
Using Multimodal Input in Augmented
Virtual Teleportation
04th November 2020
Prasanth SasikumarSupervisors: Mark Billinghurst, Suranga Nanayakkara, Huidong Bai
ability to share the intimate information of human states and feelings derived from various sensors is an emergent paradigm called empathic computing
Explain that when we say remote collaboration, we mean - there is a remote expert and a local worker.
How we stich the cameras and how the world is aligned.
Technical Stuff Here
What is uuser and what is device
No of participants.
Tasks
Say that between the user centri ccues. Which one gives better perofomance.
Face to face as in real world communication.
Quick Mention about the gaze study.
We conducted this study in light of the increasing importance of the current pandemic era where travel is restricted. We collected feedback on Mixed Reality (MR) concepts for remote collaboration, and especially on the benefits, use cases features that MR systems for remote collaboration should have. We conducted and evaluated two long zoom interviews and 8 survey responses and summarized the findings and drew design implications. We found that teamwork and collaboration are essential for all the participants/organization and there were no standard tools to support physical tasks remotely. We described ideal use cases that augment existing solutions by enhancing safety, mitigating risks, improving accuracy and enabling better coordination.
Quick Mention
Quick Mention
Quick Mention
Quick Mention
Quick Mention
Quick Mention
Explain that when we say remote collaboration, we mean - there is a remote expert and a local worker.
With the rapid advancements in artificial intelligence technology to make consumer products smart, human trust on smart machines is an important factor while designing human-computer interactions.
For a user to rely on a system, she should be trusting the information provided by the system. If the task is completed as per user’s satisfaction, they will start trusting the system.
