Tish Shute, Director AR/VR, Corporate Technology Strategy, Huawei A talk from Inspire Track at AWE USA 2017 - the largest conference for AR+VR in Santa Clara, California May 31- June 2, 2017. XR is intelligent and interactive connecting real humans and physical objects with digital agents and entities. VR/AR will evolve into XR to become the future interface for Cloud Computing, IoT, Big Data, Prediction, Self-driving cars, Personalized Medicine, Robots, Drones, Cryptocurrency, Smart Cities, and AI. Social VR and AR will connect people in new and powerful ways but XR will connect the intelligence of people to the intelligence of machines in a space shared and understood by both. This talk will look at this new and intimate relationship between humans and intelligent agents. http://AugmentedWorldExpo.com

1. Tish Shute
Dir. AR/VR, Corporate Technology Strategy, Huawei USA
Co- Founder of Augmented World Expo and Augmented Reality.org
(XR)The Future of Intelligence:

2. Highlights of some of my previous work in AR/VR...
Co-Founded Augmented World Expo 2009 - present. Augmented
World Expo 2017, is the 8th year. Over 3000 attendees and 200
Exhibitors in AR, VR and Wearable Tech)
Creating Visual Effects for Film and Television - on the left The Elicon. 6DOF
Motion Control Robot. In 1991, my first start up NPlusOne‟s real time software for
motion control cameras like the Elicon and optical printers was used to create
visual effects for Little Shop of Horrors and Star Trek V. On the right, working on
the Interpreter with Sidney Pollack.
2011 - 2015 worked with Will Wright (creator of The Sims, the
largest selling PC Game in history) at Stupid Fun Club creating
next generation entertainment - smart toys, social robots, and
augmented reality television and games
2012 - 2014, I worked with Will Wright as part of the
Syntertainment founding team creating a new generation of
personalized mobile social experiences.

3. Biggest trend in AR/VR since 2010:
From Mobile First to AI First:
Bruce Sterling, AWE, 2010,
“At the 9am of the augmented reality industry…”
Bruce Sterling, June 2010, “The smartphone biz makes the personal
computer business look like a Victorian railroad..”
“AR has arisen directly from modern technical factors that just didn‟t use
to exist. It‟s made from shiny new parts, and is truly a child of the twenty-
teens, a genuine digital native. It‟s a little kid and it has to walk before it
can run, but it‟s great to see it walking.”
Satya Nadella, June 2016, “We‟ve been living in a world that has
slowly become mobile-first, he explained, but we‟re moving to one
that is ultimately AI-first.”
Microsoft CEO Satya Nadella

4. Augmentation ...the upgrading of human beings
“Up till now increasing human power relied mainly on upgrading our external
tools. In the future it may rely more on upgrading the human body and mind, or on
merging directly with our tools.” Harari, Yuval Noah

5. The upgrading of humans may follow any of three paths
Cyborg Engineering
Biological Engineering
The Engineering of Non-organic Beings.
Harari, Yuval Noah (2017-02-21). Homo Deus: A Brief History of Tomorrow (p. 43). HarperCollins.
Kindle Edition.

6. Nadella‟s 10 Laws of AI
1. AI must be designed to assist humanity. Nadella says that machines that work alongside humans should
do "dangerous work like mining" but still "respect human autonomy."
2. AI must be transparent. "We want not just intelligent machines but intelligible machines," says Nadella.
"People should have an understanding of how the technology sees and analyzes the world."
3. AI must maximize efficiencies without destroying the dignity of people. "We need broader, deeper, and
more diverse engagement of populations in the design of these systems. The tech industry should not
dictate the values and virtues of this future."
4. AI must be designed for intelligent privacy. Nadella asks for "sophisticated protections that secure
personal and group information."
5. AI must have algorithmic accountability. So that "humans can undo unintended harm."
6. AI must guard against bias. "Proper and representative research" should be used to make sure AI doesn't
discriminate against people (like humans do).
1. It‟s critical for humans to have empathy.
2. It‟s critical for humans to have education.
3. The need for human creativity won‟t change.
4. A human has to be ultimately accountable for the outcome of a computer-generated diagnosis or decision

7. #4 Intelligent Privacy
Federated Learning for privacy, smarter models, & low latency
(Google Research)
in addition to providing an update to the shared model, the improved model on your phone can also be used
immediately, powering experiences personalized by the way you use your phone.
https://research.googleblog.com/2017/04/federated-learning-collaborative.html
Federated Learning enables mobile phones to collaboratively learn a shared prediction model while keeping all the
training data on device, decoupling the ability to do machine learning from the need to store the data in the cloud.

8. Merging Directly with our Tools:
The Holodeck happens at the intersection of VR and AI
Enhancing the capabilities of CGI with
Machine Perception
• Nvidia‟s Holodeck
• Photorealistic models,
• interactive physics
• collaboration
• Early Access in September
• Christian Koenig -
• Koenigsegg hyper car
• Inventory of parts
Page 8

9. Indexed websites and Apps
billions of users, millions of servers
HTML pages, multimedia
Networking humans, smart machines and virtual entities
Site 1 Site 2 Site 3 Site n
Indexed meta verse + Bots, virtual
humans, digital entities
billions of users, millions of servers
3D Content (reality and CG)
Digital Avatar
World 1 World 1 World 3 World n
VR Users AR Users Wearable
Natural Human Interfaces
(6DoF controller, eye tracking,
Hand gesture)
Low Latency
(<100ms)
Distributed
PC Mobile TV
2D Interfaces
(keyboard, mouse, touchscreen)
High Latency
(<200ms)
Centralized
In 10 years
HighFidelity.com

10. Social Intelligence and Augmented Reality Maps of the World,
Blaise Aguera Y Arcas. ARE 2010, TED 2010
“I'll just mention that here we're using this for telepresence, but you can equally well
use this on the spot, for augmented reality. When you use it on the spot, it means
that you're able to bring all of that metadata and information about the world to you.
So here, we're taking the extra step of also broadcasting it. That was being
broadcast, by the way, on a 4G network from the market.”
Are2010, photo from Chris Cameron’s Flickr stream Dynamic Augmented Reality Mappig

11. Social Intelligence Because Reality is Always Changing!
Page 11
Relocalization
(the problem of recalling revisited
scenes)
•Revisit the same location later
•Share between different users
Combine multiple maps in the cloud from
different users

12. Upgrading the body:
Cyborg Engineering at the intersection of Computer
Vision and AI
“It‟s pretty clear that the people who would weep for joy to have Augmented Reality
are people who need reality augmented as a prosthetic...in other words, so that they
can achieve an „everyday life‟”. Bruce Sterling, Augmented Reality Event, 2010
Oxsight - Augmented Reality to aid the visually impaired

13. “Extended Cognition - your glasses are there to become a
personal extension of how you see the world.”
Stephen Hicks, Co-Founder, OxSight
,Semantic object recognition to personalize experience

14. Basic Machine Perception = Basic Machine Creativity
https://www.ted.com/talks/blaise_aguera_y_arcas_how_computers_are_learning_to_be_creative#t-701406
"Perception and creativity are very intimately connected," Blaise Agüera y Arcas
says. "Any creature, any being that is able to do perceptual acts is also able to
create."

15. “Artificial intelligence will take over the user experience to analyze the reaction to
various experiences and then make changes on the fly in relation to those
assessments. UX will become a much more intuitive and personalized experience in
the next few years.” - Chalmers Brown
AI for Natural Interfaces to help determine a users‟ intent
https://www.forbes.com/sites/forbestechcouncil/2017/03/17/expect-voice-vr-and-ar-to-dominate-ux-
design/#74bd5ee0598c
AI as an intention amplification system
“What is the equivalent of Doug Englebart‟s mouse for VR?”
David Smith, IEEE Conf.

16. Next-Gen wear-all-day eyewear will be an extension of identity
From Silicon Photonics to
human photonics

17. Bernard Kress – display progression
The Era of Hyper Personal Computing:
Displays integrated with Human Perception
● new ways to personalize
experience , deep learning
and beyond
● computer vision, speech
recognition, natural
language processing, and
related technologies

18. Fashion Forward Wear-all-day Displays:
Consumer eyewear made possible with a new optical toolkit
“our newly stocked and updated toolbox,” Bernard Kress

19. AR will be the user interface to…
IoT
Cloud Computing
Big Data
Prediction
Self-driving cars
Personalized Medicine
Robots
Drones
Cryptocurrency
Smart Cities
AI

20. A dialogue without words between the driver and his vehicle.
http://continental-head-up-display.com/
XR is an Empathy Platform where humans, machines, and
digital agents can gain a new understanding of each other.

21. Learning to trust our self-driving cars:
AR & VR are essential to mediate our new relationship with
intelligent machines
If one of the driver assistance systems detects a relevant situation, virtual graphical information in the
AR-HUD makes the driver aware of this … In addition to the direct increase in safety, this form of dialog
is also a key technology for automated driving. The augmentation makes it easier for the person driving
to build up trust in the new driving functions.” Dr Richter is Principal Technical Expert in Optical
Technologies for Continental ...
Continental - Heads up Displays

22. XR mediates human and machine intelligence in a language of
perception understood by both
In XR things that can be perceived and acted on in the physical world will
have virtual counterparts that can be observed and acted upon by virtual
entities and physical humans. And vice versa - virtual entities can also have a
presence in the physical world where they can be observed and acted on by
humans and other virtual entities.

23. XR Puts the Universe in the
palm of your hand
XR Collapse time and geography
XR Shifts our perspective
There are no limits! XR doesn‟t need to be weighed down by
the anchor of „reality‟.

24. Humans and intelligent digital entities connected in new
and intimate everyday relationships

25. .
Engineering Non-Organic Beings in Virtual Reality
https://deepmind.com
/blog/open-sourcing-
deepmind-lab/
https://www.linkedin.c
om/in/pedro-
domingos-77b183
https://openai.com/blog/uni
verse/
https://www.microsoft.co
m/en-
us/research/project/proje
ct-malmo/
Agent Based AI Research

26. .
A Platform for Autonomous Agents: Deep Mind Lab
DeepMind Lab emphasizes navigation, memory, 3D vision from a first person
viewpoint, motor control, planning, strategy, time, and this is just the beginning!
“..fully autonomous agents that must learn for themselves what tasks to perform by exploring their environment. All
these factors make learning difficult. Each are considered frontier research questions in their own right. Putting
them all together in one platform, as we have, represents a significant new challenge for the field.”
https://deepmind.com/blog/open-sourcing-deepmind-lab/

27. Digital strategists to run “what if” scenarios on your potential
futures
Hit the experience button to check out your
strategist‟s top return.

28. Distributed processing and swarm/flock characteristics
Memory entangled between 2 (or more) nodes allows for fast secure communications
Q-bits allow for multi-valued logic and parallel processing to a degree never achieved before
Quantum Computing and XR
AI + HI will optimize the process of learning to learn

29. XR will extend people‟s cognitive abilities of perception and
reasoning, influencing their actions in physical & virtual realities
“Arrows 1 and 2 are still the perception and action of the
individual, I, as before. But I can wrap itself up including
objects in its environment, J, to create a better-thinking
agent. There is no reason why this process could not be
repeated by adding more objects, K, which may be less
immediately accessible than J. I am uneasy with the idea
that it is „mind‟ that is being extended here – using a
rather ambiguous dualist term; I prefer „extension of
agency‟ but that is less attention-grabbing.” Andy Clark
Andy Clark
Philosopher and Cognitive Scientist; Professor of Logic and Metaphysics, University of Edinburgh, UK;
Author: Surfing Uncertainty: Prediction, Action, and the Embodied Mind.
Extension of Agency

30. XR is the perfect domain for “predictavores”
Suitably primed, you can decode a garbled speech waveform, Mehau Kulyk/Getty
“We are not cognitive couch potatoes idly awaiting the next „input‟, so much as
proactive predictavores – nature‟s own guessing machines forever trying to stay one
step ahead by surfing the incoming waves of sensory stimulation”.
Surfing Uncertainty, Andy Clark
“..perception and cognition may not be as distinct as we once believed...what we know
and how we think can influence perception, which in turn modulates thought and
knowledge.” Anil Ananthaswamy

31. Are we closing in on the secret of how consciousness emerges
Christopher Anderson/Magnum Photos
“After all, it has been done at least once by evolution.” Gerald Edelman and Giulio
Tononi, A Universe of Consciousness
Surfing Uncertainty, Andy Clark
“A brain filled with such internal models
can generate sensory data on its own,
independent of actual sensations, allows
us to imagine, to dream and to time-travel
mentally.”

32. “When you expand the
nature of communication
you expand what it
means to be human [or
non-human]” David A.
Smith
Expanding what it means to be human

33. The Future that arises from the re-engineered human mind is
unimagineable..
“Any worthwhile prediction [of the future] must take into account
the ability to re-engineer human minds, and this is impossible.
Harari, Yuval Noah (2017-02-21). Homo Deus: A Brief History of Tomorrow (p. 46).
HarperCollins. Kindle Edition.
?

34. And from here? … transcending screens and language to
communicate with thought
• ,
“Ultimately I don’t want to wear anything.” Mary Lou Jepsen, Openwater

35. The co-evolution of Human Intelligence (HI) + Artificial Intelligence (AI) will shape the
future of humanity
Biological Engineering: The next generation of computing will
be intimately integrated with human systems of perception
and biology.
❖Neurolink, Elon Musk
❖Kernal, Bryan Johnson

36. And you thought VR sickness was challenging?
Neural Lace: Biological regulation and interfacing, communications, memory
storage, machine interfacing, awareness speed enhancement, Mind-state creation

37. Thank you!
@tishshute
tish.shute@gmail.com