OpenAI presentation at OpenPOWER Community Meetup .

1. Robotics as the path towards AGI
by Wojciech Zaremba

2. About OpenAI
OpenAI is a
non-profit AI research company,
discovering and enacting
the path to safe
artificial general intelligence (AGI)

3. What is Artificial General Intelligence (AGI)?
An AGI agent
is an entity that can
match or surpass
human performance
on any mental task

4. Why robotics?

5. How to advance the AI ?
• New fields inspire new ideas. Seen in the past for:
• Vision : CNNs
• Speech, Translation : RNNs, LSTMs, Attention
• Games : RL
• Robotics-control hasn’t been affected by AI

6. Robotics is the perfect environment for AGI
• no obvious reward function for most tasks
• Hindsight experience replay
• small number of samples
• Transfer learning
• it’s been done before (evolution)

12. Transfer learning

14. Sim2Real
[Peng et al., 2017]
• Train robots in simulation.
• Transfer the policies to a real robot.

15. Key idea : simulation randomization
• Randomize simulation parameters
• Gravity
• Friction
• Torques
• Width and length of different geometric shapes
• Type of contact simulation
• Etc
• Train a policy that can adapt to all settings of simulation parameters.

16. This is a meta learning approach
• Policy quickly infers simulation parameters
• Could it infer the “simulation parameters” of the real world?

17. Baseline

18. Results

19. Our Belief: Self Play

20. • TD-Gammon (Tesauro, 1992)
• Incredibly old work:
• Q-learning + neural networks +
self-play
• Beat all humans, discovered
unconventional strategies that
were deemed to be better!
• Approach was dormant until DQN
for Atari
Self Play : TD-Gammon

21. Self Play : AlphaGo Zero

22. • Pure self play
• Popular competitive online
e-sports game
• Serious professional scene:
$140M awarded in prizes in 2016
• 5v5 is main variant; 1v1 also
played
• OpenAI beat all the pros 1v1
Self Play : Dota 2

23. • Simple environment → extremely
complex strategy
• Convert compute into data
• Perfect curriculum
Self Play : The Promise

24. • Environment is
simple, behavior is
very complex
• Pre-train “general
dexterity” by
competing against an
opponent
[Bansal et al., 2017]
Self Play for physicality and dexterity

25. Self Play : The Future
• Main open question: design the self play environment so that the result
will be useful to some external task

26. Self Play : Rapid increase in competence
• Dota 2 1v1: 5 months of
scaling and bug fixes
• Very rapid increase in
performance
• More compute = more
and better data

27. Question : can we train AGI via self play?
• Social life incentivizes evolution of intelligence
• “Because corvids and apes share these cognitive
tools, we argue that complex cognitive abilities
evolved multiple times in distantly related species
with vastly different brain structures in order to
solve similar socioecological problems.”
—Science, Vol. 306, Issue 5703, pp. 1903-1907
• Open-ended self play produces:
• Theory of mind, negotiation, social skills,
empathy, real language understanding
• Alignment issues

28. Similar rapid increase in competence?

29. Thank You
Visit openai.com for more information.
FOLLOW @OPENAI ON TWITTER