The Second orientation material for the WBA Hackathon 2017

1. The Second Orientation
for the WBA Hackathon 2017
Material to be Distributed
2017-07-23

2. WBAI理念
Vision: to create a world in which AI exists in
harmony with humanity
Values
Study: Deepen and spread our expertise
Imagine: Broaden our views through public dialogue
Build: Create AGI through open collaboration
Mission: to promote the open
development of WBA
Basic Ideas

3. The Third Whole Brain Architecture Hackathon
Let’s Create AGI
Prototypes!
Wake up,
Hippocampus!
Hackathon:
September
16th -18th, 2017
Venue: φcafe, Tokyo
End of
Registration:
August 8th

4. Hiroshi Yamakawa
The Aim of the Hackathon

5. Human Intelligent Features
• Having values
• Having survival capablity
• Being conscious
• Having an Ego
• Versatile for various tasks
→ General Intelligence
• Etc.
GI that WBAI aims for is one of
the intelligent characteristics that
the current AI has not realized at
the human level.

6. Evaluating Generality (Utility)
Tasks to be solved at a satisfactory level
Task
Areas
Satisfactory
Level

7. Evaluating Generality (Utility)
Tasks to be solved at a satisfactory level
7
Task
Areas
Satisfactory
Level

8. Advances in DL
Learning from
the Brain
Nengo
(2015〜)Symbol
Emergence in
Robotics
A Map of AGI Developing Organizations
Engineering
Realization
Neocortex
oriented
Whole Brain
oriented

9. ML
Knowledge
Design
Connectome
Two-photon
Imaging
The Brain is ready to serve for AI!
ANNAI
(Cog. Sci)
Measuring
in Neuro-
science
Screws
Pistons
Automobile
Analogy
Overall
Design
Human
Brain
Engine
MacroMesoMicro
Higher-level
Description
Language
Whole
Brain ANN
model
Whole
Brain
Cognitive
Architecture
McCulloch
& Pitts
Model
Programming
Language
Neocortical
Areas
Neurons
Local
Neural
Circuits
Electrodes
fMRI, EEG
Deep Learning
Unified
Understanding
Multi-
layered
perceptron
Perceptron
Constraining
AI Architecture

10. WBA can be the fastest path to AGI.
‘to create a human-like artificial general
intelligence (AGI) by learning from the
architecture of the entire brain.’
An anatomical illustration from
Sobotta's Human Anatomy (1908)

11. Brain-Inspired Open Platform Strategy＠WBAI
Systematizing
catch-up to
reproduce
(verify)
advancing
technologies
Collaborative
development on a
brain-inspired open
platform, enabling
technological
integration
Rapid Advances in technologies around AGI
(e.g., @DeepMind)
To “harmonize AGI with human beings” by
democratizing AI technologies

12. Hackathon Teams & Sample Code
Neuroscientific Info.
(e.g., connectome)
Virtual Mouse
Virtual Experiment Env. (Unity)
Architecture
Description
Middleware
(BriCA Core ver.1)
ML Modules
(Refactored DQN)
・Neocortex: CNN
・BG: Q-Learning
・Hippocampus:
Episodic replay
Virtual Task Env.
ROS-like, light-
weight, multi-
platform with
virtual time
support
Engineer
Architect
Neuroscientist

13. BriCA Language Description (in JSON)
"Modules" : [{
"Name" : "BriCA1.MainModule",
"Ports" : [ "Port1", "Port2" ],
"SuperModule" : "SuperMainModule",
},
{
"Name" : "SuperMainModule",
"Ports" : [ "PortS1", "PortS2" ],
}
],
"Connections" : [{
"Name" : "Con3",
"FromModule" : "SuperMainModule",
"FromPort" : "PortS1",
"ToModule" : "BriCA1.MainModule",
"ToPort" : "Port1",
},
{
"Name" : "Con4",
"FromModule" : "BriCA1.MainModule",
"FromPort" : "Port2",
"ToModule" : "SuperMainModule",
"ToPort" : "PortS2”
]
}
Basic elements for architecture
–Modules
–Ports
–Connections
SuperMainModule
"Ports" : [{
"Name" : "Port1",
"Module" : "BriCA1.MainModule",
"Type" : "Input",
"Shape" : [3,1,1],
},
{
"Name" : "Port2",
"Module" : "BriCA1.MainModule",
"Type" : "Output",
"Shape" : [3,1,1],
….
BriCA1.MainModule
Port1 Port2
PortS1
Con3
PortS2
Con4

14. BriCA Language for Brain-inspired Platform Description
BriCA Language: DSL to describe the structure of
cognitive architecture
Points:
• Bridging neuroscientific findings (connectome) & AI
• Basis for collaborative development of architecture
– Combining ML Modules (ANNs, graphical models)
– Middleware independent (Brica Core, ROS, etc.)

15. Wake Up! Hippocampus
Major functions of the Hippocampus:
• Episodic Memory ⇒ Long-term Memory
• Navigation／Self Location
• At the top of the perception

16. Overall WBA for the Hackathon
(Containing Hippocampus)

17. The Aim of the Hackathon
• Goal:
– To make the first working WBA
• Sample Code:
– Deep Reinforcement Learning (DQN) refactored for
a simple WBA
– The Hippocampus module works as the replay
memory of DQN.
• To do in the Hackathon:
– Implement hippocampus functions such as episodic
memory and space cognition by modifying the
sample code

18. H. Mizutani
Hackathon Tasks

19. Hippocampus & the realization of AGI
• There have been partial implementations for
hippocampal functions (listed below) ,
but not done on a single system.
・Episodic Memory ・One-shot Learning ・Working Memoryー
・Space Cognition ・Self Location ・Interaction with Environment
・Transfer Learning ・Efficient Learning ・Decision Making
・Action Generation ・Attention ・Planning ・Imagination
・Intuition ・Consciousness
• Hackathon participants will develop brain-inspired
cognitive architecture, which could be a prototype
of AGI by implementing the Hippocampus.

20. Hippocampus: the Seat of Memory
Major functions of Hippocampus
• Episodic Memory
• Space Cognition, Navigation
–Place Cells: fire only when the animal passes particular locations in the
environment.
–Grid Cells: fire when the animal comes to certain places located at the vertices
of triangles in the environment
–Head Direction Cells: fire according to the direction of the head
Hippocampus
Laszlo Seress
©CC-BY-SA
We offer Maze Tasks that cannot be
solved without Hippocampal functions

21. Provided Maze Tasks

22. Provided Maze Task (Place Conditioning)
• Linear Maze Task
• The linear track has patterns on the walls and floor and
reward points indicated by blue, green, and red.
• The mouse starts from an end of the track and receives
a reward if it waits at a reward point for two seconds.
• A reward is placed at a presented green spot at the
center. The mouse receives a reward if it waits at the
spot for two seconds.
– Q: how to learn by utilizing place cells and projections to/from
the cortex? (Masaaki S et al., eNeuro 2017)
• Re-learning by placing rewards at other spots (blue or
red) .
– Q: What kind of cell activities (e.g., of place cells)
facilitates re-learning?

23. Evaluation Task Styles
• Systems developed in the hackathon will be
evaluated with tasks on an environment
simulator (Unity/LIS).
• While the following task styles are available,
a system with higher generality to realize
multiple tasks and functions will be evaluated
highly.
• Provided Tasks
–Provided by WBAI
• Free Tasks
–Set by participants

24. Evaluation (Judging) Policies
• Developed systems will be evaluated by juries with
the following criteria:
–Neuroscientific reality
• Realization of various cognitive functions on hippocampus
• Neural connectivity between hippocampus and other brain
areas
• Correspondence with neuronal activities of hippocampus
and other brain areas
–Engineering utility
–Originality
• Systems utilizing the software provided by WBAI,
such as BriCA, WBCA, BiCAmon will get extra points.
• The winner will be awarded with cash prize.

25. An example of hippocampal modeling (robot navigation)
Tang et al., Neural Networks archive
Volume 87: 27-37 (2017)
Pro: Navigation with hippocampal neural network.
Con: an implementation for the specific task.
’s model

26. An example of hippocampal modeling (Place Cells)
Lőrincz A. and Szirtes G.
Neural Networks 22 (2009) 738–747
Pro: Reproducing Place Cells
Con: Neuroscientific reality of the algorithm is debatable.
’s model
ICA was used for pattern separation.

27. OpenRatSLAM (Self Location/Navigation)
Pro: Self Location
Con: Connections within the hippocampus are not reproduced.

28. CA1
EC(MEC/LEC)
GC
CA3
Ⅴ&Ⅵ
Ⅲ
Ⅱ
Coordinate
transformatio
n
Goal state
P
State
prediction
error
Pattern
separation
Newborn
cell
MC
Generate
intention
sequence
Sb
Nucleus
accumbens,
Medial septum
Self-position
estimation
Deep
Shallow
PER/PO
R
Unimodal/
Polymodal
Ctx
Generate
intention
PreSb
ParaSb
P
P
Mammillary
bodies, ATN of
thalamus
Abstraction
of intention
Medial
septu
m
HD, B, P, G, Band-like
HD
HD,B,G
HD, B,P, G
HD, B, P
HD,B,G
B,G
Current state
Global current state (MEC only)
Global goal state
Shallow
Deep
Shallow
Shallow
HD: Head-direction cell, B： Border cell, G: Grid cell, P: Place
cell
Abstraction of
intention
HP
Recurrent
network
Spatial depended cell (※ Notation in EC exists only in MEC)
A Hippocampal Model by (Yamakawa, JSAI2015)
Gatsby-Kaken Joint Workshop on AI and Neuroscience
Pro: covering many sub-modules; Con: no implementation.

29. Check sheet
Episodic Memory Space Cognition Place Cells Hippocampal
Circuits
A STAR NG ? NG ✔
Lorincz NG NG ✔ ?
RatSLAM NG ✔ NG NG

30. The points of the Hackathon
• While Neocortex Module, Basal Ganglia Module,
& Hippocampus Module are offered, the
Hippocampus Module only implements
experience replay.
• Modules & connections among them are
specified based on connectome.
Participants are to compete for solving tasks by
implementing Hippocampus Modules.
• If an AI system realizes multiple cognitive
functions here, it would advance AGI
development.

31. N. Sakai & M. Ueno
The System

32. Agenda
1. Systems Summary
2. Systems Requirement & Environment Building
3. Products & supports offered by WBAI
4. Demonstration & comments

33. 1. Systems Summary
Demand action direction from Unity Task Environment to Agent
Unity Environment
Participant PC
Agent Server
Images from Agent’s
point of view, End
signal, etc.
Next Action
HTTP Async.
Com.
? Async.
Conn.:
Virtual
Time
Scheduler
Neocortex:
Feature Extractor
Component
Hippocampus:
Experience Component
Basal Ganglia:
Q-net Component
BriCA:
Cognitive Architecture Framework
ML modules connected based on WBRA
Component Connection
Setting

34. 2. Systems Requirement & Environment
Building
Recommende
d
Environment
Development
Environment
How to run
• CPU: Intel i5 or later
• Memory: 8G or more
• Download standard Unity IDE
• Download Python-enabled IDE
• Start the Python Agent with a command
• Run with Unity IDE or with a shell script

35. 3. Products & supports offered from WBAI
Products
Supports
• Unity Task Environment
➢ Mazes, etc.
• Agent Server
Source Code
Documentation
• Glossary
• Architecture Summary
• Modification for the hackathon
• Environment Building
• Q&A on Slack channels
➢ Advisory team for task & technical questions
➢ Real-time response is not guaranteed.
➢ FAQ to be compiled

36. Q-Learning
To obtain Function Q to estimate the Value of Action a at State s;
Approximate optimal Q Function by updating the following
equation at each step.
Q (s, L) = Low
Q (s, R) = High
Q (s, No-op) = Low
s: State , a: Action (pressing button, etc) , r: reward (game score, etc)
Preprocessed s: (self location, ball position)

37. Feature Extraction with DNN
37
End-to-end learning from input to estimation with a multi-layer
neural network enables feature extraction for the task.
No human feature design is necessary.
(Common use of CNN in image processing)
https:
//www.slideshare.net/nlab_utokyo/
deep-learning-49182466
Feature expression obtained by learning

38. DQN (Deep Q-Network)
38
The Q-function of Q-Learning is approximated with
DNN
By end-to-end learning with DNN, the value can be
estimated directly from images.
Value
DEEP
https:
//www.slideshare.net/nlab
_utokyo/deep-learning-
49182466

39. observation
- RGB Image
277*277
- depth Image 32*32
odometry
Velocity & angular
velocity
Episode end signal
reward
- bool
- int value
Unity Design – What is sent to the Agent

40. Switching Tasks
- Tasks are designed
for Unity scenes．
- After achieving the
task for a certain times,
the Agent moves to
the next task.
- Environment waits for
Agent’s action.
Unity Design – Task Progress

41. Hippocampal I/O based on Connectome
(Containing Hippocampus)

42. Agent Design
[velocity, angular velocity]
action value
(0, 1 or 2)

43. BriCA Core and Timing
BriCA Core:
• Framework for constructing brain-inspired
cognitive architecture
• Can connect & integrate ML modules in an
asynchronous manner.
• The time & timing of ML module execution can be
designed with its time scheduler.
In this hackathon, as Unity and BriCA work synchronously,
if the agent uses too much time, the system would not
work smoothly.

44. Evaluation of the neuroscientific reality
• Realization of various cognitive functions on hippocampus
– E.g.:
• The Agent must wait in reward areas for a moment.
• The Agent may locate itself from landmark information
without the information on wall color to obtain rewards.
• Neuro-connectivity between hippocampus and other brain
areas
– The proposed model should be justified in the presentation.
• Correspondence with neuronal activities of hippocampus
and other brain areas
– Activities of reproduced Place Cells, Grid Cells and Head Direction
Cells will be evaluated.

45. H. Mizutani
How to Participate

46. Timeline till the Hackathon
• 07-23: 2nd Orientation
• 08-08: Due Date for Registration
– From the Registration Form
https: //goo.gl/forms/PqAPlNoln53nyCG92
– Please make “WBAhackathon2017” folder in your team Github.
• 08-10: Acceptance notice
• 08-18: Sample Code to be published
• 08-19: Sample Code orientation & presentation by
participants
• 09-16
– Opening: 10h JST
Talk by Kenji Doya (11h)
Reception in the evening (talking on the hippocampus)
• 09-18
– End of work: Noon
– Review, Presentation, and Awarding: till 17h

47. The Third WBA Hackathon Summary
 Date: September 16th, 17th, and 18th, 2017
 Venue: Φ Café, Tokyo, Japan
 Cost
 Admission Free
 Transportation, Lodging & Reception fees will be paid by
the participants.
Students may receive financial aids (to be explained below).
Remote sites can be set up.

48. Lodging
Participants (up to 20 persons) can stay at a hotel
reserved by WBAI for the evenings of September
15th, 16th, and 17th.
In case you’d like to stay in the hotel (for 4,290JPY
per person/night), please inform us from the
registration form.

49. Other information
• Computation Environment
– BYOD (Bring your own device)
• Making your code public
We ask you to publish the code and presentation material
you made for the hackathon for open development of
WBA under:
– WBAI Contributor Agreement
– Apache License (Version 2.0)
on Github with README.
• Financial aids for students
Students meeting all the following conditions will be eligible for
financial aids with regard to travel and lodging expenses up to
JPY65,000/person.
– You should fully participate in the hackathon at Φ Cafe on
September 16th, 17th, and 18th.
– Your work at the hackathon with a Readme file should be
published on GitHub.