This is the slide used for PFI/PFN weekly seminar on June 18, 2015. Video (in Japanese): http://www.ustream.tv/recorded/64082997

1. Introduction to Chainer:
A Flexible Framework for Deep Learning
2015-‐‑‒06-‐‑‒18 PFI/PFN Weekly Seminar
Seiya Tokui (Preferred Networks)

2. Self-‐‑‒Introduction
l Seiya Tokui @beam2d (Twitter, GitHub)
l Researcher at Preferred Networks
l Main focus: machine learning
– Learning to Hash (master degree)
– Deep Learning, Representation Learning (current focus)
2

3. 3
A Powerful, Flexible, and Intuitive Framework of Neural Networks

4. Today I will introduce:
l The features of Chainer
l How to use Chainer
l Some planned features
l (Slide in English, talk in Japanese)

5. : The Concept
5

6. Chainer is a framework of neural networks
l Official site: http://chainer.org
l Repository: https://github.com/pfnet/chainer
l Provided as a Python library (PyPI: chainer)
l Main features
– Powerful:Supports CUDA and multi-‐‑‒GPU capability
– Flexible: Support almost arbitrary architectures
– Intuitive: Forward prop can be written as a regular Python code

7. Elements of a neural network framework
l Multi-‐‑‒dimensional array implementations
l Layer implementations
– Called in various names (layers, modules, blocks, primitives, etc...)
– The smallest units of automatic differentiation
– Contain forward and backward implementations
l Optimizer implementations
l Other stuffs (data loading scheme, training loop, etc...)
– These are also very important, though Chainer currently does not
provide their abstraction (future work)
7

8. Forward prop / Backprop
l Forward prop is how we want to process the input data
l Backprop computes its gradient for the learnable parameters
l Given backward procedures of all layers, backprop can be written as
their combination (a.k.a. reverse-‐‑‒mode automatic differentiation)
8
input hidden output groundtruth
loss func
gradgradgrad
hidden

9. Backprop Implementation Paradigm (1)
Define-‐‑‒and-‐‑‒Run
l First, a computational graph is constructed. Then, it is periodically fed
with minibatches to do forward/backward
l The computational graph can be seen as a program and the forward/
backward computation is done by its interpreter
u Caffe: the program is written by Prototxt
u Torch: the program is constructed by Lua scripts
u Theano-‐‑‒based frameworks: the program is constructed by Python
scripts

10. Backprop Implementation Paradigm (2)
Define-‐‑‒and-‐‑‒Run (cont.)
l Pros
– (Almost) No need of memory management
– The computational graph can be implicitly optimized (cf. Theano)
l Cons
– The program is fixed within the training loop
– The interpreter must have capability of defining various forward
computations, including control-‐‑‒flow statements like if and for
u Theano has the dedicated functions for them (ifelse and scan),
which are unintuitive and not Pythonic
– Network definition is hard to debug, since an error occurs at the
forward computation that is far apart from the network definition

11. Backprop Implementation Paradigm (3)
Define-‐‑‒by-‐‑‒Run
l The forward computation is written as a regular program code with
special variables and operators, executing which simultaneously involves
the forward computation and the graph construction (just by storing the
order of operations).
l The graph is used for the backward computation.
l This paradigm enables us to use arbitrary control flow statements in the
forward computation
– No need of a mini language and its interpreter
l It also makes the forward computation intuitive and easy to debug

12. Backprop Implementation Paradigm (4)
Define-‐‑‒by-‐‑‒Run (cont.)
l The computational graph can be modified within each iteration
l Example: Truncated BPTT (BackProp Through Time)
– BPTT: Backprop on a recurrent net
– Truncated BPTT: Truncate the backprop at some time point
– Truncation is one type of modification of the computational graph
Truncated

13. Features of Chainer
l Define-‐‑‒by-‐‑‒Run scheme
– Forward computation can contain any Python code
u if-else, for-else, break, continue, try-except-finally,
list, dict, class, etc...
– User can modify the graph within the loop
u E.g. truncation can be done by unchain_̲backward (which
unchains the graph backward from some variable)
u See the tutorial on recurrent nets
http://docs.chainer.org/en/latest/tutorial/recurrentnet.html
l Predefined functions
l Support GPU(s) via PyCUDA

14. Example: Training a multi-‐‑‒layer perceptron in one page
Full code is in the tutorial and the example directory.
# Model definition
model = FunctionSet(
l1=F.Linear(784, 100),
l2=F.Linear(100, 100),
l3=F.Linear(100, 10))
opt = optimizers.SGD()
opt.setup(
model.collect_parameters())
# Forward computation
def forward(x, t):
h1 = F.relu(model.l1(x))
h2 = F.relu(model.l2(h1))
y = model.l3(h2)
return F.softmax_cross_entropy(y, t)
# Training loop
for epoch in xrange(n_epoch):
for i in xrange(0, N, batchsize):
x = Variable(...)
t = Variable(...)
opt.zero_grads()
loss = forward(x, t)
loss.backward()
opt.update()

15. Example: Recurrent net language model in one page
Full code is in the tutorial and the example directory.
# Model definition
model = FunctionSet(
emb=F.EmbedID(1000, 100),
x2h=F.Linear( 100, 50),
h2h=F.Linear( 50, 50),
h2y=F.Linear( 50, 1000))
opt = optimizers.SGD()
opt.setup(
model.collect_parameters())
# Forward computation of one step
def fwd1step(h, w, t):
x = F.tanh(model.emb(w))
h = F.tanh(model.x2h(x) + model.h2h(h))
y = model.h2y(h)
return h, F.softmax_cross_entropy(y, t)
# Full RNN forward computation
def forward(seq):
h = Variable(...) # init state
loss = 0
for curw, nextw in
zip(seq, seq[1:]):
x = Variable(curw)
t = Variable(nextw)
h, new_loss = fwd1step(h, x, t)
loss += new_loss
return loss

16. : How to Use It
16

17. Install Chainer
l Prepare a Python 2.7 environment with pip
– (Pyenv+)Anaconda is recommended
l Install Chainer just by
pip install chainer
l If you want to use GPU(s), do:
– Install CUDA and the corresponding NVIDIA driver
– Install dependent packages by
pip install chainer-cuda-deps
– You may have to update the six package
pip install –U six

18. Run the MNIST example (quick start)
l Require scikit-‐‑‒learn installed: pip install scikits.learn
l Clone the repository of Chainer:
git clone https://github.com/pfnet/chainer
l Go to the example directory at examples/mnist
l Then, run python train_mnist.py
– Run on GPU by passing --gpu=0
l Other examples can be similarly executed (some needs manual
preparation of datasets)

19. Read the documents
l Read the documents at http://docs.chainer.org
l It includes:
– Tutorial
– Reference manual
l All features given in this talk are introduced by the tutorial, so please try
it if you want to know the detail.

20. Basic concepts (1)
l Essential part of Chainer: Variable and Function
l Variable is a wrapper of n-‐‑‒dimensional arrays (ndarray and GPUArray)
l Function is an operation on Variables
– Function application is memorized by the returned Variable(s)
– All operations for which you want to backprop must be done by
Functions on Variables
l Making a Variable object is simple: just pass an array
x = chainer.Variable(numpy.ndarray(...))
– The array is stored in data attribute (x.data)

21. Basic concepts (2)
l Example of the computational graph construction
x = chainer.Variable(...)
y = chainer.Variable(...)
z = x**2 + 2*x*y + y
l Gradient of z(x, y) can be computed by z.backward()
l Results are stored in x.grad and y.grad
x
y
_ ** 2
2 * _ _ * _ _ + _ z
_ + _
Actually, Split nodes are automatically
inserted (they accumulate the gradients
on backprop)

22. Basic concepts (3)
l Chainer provides many functions in chainer.functions subpackage
– This package is often abbreviated to F
l Parameterized functions are provided as classes
– Linear, Convolution2D, EmbedID, PReLU, BatchNormalization, etc.
– Their instances should be shared across all iterations
l Non-‐‑‒parameterized functions are provided as Python functions
– Activation functions, pooling, array manipulation, etc.

23. Basic concepts (4)
l Use FunctionSet to manage parameterized functions
– It is an object with Function attributes
– Easy to migrate functions onto GPU devices
– Easy to collect parameters and gradients (collect_̲parameters)
l Use Optimizer for numerical optimization
– Major algorithms are provided:
SGD, MomentumSGD, AdaGrad, RMSprop, ADADELTA, Adam
– Some parameter/gradient manipulations are done via this class:
weight decay, gradient clip,

24. Easy to debug!
l If the forward computation has a bug, then an error occurs immediately
at the appropriate line of the forward definition
l Example
– This code has inconsistency of the array size:
x = Variable(np.ndarray((3, 4), dtype=np.float32)
y = Variable(np.ndarray((3, 3), dtype=np.float32)
a = x ** 2 + x
b = a + y * 2
c = b + x * 2
– Since an exception is raised at the appropriate line, we can easily find
the cause of bug (this is one big difference from Define-‐‑‒and-‐‑‒Run
frameworks)
← an exception is raised at this line

25. Graph manipulation (1)
l Backward unchaining: y.unchain_backward()
– It purges the nodes backward from y
– It is useful to implement truncated BPTT (see PTB example)
x f y g z
y g z
y.unchain_backward()

26. Graph manipulation (2)
l Volatile variables: x = Variable(..., volatile=True)
– Volatile variable does not build a graph
– Volatility can be accessed directly by x.volatile
x = Variable(..., volatile=True)
y = f(x)
y.volatile = False
z = h(y)
x f y g z

27. Example: Training a multi-‐‑‒layer perceptron in one page
Note: F = chainer.functions
# Model definition
model = FunctionSet(
l1=F.Linear(784, 100),
l2=F.Linear(100, 100),
l3=F.Linear(100, 10))
opt = optimizers.SGD()
opt.setup(
model.collect_parameters())
# Forward computation
def forward(x, t):
h1 = F.relu(model.l1(x))
h2 = F.relu(model.l2(h1))
y = model.l3(h2)
return F.softmax_cross_entropy(y, t)
# Training loop
for epoch in xrange(n_epoch):
for i in xrange(0, N, batchsize):
x = Variable(...)
t = Variable(...)
opt.zero_grads()
loss = forward(x, t)
loss.backward()
opt.update()

28. Example: Recurrent net language model in one page
# Model definition
model = FunctionSet(
emb=F.EmbedID(1000, 100),
x2h=F.Linear( 100, 50),
h2h=F.Linear( 50, 50),
h2y=F.Linear( 50, 1000))
opt = optimizers.SGD()
opt.setup(
model.collect_parameters())
# Forward computation of one step
def fwd1step(h, w, t):
x = F.tanh(model.emb(w))
h = F.tanh(model.x2h(x) + model.h2h(h))
y = model.h2y(h)
return h, F.softmax_cross_entropy(y, t)
# Full RNN forward computation
def forward(seq):
h = Variable(...) # init state
loss = 0
for curw, nextw in
zip(seq, seq[1:]):
x = Variable(curw)
t = Variable(nextw)
h, new_loss = fwd1step(h, x, t)
loss += new_loss
return loss

29. CUDA support (1)
l Chainer supports CUDA computation
l Installation
– Install CUDA 6.5+
– Install CUDA-‐‑‒related packages by
pip install chainer-cuda-deps
u Build of PyCUDA may fail if you install CUDA into non-‐‑‒standard
path. In such case, you have to install PyCUDA from source code
with appropriate configuration.

30. CUDA support (2)
l Call cuda.init() before any CUDA-‐‑‒related operations
l Converts numpy.ndarray into GPUArray by chainer.cuda.to_gpu
data_gpu = chainer.cuda.to_gpu(data_cpu)
l A GPUArray object can be passed to the Variable constructor
x = Variable(data_gpu)
l Most functions support GPU Variables
– Parameterized functions must be sent to GPU beforehand by
Function.to_gpu or FunctionSet.to_gpu
l Extracts the results to host memory by chainer.cuda.to_cpu
l All examples support CUDA (pass --gpu=N, where N is the GPU ID)

31. MLP example for CUDA
# Model definition
model = FunctionSet(
l1=F.Linear(784, 100),
l2=F.Linear(100, 100),
l3=F.Linear(100, 10)).to_gpu()
opt = optimizers.SGD()
opt.setup(
model.collect_parameters())
# Forward computation
def forward(x, t):
h1 = F.relu(model.l1(x))
h2 = F.relu(model.l2(h1))
y = model.l3(h2)
return F.softmax_cross_entropy(y, t)
# Training loop
for epoch in xrange(n_epoch):
for i in xrange(0, N, batchsize):
x = Variable(to_gpu(...))
t = Variable(to_gpu(...))
opt.zero_grads()
loss = forward(x, t)
loss.backward()
opt.update()

32. CUDA support (3)
l Chainer also supports computation on multiple GPUs (easily!)
l Model parallel
– Send FunctionSets to appropriate devices (to_̲gpu accepts GPU ID)
model_0 = FunctionSet(...).to_gpu(0)
model_1 = FunctionSet(...).to_gpu(1)
– Copy Variable objects across GPUs by copy function
x_1 = F.copy(x_0, 1)
u This copy is tracked by the computational graph, so you donʼ’t
need to deal with it on backprop

33. CUDA support (4)
l Chainer also supports computation on multiple GPUs
l Data parallel
– FunctionSet can be copied by copy.copy
model = FunctionSet(...)
model_0 = copy.copy(model_0).to_gpu(0)
model_1 = model_1.to_gpu(1)
– Set up the optimizer only for the master model
opt.setup(model_0.collect_parameters())
– After data-‐‑‒parallel gradient computation, gather them
opt.accumulate_grads(model_1.gradients)
– After the update, share them across model copies
model_1.copy_parameters_from(model_0.parameters)

34. Model Zoo support (in the near future)
l Model Zoo is a place that pretrained models are registered
– Provided by BVLC Caffe team
– It contains the Caffe reference models
l We are planning to support the Caffe reference models in three weeks
(the next minor release)
– Current design (it may be changed):
f = CaffeFunction(‘path/to/model.caffemodel’)
x, t = Variable(...), Variable(...)
y = f(inputs={‘data’: x, ‘label’: t}, outputs=[‘loss’])
– It emulates Caffe networks by Chainerʼ’s functions

35. Note: development process
l Schedule
– We are planning to release updates biweekly
– Updates are classified into three groups
u Revision: bug fixes, updates without adding/modifying interfaces
u Minor: Updates that add/modify interfaces without lacking
backward compatibility
u Major: Updates that are not backward-‐‑‒compatible
l We are using the GitHub-‐‑‒flow process
l We welcome your PRs!
– Please send them to the master branch

36. Wrap up
l Chainer is a powerful, flexible, and intuitive framework of neural
networks in Python
l It is based on Define-‐‑‒by-‐‑‒Run scheme, which makes it intuitive and
flexible
l Chainer is a very young project and immature
– Its development started at mid. April (just two months ago)
– We will add many functionailities (especially more functions)
– We may add some abstraction of whole learning processes