- The document discusses deep learning and Apache MXNet. It provides an overview of neural networks and their history, the training process using backpropagation and stochastic gradient descent, and techniques like convolutional neural networks. - It then introduces Apache MXNet, describing it as programmable, portable, high performance, supporting multiple languages and optimized for deep learning on AWS. - Finally, it proposes some examples of coding with MXNet, from basic networks to pre-trained models, training from scratch, fine-tuning, and using MXNet as a Keras backend.

1. ©2017, Amazon Web Services, Inc. or its affiliates. All rights reserved
Deep Learning with Apache MXNet
Julien Simon, Principal Technical Evangelist
@julsimon

2. Selected customers running AI on AWS

3. Questions
• What’s the business problem my IT has failed to solve?
– That’s probably where Deep Learning can help
• Should I design and train my own Deep Learning model?
– Do I have the expertise?
– Do I have enough time, data & compute to train it?
• Should I use a pre-trained model?
– How well does it fit my use case?
– On what data was it trained? How close is this to my own data?
• Should I use a SaaS solution?

4. Amazon AI for every developer
More information at aws.amazon.com/ai

5. Neural Networks

6. Frank Rosenblatt
Artificial Intelligence pioneer
1957 – The Perceptron
Appropriate weights are applied to the inputs, and
the resulting weighted sum is passed to a function
that produces the output.

7. The neuron
x = [x1, x2, …. xI ]
w = [w1, w2, …. wI ]
u = x.w
Activation function

8. The neural network
x =
x11, x12, …. x1I
x21, x22, …. x2I
x…, x.., …. x.I
xm1, xm2, …. xmI
I features
m samples
y =
y1
y2
y…
ym
m labels
Total number of predictions
Accuracy =
Number of correct predictions

9. The training process
• The difference between the predicted output and the actual output (aka
ground truth) is called the prediction loss.
• There are different ways to compute it (loss functions).
• The purpose of training is to iteratively minimize loss and maximize
accuracy for a given data set.
• We need a way to adjust weights (aka parameters) in order to gradually
minimize loss
à Backpropagation + optimization algorithm

10. Paul Werbos
Artificial Intelligence pioneer
IEEE Neural Network Pioneer Award
1974 – Backpropagation
The back-propagation algorithm acts as an error
correcting mechanism at each neuron level, thereby
helping the network to learn effectively.

11. Stochastic Gradient Descent (SGD)
Imagine you stand on top of a mountain with skis
strapped to your feet. You want to get down to the
valley as quickly as possible, but there is fog and
you can only see your immediate surroundings.
How can you get down the mountain as quickly as
possible? You look around and identify the steepest
path down, go down that path for a bit, again look
around and find the new steepest path, go down
that path, and repeat—this is exactly what gradient
descent does.
Tim Dettmers
University of Lugano
2015
Source: https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-history-training/
The « step size » is called
the learning rate

12. There is such a thing as « learning too well »
• If a network is large enough and given enough time, it will perfectly
learn a data set (universal approximation theorem).
• But what about new samples? Can it also predict them correctly?
• Does the network generalize well or not?
• To prevent overfitting, we need to know when to stop training.
• The training data set is not enough.

13. Data sets
Full data set
Training data set
Validation data set
Test data set
Training set: this data set is used to
adjust the weights on the neural
network.
Validation set: this data set is used to
minimize overfitting. You're not
adjusting the weights of the network
with this data set, you're just verifying
that any increase in accuracy over the
training data set actually yields an
increase in accuracy over a data set
that has not been shown to the
network before.
Testing set: this data set is used only
for testing the final weights in order to
benchmark the actual predictive power
of the network.

14. Training
Training data set Training
Trained
neural network
Number of epochs
Batch size
Learning rate
Hyper parameters

15. Validation
Validation Data set Trained
neural network
Validation
accuracy
Prediction at
the end of
each epoch
Stop training when validation accuracy stops increasing
Saving parameters at the end of each epoch is a good idea ;)

16. Convolutional Neural Networks
Le Cun, 1998: handwritten digit recognition, 32x32 pixels
Feature extraction and downsampling allow smaller networks
https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/

17. Apache MXNet

18. Apache MXNet
Programmable Portable High Performance
Near linear scaling
across hundreds of GPUs
Highly efficient
models for mobile
and IoT
Simple syntax,
multiple languages
Most Open Best On AWS
Optimized for
deep learning on AWS
Accepted into the
Apache Incubator

19. import numpy as np
a = np.ones(10)
b = np.ones(10) * 2
c = b * a
d = c + 1
• Straightforward and flexible.
• Take advantage of language native
features (loop, condition,
debugger).
• E.g. Numpy, Matlab, Torch, …
• Hard to optimize
PROS
CONS
Easy to tweak
in Python
Imperative Programming

20. • More chances for optimization
• Cross different languages
• E.g. TensorFlow, Theano, Caffe
• Less flexible
PROS
CONS
C can share memory with D
because C is deleted later
A = Variable('A')
B = Variable('B')
C = B * A
D = C + 1
f = compile(D)
d = f(A=np.ones(10),
B=np.ones(10)*2)
A B
1
+
x
Declarative Programming

21. 0
4
8
12
16
1 2 4 8 16
Ideal
Inception v3
Resnet
Alexnet
91%
Efficiency
Multi-GPU Scaling With MXNet

22. Input Output
1 1 1
1 0 1
0 0 0
3
mx.sym.Convolution(data, kernel=(5,5), num_filter=20)
mx.sym.Pooling(data, pool_type="max", kernel=(2,2),
stride=(2,2)
lstm.lstm_unroll(num_lstm_layer, seq_len, len, num_hidden, num_embed)
4 2
2 0
4=Max
1
3
...
4
0.2
-0.1
...
0.7
mx.sym.FullyConnected(data, num_hidden=128)
2
mx.symbol.Embedding(data, input_dim, output_dim = k)
0.2
-0.1
...
0.7
Queen
4 2
2 0
2=Avg
Input Weights
cos(w, queen) = cos(w, king) - cos(w, man) + cos(w, woman)
mx.sym.Activation(data, act_type="xxxx")
"relu"
"tanh"
"sigmoid"
"softrelu"
Neural Art
Face Search
Image Segmentation
Image Caption
“People Riding
Bikes”
Bicycle, People,
Road, Sport
Image Labels
Image
Video
Speech
Text
“People Riding
Bikes”
Machine Translation
“Οι άνθρωποι
ιππασίας ποδήλατα”
Events
mx.model.FeedForward model.fit
mx.sym.SoftmaxOutput

23. Let’s code
1. Our first network
2. Using pre-trained models
3. Learning from scratch (MNIST, MLP, CNN)
4. Learning and fine-tuning (CIFAR-10, Resnext-101)
5. Using Apache MXNet as a Keras backend
6. Fine-tuning with Keras/MXNet (CIFAR-10, Resnet-50)
7. A quick word about Sockeye
http://mxnet.io
http://medium.com/@julsimon
https://github.com/juliensimon/aws/tree/master/mxnet

24. Thank you!
http://aws.amazon.com/evangelists/julien-simon
@julsimon