1.
Introduction to Deep Learning
July 12th 2017
Prepared for: Charlotte Bots & AI Meetup
Presenter: Abhishek Bhandwaldar
Data scientist at botsplash.com

2.
They are all talking about AI especially
advancements in Deep Learning!
Photo credit: http://www.latercera.com/

3.
AI Evolution
Credit: Nvidia.com at https://goo.gl/mY55TM

4.
Artificial Intelligence Techniques

5.
Cognitive Use Cases (Why DL?)
Computer Vision Self Driving Cars
Faces, Gaming
Medical / Sensors
Speech Processing Voice Recognition
Music Generation
Language Translation
Natural Language
Processing
CRM
Chatbots
ADs
Credit: kdnuggets.com

6.
Deep Learning through Ages
1958
Perceptron
algorithm
Rosenblatt
created the
perceptron
algorithm
1965
Multilayer
perceptron
First algorithm for
multilayer
perceptron was
publisher by
Ivakhnenko
Image Recognition
Yan LeCun et al.
successfully applied
deep neural network
for image recognition
using convolution
1969 1998
Neural Network
Setbacks
Minsky and Papert
proved in their
published book
‘Perceptrons’ that it
has many limitation

7.
Deep Learning Process
• Data Collection and Pre-processing
• Data division in train, dev and test sets
• Selection of Model architecture
• Training and performance benchmarking
• Tuning hyper-parameters and Repeat training
• Testing with Test set

8.
Getting Started with Deep Learning
1. Python programming or alternative
2. GPU based Hardware
3. Deep Learning Frameworks
4. Basic knowledge of Neural Networks
5. Data sets (many available online)

9.
Python Programming
Basic level skill. Learn from youtube.com or coursera.com

10.
GPU Hardware and Training in Cloud
• CPU will work for simple workloads
Cloud Provider Pros Cons
Floydhub.com   Similar to heroku, Easy to
get started and use, Free
credits
No GUI. Difficult to get GPU
working for non Tensorflow
solutions.
Paperspace.com  Full blown desktop in cloud
with good GPU support.
UI lags and buggy but
usable.
Google ML Engine  Best for Tensorflow only
solution. Data labs is good
notebook environment.
No support beyond
Tensorflow
Amazon AWS   GPU supported systems Self hosting and
maintenance

11.
Deep Learning Frameworks
• Keras is wrapper and makes it easy to work with DL frameworks!
Framework Sponsor Best for:
Tensorflow Google Popularity and ease of use
CNTK Microsoft Fast, Accurate and Growing
PyTorch Facebook Early adopter
MxNET Amazon Group of companies
Caffe, Theano Schools Researchers

12.
Neural Networks Overview
• Linear and Non Linear Models
• Deep Neural Networks (Deep Learning name comes from here)
 Training
 Architecture
 Convolutional Neural Networks
 Recurrent Neural Networks
 Generative Adversial Neural Networks

13.
Linear Models
• Output is linear function of input.
• The model is limited in learning
and addition of layers has no
effect.
• Computation on GPU is very
efficient.
Input
Linear
Function
Output
z = b + ∑ xiwi
Image source:
https://en.wikipedia.org/wiki/Generalized_linear_model

14.
Non Linear Model and Relu
• By introducing non linearity model is able
to learn much better.
• The non linearity function most widely
used is Relu.
• f(x)=max(0,x)
• Other activations functions used :
• Sigmoid
• Tanh
• Relu is less computationally expensive.
Input
Linear
Function
Output
z = b + ∑ xiwi
Relu
output = max(0, z)
Image source: http://cs231n.github.io/neural-networks-
1/

15.
Deep Neural Network & Architecture
• When we connect multiple neuron
together we have a fully connected
deep neural network.
• Making network deeper than wider.
• This helps in learning hierarchical
representation (Low level details to
high level concepts).
• Also decreases learnable
parameters.
Image source: http://cs231n.github.io/neural-networks-1/

16.
Back Propagation and Neural Network Training
• Back propagation is the algorithm we use for neural network learning.
• The cost function widely used is the cross entropy cost function for calculating loss.
• Two Steps:
• Forward Pass: The data is passed through the network and the loss is calculated
• Backward Pass: Loss is propagated backwards by changing weights i.e. optimization
• Various methods for NN optimization:
• Stochastic Gradient Descent, Momentum, Nesterov accelerated gradient, RMSprop, Adagrad, Adam

17.
Demo
Feed Forward Neural Network
at Tensorflow Playground

18.
Convolutional neural network
Has 3 Types of Layers.
• Convolution Layer: A convolutional layer applies convolution to input
• Pooling Layer/Subsampling: This layer combines all output into single value.
• Fully-connected layer: Simple fully connected network
Image source: LeCun, Yann; Léon Bottou; Yoshua Bengio; Patrick Haffner (1998). "Gradient-based learning applied to document recognition" (PDF).
Proceedings of the IEEE.

19.
CNN Example
LeNet-5, convolutional neural
networks
source:http://yann.lecun.com/exdb/lenet/

20.
Code
Walkthrough
Handwriting Recognition on
MNIST data set

21.
Recurrent Neural Network
• This type of Neural network is used for sequence data. Ideal for text data.
• The output of hidden layer is fed back into itself (feedback).
• RNN are Turing-complete. But practically they are very difficult to implement
because of the gradient explosion/vanishing problem.
• To tackle this issue we have the LSTM network.
Image source: http://karpathy.github.io/2015/05/21/rnn-
effectiveness/

22.
Generative Adversarial Networks
• According to Yan LeCun GAN were the next big thing.
• Architecture is Simple: We have Discriminative model has task of
determining whether image is image looks natural or not.
• The task of generator is to generate image in order to fool discriminator.

23.
Tips for training Deep Neural Network
• Batch Learning:
• Pass over complete training examples is made and then weights are updated.
• In Mini-batch pass over small batch is made and weights are updated after every batch
• Fast and parallel training can be implemented on GPU. Widely used.
• Online Learning:
• After completing pass over every example weights are updated. Easy to train on new examples
• Very slow.
• Convolutions can be implemented parallelly and gain speed up on GPU.
• In RNN, multiple examples can be processed parallelly if using batch learning.
• Using techniques like dropout and regularization to prevent overfitting.
• Gathering more examples to prevent overfitting and generalize better.
• Increase number of layer to prevent underfitting.

24.
Resources
Neural Network:
• Neural Networks for Machine Learning | Coursera
• Neural Networks by Hugo Larochelle
• Neural Networks, Manifolds, and Topology -- colah's blog
• Distill — Latest articles about machine learning
• Deep Learning Book
• An overview of gradient descent optimization algorithms
• Deep Learning By Google
• SIRAJ RAVAL'S DEEP LEARNING (Also available on Siraj Rawal’s YouTube channel)
• Neural Networks and Deep Learning
• Understanding Activation Functions in Neural Networks
RNN:
• The Unreasonable Effectiveness of Recurrent Neural Networks
• Recurrent Neural Networks Tutorial
• How to build a Recurrent Neural Network in TensorFlow
CNN:
• Convolutional Neural Network - Deep Learning
• Convolutional Neural Networks (LeNet)

25.
Questions & Feedback