An introduction to Neural Networks and Deep Learning using simple highschool mathematics

2. An attempt to simplify Deep Learning concepts to
mathematics haters’

3. Guess the next 3 points!
Temperature
Time

4. Temperature
Time
Guess the next 3 points!

5. Temperature
Time
Guess the next 3 points!

6. Temperature
Time
Guess the next 3 points!

7. Temperature
Time
Clouds
Guess the next 3 points!

8. Temperature
Time
Clouds
Guess the next 3 points!

9. Temperature
Time
Clouds
11 13 18 22 28 27 29 18 23 12 7 15 22 y …T
y =f(X)
Guess the next 3 points!

10. 11 13 18 22 28 27 29 18 23 12 7 15 22 y …T
int function f(X) {
int y;
// computations
return y;
}
y =f(X)
Prediction function
Guess the next 3 points!

11. Machine learning
Is all about learning to predict
But what kind of predictions?

12. x
x
Regression Classification Clustering
Machine learning

13. x
x
Regression Classification Clustering
Supervised learning Unsupervised learning
Machine learning

14. x
Unsupervised learning
y
x’
y’
Learning
phase
Prediction
phase
Supervised learning
Machine learning
(predicted result)
(data)
(result)
(new data)

15. x
Unsupervised learning
y
x
x’
y’ y’
Learning
phase
Prediction
phase
Supervised learning
Machine learning
(predicted result)
(data)
(result)
(new data)
(data)
(predicted
result)

16. x
Unsupervised learning
y
x
x’
y’ y’
Learning
phase
Prediction
phase
Supervised learning
Machine learning
Reinforcement learning
Same but replace « y »
by the feedback (score)
(predicted result)
(data)
(result)
(new data)
(data)
(predicted
result)

17. y’
𝑥1
𝑥2
𝑥 𝑛
…
𝑓(Σ)
How to predict?

18. y’
𝑥1
𝑥2
𝑥 𝑛
…
𝑓(Σ)
How to predict?
𝑦′
= 𝑓(Σ)
Sign
Tanh
Sigmoid
…
Activation
function

19. y’
𝑥1
𝑥2
𝑥 𝑛
…
𝑓(Σ)
How to predict?
𝑤1
𝑤2
𝑤 𝑛
𝑦′
= 𝑓(Σ)
Sign
Tanh
Sigmoid
…
Weighted sum
Activation
function
Σ = 𝑊 𝑇 ∗ 𝑋

20. y’
𝑥1
𝑥2
𝑥 𝑛
…
𝑓(Σ)
How to predict?
𝑤1
𝑤2
𝑤 𝑛
𝑦′
= 𝑓(Σ)
Sign
Tanh
Sigmoid
…
Weighted sum
𝑏
Activation
function
Σ = 𝑊 𝑇 ∗ 𝑋 + 𝑏
(bias)

21. y’
𝑥1
𝑥2
𝑥 𝑛
…
𝑓(Σ)
How to predict?
𝑤1
𝑤2
𝑤 𝑛
𝑦′
= 𝑓(Σ)
𝑏
Activation fct:
Σ = 𝑊 𝑇 ∗ 𝑋 + 𝑏
(bias) 𝑓(Σ) =
1 𝑖𝑓 Σ ≥ 0
−1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Perceptron (1957, F. Rosenblatt)

22. y’
𝑥1
𝑥2
𝑥 𝑛
…
𝑓(Σ)
How to predict?
𝑤1
𝑤2
𝑤 𝑛
𝑦′
= 𝑓(Σ)
𝑏
Activation fct:
Σ = 𝑊 𝑇 ∗ 𝑋 + 𝑏
(bias) 𝑓(Σ) =
1 𝑖𝑓 Σ ≥ 0
−1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Perceptron (1957, F. Rosenblatt)
Updating rule: 𝑊 𝑡+1 = 𝑊 𝑡 + 𝜂. 𝐸 ∗ 𝑋Learning phase
𝐸 = 𝑦′ − 𝑦

23. y’
𝑥1
𝑥2
𝑥 𝑛
…
𝑓(Σ)
How to predict?
𝑤1
𝑤2
𝑤 𝑛
𝑦′
= 𝑓(Σ)
𝑏
Activation fct:
Σ = 𝑊 𝑇 ∗ 𝑋 + 𝑏
(bias) 𝑓(Σ) =
1 𝑖𝑓 Σ ≥ 0
−1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Perceptron (1957, F. Rosenblatt)
Updating rule: 𝑊 𝑡+1 = 𝑊 𝑡 + 𝜂. 𝐸 ∗ 𝑋Learning phase
𝐸 = 𝑦′ − 𝑦
Repeat until E = 0 (or max iter)

24. How to predict?
𝑦′
= 𝑓(Σ)
Activation fct:
Σ = 𝑊 𝑇 ∗ 𝑋 + 𝑏
𝑓(Σ) =
1 𝑖𝑓 Σ ≥ 0
−1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Perceptron (1957, F. Rosenblatt)
Updating rule: 𝑊 𝑡+1 = 𝑊 𝑡 + 𝜂. 𝐸 ∗ 𝑋
𝐸 = 𝑦′ − 𝑦
Binary Classification

25. How to predict?
𝑦′
= 𝑓(Σ)
Activation fct:
Σ = 𝑊 𝑇 ∗ 𝑋 + 𝑏
𝑓(Σ) =
1 𝑖𝑓 Σ ≥ 0
−1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Perceptron (1957, F. Rosenblatt)
Updating rule: 𝑊 𝑡+1 = 𝑊 𝑡 + 𝜂. 𝐸 ∗ 𝑋
𝐸 = 𝑦′ − 𝑦
Binary Classification
Use non-linear activation

26. What if?
Binary Classification
Non linear classification Multi-class classification

27. 𝑥1 𝑓(Σ)
𝑥2 𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑦′1
𝑦′2
Input layer
Hidden layer(s)
Output layer
Multi-layer perceptron (MLP)

28. 𝑥1 𝑓(Σ)
𝑥2 𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑦′1
𝑦′2
Input layer
Hidden layer(s)
Output layer
𝑤11
(1)
𝑤12
(1)
𝑤13
(1)
𝑤21
(1)
𝑤22
(1)
𝑤23
(1)
𝑤11
(2)
𝑤12
(2)
𝑤21
(2)
𝑤31
(2)
𝑤32
(2)
𝑤22
(2)
Multi-layer perceptron (MLP)

29. 𝑥1 𝑓(Σ)
𝑥2 𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑦′1
𝑦′2
Input layer
Hidden layer(s)
Output layer
𝑤11
(1)
𝑤12
(1)
𝑤13
(1)
𝑤21
(1)
𝑤22
(1)
𝑤23
(1)
𝑤11
(2)
𝑤12
(2)
𝑤21
(2)
𝑤31
(2)
𝑤32
(2)
𝑤22
(2)
𝐸 =
1
𝑛
𝑖
(𝑦𝑦 − 𝑦′
𝑦
)²
MSE
Multi-layer perceptron (MLP)

30. 𝑥1 𝑓(Σ)
𝑥2 𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑓(Σ)
𝑦′1
𝑦′2
Backward propagation (update weights by using the partial derivatives of the error)
𝑤11
(1)
𝑤12
(1)
𝑤13
(1)
𝑤21
(1)
𝑤22
(1)
𝑤23
(1)
𝑤11
(2)
𝑤12
(2)
𝑤21
(2)
𝑤31
(2)
𝑤32
(2)
𝑤22
(2)
𝐸 =
1
𝑛
𝑖
(𝑦𝑦 − 𝑦′
𝑦
)²
MSE
Feedforward (calculate activations)
Multi-layer perceptron (MLP)

31. Multi-layer perceptron (MLP)
What’s really
happening inside
the MLP?

32. The hidden layers project
the data into a higher
dimessional space
Multi-layer perceptron (MLP)

33. The hidden layers project
the data into a higher
dimessional space
The neurons separate data
using a hyperplan
Multi-layer perceptron (MLP)

34. The hidden layers project
the data into a higher
dimessional space
The neurons separate data
using a hyperplan
We return to the low
dimenssional space
Multi-layer perceptron (MLP)

35. A preprocessing problem
The MLP expects
a vector as input
[Mahmoudi & al. 2016, “Predicting the Earthquake
Magnitude Using the Multilayer Perceptron Neural
Network with Two Hidden Layers”]

36. A preprocessing problem
Source: https://blog.insightdatascience.com/automating-breast-cancer-detection-with-deep-learning-d8b49da17950

37. Convolutional Neural Networks (CNN)
Source: https://blog.insightdatascience.com/automating-breast-cancer-detection-with-deep-learning-d8b49da17950

38. Convolutional Neural Networks (CNN)
Convolution with a
3x3 filter (kernel)
Source: http://intellabs.github.io/RiverTrail/tutorial/

39. Convolutional Neural Networks (CNN)
Maxpooling with a
2x2 filter
Source: https://computersciencewiki.org/index.php/File:MaxpoolSample2.png

40. Convolutional Neural Networks (CNN)
Source: https://www.superdatascience.com/convolutional-neural-networks-cnn-summary

41. Convolutional Neural Networks (CNN)
Source: https://www.superdatascience.com/convolutional-neural-networks-cnn-summary
Let the learning algorithm learn the best kernels!

42. An example of features learned by each layer
Source: [Honglak Lee, et al, “Convolutional Deep Belief Networks for
Scalable Unsupervised Learning of Hierarchical Representations”]

43. CNN, what else?
Source: http://www.asimovinstitute.org/neural-network-zoo

44. CNN, what else?
Source: http://www.asimovinstitute.org/neural-network-zoo

45. CNN, what else?
Source: https://in.mathworks.com/help/stats/machine-learning-in-matlab.html