Introductory talk about how to think about various deep learning architectures and their suitability to different types of data. Presented at Data Science Meetup @ University of Tartu

1. by Ilya Kuzovkin
ilya.kuzovkin@gmail.com
The First Day at the
Deep Learning Zoo
Which architecture should I choose?

2. http://www.emergentmind.com/neural-network

3. http://www.emergentmind.com/neural-network
Input 1
Input 2
Target

4. http://www.emergentmind.com/neural-network
Input 1
Input 2
Target
1
1

5. http://www.emergentmind.com/neural-network
Input 1
Input 2
Target
1
1

6. http://www.emergentmind.com/neural-network
Input 1
Input 2
Target
Backpropagation
1
1

7. http://www.emergentmind.com/neural-network
Input 1
Input 2
1
1

8. http://www.emergentmind.com/neural-network
Input 1
Input 2
1
1
-0.5
+1.1

9. http://www.emergentmind.com/neural-network
Input 1
Input 2
1
1
-0.5
+1.1
1

10. http://www.emergentmind.com/neural-network
Input 1
Input 2
1
1
-0.5
+1.1
1
-0.5
1
1
+1.1
+0.9
+0.9

11. http://www.emergentmind.com/neural-network
Input 1
Input 2
1
1
-0.5
+1.1
1
-0.5
1
1
+1.4
0
+1.1
+0.9
+0.9
-0.9
-0.9

13. Easy for you, but how a neural network would do it?
?

14. http://playground.tensorflow.org/#activation=relu&batchSize=10&dataset=gauss&regDataset=reg-
plane&learningRate=0.03&regularizationRate=0&noise=10&networkShape=&seed=0.53101&showTestData=false&discretize=true&percTrainData=50&x=false&y=false&x
TimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=fal
se
http://playground.tensorflow.org

15. http://playground.tensorflow.org/#activation=relu&batchSize=10&dataset=gauss&regDataset=reg-
plane&learningRate=0.03&regularizationRate=0&noise=10&networkShape=&seed=0.53101&showTestData=false&discretize=true&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=fa
lse&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false

16. http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=xor&regDataset=reg-
plane&learningRate=0.03&regularizationRate=0&noise=10&networkShape=3,3,3&seed=0.28579&showTestData=false&discretize=true&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&co
sX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false

17. Fully Connected Feed-Forward Network

18. Fully Connected Feed-Forward Network

19. Fully Connected Feed-Forward Network

20. Fully Connected Feed-Forward Network

21. Fully Connected Feed-Forward Network
Input data
does not
have clear
structure

22. Fully Connected Feed-Forward Network
Normalize!
Input data
does not
have clear
structure
http://keras.io

23. Convolutional Neural Network
http://www.asimovinstitute.org/neural-network-zoo/

24. Convolutional Neural Network
http://www.asimovinstitute.org/neural-network-zoo/

25. Convolutional Neural Network
http://www.asimovinstitute.org/neural-network-zoo/

26. Convolutional Neural Network
http://www.asimovinstitute.org/neural-network-zoo/

27. Convolutional Neural Network
Input data
has spatial
structure
http://www.asimovinstitute.org/neural-network-zoo/

28. Convolutional Neural Network
Input data
has spatial
structure
Exploit it!
http://scs.ryerson.ca/~aharley/vis/conv/
http://www.asimovinstitute.org/neural-network-zoo/

29. Convolutional Neural Network
Input data
has spatial
structure
Exploit it!
http://scs.ryerson.ca/~aharley/vis/conv/
http://keras.io

30. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer

31. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer

32. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer

33. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer

34. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer

35. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer

36. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer
Input data
has
temporal
structure

37. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer
output
input at time t
hidden layer
Input data
has
temporal
structure

38. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer
output
input at time t
hidden layer
http://karpathy.github.io/2015/05/21/rnn-effectiveness
Input data
has
temporal
structure

39. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer
output
input at time t
hidden layer
http://karpathy.github.io/2015/05/21/rnn-effectiveness
Input data
has
temporal
structure

40. Recurrent Neural Network
input at time 1 input at time 2 input at time 3
output
hidden layer same hidden layer same hidden layer
output
input at time t
hidden layer
http://karpathy.github.io/2015/05/21/rnn-effectiveness
Input data
has
temporal
structure

41. Unstructured
Spatially
structured
Temporally
structured
Zoo
Generative Adversarial Network AutoencoderNeural Turing Deep Belief Network
Fully Connected Convolutional Recurrent