Introduction to Deep Learning. Autoencoders. Conv Nets. Word Embeddings. Applications to Telenor data.

1. Cyril Banino-Rokkones
Telenor Research

2. 2

3. I know nothing about Deep Learning
3

4. Outline
• Intro to DL (A. Ng)
• Intro to Neural Nets
• Training NN
• Conv Nets
• Autoencoders
• Word Embeddings
• DL@TRD
• Bonus
4

5. AI was the weak link until Deep Learning
matured
5
China's Search Giant Goes Deep

6. AI was the weak link until Deep Learning
matured
6
http://www.iro.umontreal.ca/~bengioy/dlbook/intro.html

7. Loose inspiration from the brain
7
China's Search Giant Goes Deep

8. Large Neural Nets perform better than small ones
8
China's Search Giant Goes Deep

9. Google Brain project – 1 billion connections – 1
week of youtube watching.
9 China's Search Giant Goes Deep

10. From 16k CPUs to 3 GPUs
From 1M connections to 10 B
10
China's Search Giant Goes Deep

11. Applications of Deep Learning
11
China's Search Giant Goes Deep

12. Voice interface to assist computer-illiterates
12
China's Search Giant Goes Deep

13. Image-search for impossible queries
13
China's Search Giant Goes Deep

14. Image-search for impossible queries
14
China's Search Giant Goes Deep

15. Image-queries to find stuff impossible to describe
15
China's Search Giant Goes Deep

16. 16
“Whoever wins AI wins the Internet.” A. Ng.
Google, Facebook and other tech companies race to develop artificial intelligence

17. Outline
• Intro to DL (A. Ng)
• Intro to Neural Nets
• Training NN
• Conv Nets
• Autoencoders
• Word Embeddings
• DL@TRD
• Bonus
17

18. Perceptrons makes decisions by weighing evidence
18 http://neuralnetworksanddeeplearning.com/chap1.html

19. Example: NAND gate
19 http://neuralnetworksanddeeplearning.com/chap1.html

20. Wiring several perceptrons for more abstract and complex
decisions
20 http://neuralnetworksanddeeplearning.com/chap1.html

21. A simple network to classify handwritten digits (MNIST)
21 http://neuralnetworksanddeeplearning.com/chap1.html

22. Outline
• Intro to DL (A. Ng)
• Intro to Neural Nets
• Training NN
• Conv Nets
• Autoencoders
• Word Embeddings
• DL@TRD
• Bonus
22

23. Training Neural Networks: Gradient descent
23

24. Learning to solve a problem
24 http://neuralnetworksanddeeplearning.com/chap1.html

25. Forward and Backward passes
25
http://caffe.berkeleyvision.org/tutorial/forward_backward.html

26. The Unstable Gradient Problem
26
Why it is difficult to train an RNN
Why are deep neural networks hard to train?

27. Practical advices when training neural networks
(by Ilya Sutskever)
27
• Get good data
• Preprocessing
• Minibatches
• Gradient normalization
• Learning rate schedule
• Learning rate
• Weight Initialization
• Data augmentation
• Dropout
• Ensembling
A Brief Overview of Deep Learning

28. Outline
• Intro to DL (A. Ng)
• Intro to Neural Nets
• Training NN
• Conv Nets
• Autoencoders
• Word Embeddings
• DL@TRD
• Bonus
28

29. Convolutional Neural Network have been here for a while
29

30. Convolutions
30 Understanding Convolutions

31. Convolutional Neural Network
31
Conv Nets: A Modular Perspective

32. Convolutional Neural Network
32
Human-level control through deep reinforcement learning

33. Intriguing properties of Conv Nets
33
Intriguing properties of neural networks

34. Outline
• Intro to DL (A. Ng)
• Intro to Neural Nets
• Training NN
• Conv Nets
• Autoencoders
• Word Embeddings
• DL@TRD
• Bonus
34

35. Stacked Autoencoders
35
Reducing the Dimensionality of Data with Neural Networks

36. Stacked Autoencoders
36
Reducing the Dimensionality of Data with Neural Networks

37. Stacked Autoencoders – semantic Hashing
37
Semantic Hashing
Reducing the Dimensionality of Data with Neural Networks

38. Behavioral micro-segmentation (training set)
38
1008
275
275
1008
8
150
150
0.011|0.98|0.2| … 0
Bit code
1 0 …~

39. Outline
• Intro to DL (A. Ng)
• Intro to Neural Nets
• Training NN
• Conv Nets
• Autoencoders
• Word Embeddings
• DL@TRD
• Bonus
39

40. 40
Word embeddings
Deep Learning, NLP, and Representations

41. 41
Word embeddings and Shared representations
Deep Learning, NLP, and Representations Deep Visual-Semantic Alignments for Generating Image
Descriptions

42. 42
Word embeddings and Recurrent Neural Nets
Deep Learning, NLP, and Representations

43. 43
Word embeddings and Reversible Sentence Representation
Deep Learning, NLP, and Representations
Rich Rashid in Tianjin, October, 25, 2012

44. Telenor Norway Network topology
Word embeddings applied to Network operations
Use cases:
• Predict failures of Network components.
• Predict congestion levels on Network links.
• Detect mal-functioning devices.

45. Outline
• Intro to DL (A. Ng)
• Intro to Neural Nets
• Training NN
• Conv Nets
• Autoencoders
• Word Embeddings
• DL@TRD
• Bonus
45

46. DL@TRD - Motivations
Personal observations:
– DL is hot (hyped?)
– DL supremacy seems ineluctable
– DL can solve a whole bunch of problems
– DL is frontier technology (difficult)
– Little DL competence @ Telenor Research
Personal implications:
– Career development
– Network with partners to get momentum
– Great if this happens in Trondheim
46

47. DL@TRD - Vision
Establish a strong DL competence center in Trondheim
– A place where
• competence is gathered
• experiences are exchanged
• collaborations are fostered
– Benefits
• Share passion with others near you
• Get momentum for your work
• Funding (SFI, EU money)
– Ideally
• Collaborate across companies on problems
• Common publications
47
Next workshop: 27th March

48. DL@Telenor – Topics of Interest
NLP tasks
– Speech-to-Text
– Text-to-Speech
– Automatic summarization
– Sentiment analysis
Computer Vision
– Face detection
– Image recognition/classification
 Telenor Applications
– New Digital Services
– Managing our Networks
– Understanding our Customers
48

49. Stuff we could discuss at DL@TRD
• Training Recurrent Neural Networks
• Long Short Term Memory Networks
• Echo State Networks
• Neural Turing Machines
• Hopfield Nets
• Restricted Bolzman Machines
• Deep beliefs Networks
• Teacher – Student Nets
• Momentum
• Dropout
• Full Bayesian learning
• Hessian free optimization
• Stuff I don´t know I don´t know
49

50. Conclusion & Forecast
50
• DL techniques can be applied to all sorts of data:
– Could you apply some of these techniques to your data?
• DL models are better than humans at some tasks if fed with enough
data & trained properly
• Within 5-10 years, “information work” tasks will be augmented or even
fully automated
– See Peter Norvig´s talk at InfoQ: Machine Learning for Programming
– Models can take decisions based on millions of records while removing human
biases
 Big data + Deep Learning = unemployment
– New policies and economic measures will be needed to manage the adverse
effects of job computerization
– Schooling will need reforms: routine tasks  non-routine tasks

51. Thank you
51
btw we´re hiring…