Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
•
0 likes•481 views
Deep neural networks have revolutionized the data analytics scene by improving results in several and diverse benchmarks with the same recipe: learning feature representations from data. These achievements have raised the interest across multiple scientific fields, especially in those where large amounts of data and computation are available. This change of paradigm in data analytics has several ethical and economic implications that are driving large investments, political debates and sounding press coverage under the generic label of artificial intelligence (AI). This talk will present the fundamentals of deep learning through the classic example of image classification, and point at how the same principal has been adopted for several tasks. Finally, some of the forthcoming potentials and risks for AI will be pointed.
![Deep Learning
Representations for All
(a.k.a. The AI Hype)
Xavier Giro-i-Nieto
@DocXavi
xavier.giro@upc.edu
Associate Professor
Universitat Politècnica de Catalunya
Spring 2020
[Summer School website]](https://image.slidesharecdn.com/x-theses2020irideeplearningrepresentationsforall-200629142740/85/Deep-Learning-Representations-for-All-Xavier-Giro-i-Nieto-IRI-Barcelona-2020-1-320.jpg)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Similar to Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020 (20)
More from Universitat Politècnica de Catalunya
More from Universitat Politècnica de Catalunya (17)
Recently uploaded
Recently uploaded (20)
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
- 1. Deep Learning Representations for All (a.k.a. The AI Hype) Xavier Giro-i-Nieto @DocXavi xavier.giro@upc.edu Associate Professor Universitat Politècnica de Catalunya Spring 2020 [Summer School website]
- 2. 2 Acknowledgements Kevin McGuinness kevin.mcguinness@dcu.ie Assistant Professor School of Electronic Engineering Dublin City University
- 3. 3 Deep Neural Networks 101
- 5. Classic Machine Learning classification pipeline Raw data (ex: images) Feature Extraction Classifier Decisor y = ‘CAT’ X1: weight X2: height Probabilities: CAT: 0.7 DOG: 0.3 5Slide credits: Santiago Pascual (UPC TelecomBCN 2019). Garfield is a character created by Jim Davis.
- 6. Raw data (ex: images) Feature Extraction Classifier Decisor y = ‘CAT’ X1: weight X2: height Probabilities: CAT: 0.7 DOG: 0.3 Neural Network Shall we extract features now? 6 Classic Machine Learning classification pipeline Slide credits: Santiago Pascual (UPC TelecomBCN 2019). Garfield is a character created by Jim Davis.
- 7. Raw data (ex: images) Classifier Decisor y = ‘CAT’ Probabilities: CAT: 0.7 DOG: 0.3 Neural Network We CAN inject the raw data, and features will be learned!! End to End concept 7 Deep Learning classification pipeline Slide credits: Santiago Pascual (UPC TelecomBCN 2019). Garfield is a character created by Jim Davis.
- 8. 8
- 9. 9 DL basic unit: The Perceptron The Perceptron is seen as an analogy to a biological neuron, because it fire an impulse once the sum of all inputs is over a threshold. Minsky, Marvin, and Seymour A. Papert. Perceptrons: An introduction to computational geometry. 1969
- 10. 10 DL basic unit: The Perceptron
- 11. 11 DL basic unit: The Perceptron Weights and bias are the parameters that define the behavior. They must be estimated during training.
- 12. 12 DL basic unit: The Perceptron Multiple options as activation functions f(·):
- 13. 13 DL basic unit: The Perceptron A single perceptron can only define linear decision boundaries. Height 2D feature space Weight Slide credits: Santiago Pascual (UPC TelecomBCN 2019). Garfield and Odie are characters created by Jim Davis.
- 14. 14 A Layer of N Perceptrons
- 15. 15 Non-linear decision boundaries Real world data often needs a non-linear decision boundary ● Images ● Audio ● Text
- 16. 16
- 17. 17 ● Needs a “finite number of hidden neurons”: finite may be extremely large ● How to find the parameters (weights, biases) of these neurons ?
- 18. 18 Neural Network (single hidden layer)
- 19. 19 Multilayer Perceptron (MLP) In practice, deep neural networks nets can usually represent more complex functions with less total neurons (and therefore, less parameters)
- 20. 20 Multilayer Perceptron (MLP) INPUT(x) OUTPUT(y) FeedForward Hidden States h1 & h2 Feed-forward Weights (Wi ) Figure: Hugo Larochelle
- 21. 21 Deep Neural Networks (DNN) s1 s2 s3 CAT DEER DOG . . . Keep stacking hidden layers to build deep nets . . . . . . . . . Slide credits: Santiago Pascual (UPC TelecomBCN 2019). Garfield is a character created by Jim Davis.
- 22. 22 Deep Neural Networks (DNN) Slide credit: Santiago Pascual (UPC TelecomBCN 2019) s1 s2 s3 CAT DEER DOG . . . Keep stacking hidden layers to build deep nets . . . . . . . . . The concept of Deep Learning arises when we have deep models (many layers of processing), like in Deep Neural Networks (DNNs)
- 23. 23 Deep (Hierarchical) Data Representations Slide credit: Santiago Pascual (UPC TelecomBCN 2019) Image Speech Figure ref
- 24. 24 How to estimate the parameters ? Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. "Learning representations by back-propagating errors." Cognitive modeling 5, no. 3 (1988). Training a neural network with the back-propagation algorithm.
- 25. 25 How to learn a memory unit ? #RNN Alex Graves, “Supervised Sequence Labelling with Recurrent Neural Networks” The hidden layers and the output depend from previous states of the hidden layers Recurrent layer (RNN)
- 26. 26 How to learn a memory unit ? #RNN Alex Graves, “Supervised Sequence Labelling with Recurrent Neural Networks” Recurrent Weights (U) Feed-forward Weights (W)
- 27. 27 How to reuse neurons ? Fully Connected layer (FC) Convolutional layer (Conv) Figures: Ranzatto
- 28. 28 Convolutional Neural Network (CNN) #CNN #LeNet-5 LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.
- 29. 29Oriol Vinyals, ”The Deep Learning Toolkt”. MIT Embodied Intelligence Seminar (2020)
- 30. 30 Many other researchers have also contributed to the field as, for example, those pointed out by LSTM co-author Jürgen Schmidhuber in “Deep Learning Conspirancy”.
- 31. 31Jitendra Malik, “What lead computer vision to deep learning ?” ACM Communications 2017.
- 32. 32 Big data for Vision: ImageNet ● 1,000 object classes (categories). ● Images: ○ 1.2 M train ○ 100k test. Deng, Jia, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. "Imagenet: A large-scale hierarchical image database." CVPR 2009.
- 33. 33 Data Challenge: Social Biases #Equalizer Burns, Kaylee, Lisa Anne Hendricks, Trevor Darrell, and Anna Rohrbach. "Women also Snowboard: Overcoming Bias in Captioning Models." ECCV 2018.
- 34. 34 Data Challenge: Who owns data ? Personal data Internet of things - IoT Neil Lawrence, OpenAI won’t benefit humanity without open data sharing (The Guardian, 2015)
- 35. 35Jitendra Malik, “What lead computer vision to deep learning ?” ACM Communications 2017.
- 36. 36 Computation
- 37. 37 Computation ecological cost Strubell, Emma, Ananya Ganesh, and Andrew McCallum. "Energy and Policy Considerations for Deep Learning in NLP." ACL 2019. [tweet]
- 38. 38Jitendra Malik, “What lead computer vision to deep learning ?” ACM Communications 2017.
- 39. 39 Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." NIPS 2012 649,63 citations (June 2020)
- 40. 40 ● 1,000 object classes (categories). ● Images: ○ 1.2 M train ○ 100k test. ImageNet Challenge Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang et al. "Imagenet large scale visual recognition challenge." International Journal of Computer Vision 115, no. 3 (2015): 211-252. [web]
- 41. 41 ImageNet Challenge Russakovsky, Olga, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang et al. "Imagenet large scale visual recognition challenge." International Journal of Computer Vision 115, no. 3 (2015): 211-252. [web] Slide credit: Rob Fergus (NYU) -9.8% Classic Machine Learning
- 43. 43 ImageNet Image Recognition Electronic Frontier Foundation: “Measuring the Progress of AI Research” (2017)
- 44. 44 Learning Representations Jitendra Malik, “What lead computer vision to deep learning ?” ACM Communications 2017.
- 48. Reinforcement Learning (RL) Figure: Lilian Weng, “A (Long) Peek into Reinforcement Learning” (2018)
- 49. Deep Reinforcement Learning (DRL) Deep Reinforcement Learning (DRL) refers agents controlled by deep neural networks.
- 50. 50 Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. "Playing atari with deep reinforcement learning." NIPS Deep Learning Workshop (2013).
- 51. 51 Beyond Multimedia #AlphaGo Silver, David, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser et al. "Mastering the game of Go with deep neural networks and tree search." Nature 2016.
- 52. Wayve, “Sim2Real: Learning to Drive from Simulation without Real World Labels” (2018)
- 53. 53 The AI Hype (?)
- 54. 54 The AI Hype (?)
- 55. 55 The AI Hype (?)
- 56. 56 The AI Hype (?)
- 57. Deep Learning @ Barcelona, Catalonia
- 58. Deep Learning @ UPC TelecomBCN Foundations ● MSc course [2017] [2018] [2019] ● BSc course [2018] [2019] [2020] Multimedia Vision: [2016] [2017][2018][2019] Language & Speech: [2017] [2018] [2019] Reinforcement Learning ● [2020 Spring] [2020 Autumn]
- 59. Deep Learning @ UPC School 4th edition starts November 2020. Sign up here.