Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (13)
Similar a Deep Generative Models
Similar a Deep Generative Models (20)
Último
Último (20)
Deep Generative Models
- 2. Discriminative vs. Generative Learning Discriminative Learning Generative Learning Learn 𝑝(𝑦|𝑥) directly Model 𝑝 𝑦 , 𝑝 𝑥 𝑦 first, Then derive the posterior distribution: 𝑝 𝑦 𝑥 = 𝑝 𝑥 𝑦 𝑝(𝑦) 𝑝(𝑥) 2
- 3. Undirected Graph vs. Directed Graph Undirected Directed • Boltzmann Machines • Restricted Boltzmann Machines • Deep Boltzmann Machines • Sigmoid Belief Networks • Variational Autoencoders (VAE) • Generative Adversarial Networks (GAN) 3 a b c a b c Deep Belief Networks
- 4. Boltzmann Machines • Stochastic Recurrent Neural Network and Markov Random Field invented by Hinton and Sejnowski in 1985 • 𝑷 𝒙 = 𝐞𝐱𝐩(−𝑬 𝒙 ) 𝒁 > E(x): Energy function > Z: partition function where σ 𝑥 𝑃 𝑥 = 1 • Energy-based model: positive values all the time • Single visible layer and single hidden layer • Fully connected: not practical to implement 4
- 5. Restricted Boltzmann Machines • Dimensionality reduction, classification, regression, collaborative filtering, feature learning and topic modeling • 𝑷 𝐯 = 𝒗, 𝐡 = 𝒉 = 𝟏 𝒁 𝐞𝐱𝐩(−𝑬 𝒗, 𝒉 ) • Two layers like BMs • Building blocks of deep probabilistic models • Gibbs sampling with Contrastive Divergence (CD) or Persistent CD 5
- 6. Comparison btw BMs and RBMs Boltzmann Machines Restricted Boltzmann Machines v1 v2 v3 h1 h2 h3 h4 v1 v2 v3 h1 h2 h3 h4 6 𝑬 𝒗, 𝒉 = −𝒗 𝑻 𝑹𝒗 − 𝒉 𝑻 𝑺𝒉 − 𝒗 𝑻 𝑾𝒉 − 𝒃 𝑻 𝒗 − 𝒄 𝑻 𝒉 𝑬 𝒗, 𝒉 = −𝒃 𝑻 𝒗 − 𝒄 𝑻 𝒉 − 𝒗 𝑻 𝑾𝒉
- 7. Deep Belief Networks • Unsupervised • Small dataset • Stacked RBMs • Pre-train each RBM • Undirected + Directed 7 RBM Sigmoid Belief Net
- 8. • 𝑃 v, h1, h2, h3 = 𝑃 v h1 𝑃 h1 h2 𝑃(h2, h3) • 𝑃 v h1 = ς𝑖 𝑃(𝑣𝑖|h1) • 𝑃 h1 h2 = ς 𝑗 𝑃(ℎ𝑗 1 |h2 ) • 𝑃 h2 , h3 = 1 𝑍(𝑊3) exp(h2𝑇 𝑊3 h3 ) 8 RBM Sigmoid Belief Net ℎ1 ℎ2 ℎ3 v 𝑊3 𝑊2 𝑊1 Sigmoid Belief Net RBM
- 9. Limitations of DBN (By Ruslan Salakhutdinov) • Explaining away • Greedy layer-wise pre-training > no optimization over all layers • Approximation inference is feed-forward > no bottom-up and top-down 9 http://www.slideshare.net/zukun/p05-deep-boltzmann-machines-cvpr2012-deep-learning-methods-for-vision
- 10. Deep Boltzmann Machines • Unsupervised • Small dataset • Stacked RBMs • Pre-train each RBM • Undirected 10
- 11. • 𝑃 𝜃 v = 1 𝑍(𝜃) σh1,h2,h3 exp( v 𝑇 𝑊1h1 + h1𝑇 𝑊2h2 + h2𝑇 𝑊3h3) • 𝜃 = {𝑊1, 𝑊2, 𝑊3} • Bottom-up and Top-down: • 𝑃 ℎ𝑗 2 = 1|h1, h3 = 𝜎(σ 𝒌 𝑾 𝒌𝒋 𝟑 𝒉 𝒌 𝟑 + σ 𝒎 𝑾 𝒎𝒋 𝟐 𝒉 𝒎 𝟏 ) 11 ℎ1 ℎ2 ℎ3 v 𝑊3 𝑊2 𝑊1
- 12. Variational Autoencoders (VAE) 12 Encoder Decoderx x’ μ σ Z 𝑞(𝑧|𝑥) 𝑝(𝑥|𝑧) Sampling Reconstruct ℒ 𝑞 = −𝑫 𝑲𝑳 𝒒 𝒛 𝒙 ∥ 𝒑 𝒎𝒐𝒅𝒆𝒍 𝒛 + 𝔼 𝒛~𝒒 𝒛 𝒙 𝐥𝐨𝐠 𝒑 𝒎𝒐𝒅𝒆𝒍(𝒙|𝒛) 𝔃 ~ 𝓝(𝝁, 𝝈) http://www.slideshare.net/KazukiNitta/variational-autoencoder-68705109
- 13. Stochastic Gradient Variational Bayes (SGVB) Estimator 13 Encoder Decoderx x’ μ σ Z Back Prop. Feed Forward 𝝐 ~ 𝓝 𝟎, 𝑰 𝔃 = 𝝁 + 𝝐𝝈
- 15. Generative Adversarial Networks(GAN) 15 Data Sample Discriminator Generator Sample Noise Yes or No Generator https://ishmaelbelghazi.github.io/ALI
- 16. Convolutional Neural Networks 16 Image Classification
- 17. Deep Convolutional Generative Adversarial Networks (DCGAN) 17 https://openai.com/blog/generative-models/
- 18. Real Images vs. Generated images 18 http://kenkihara.com/projects/GAN.html