Talk on Super Resolution models to TensorFlow and Deep Learning meetup group

1. Generative Models
Part 2: Super Resolution
By Sam Witteveen

2. Neural Network
Hidden Layers
Loss Function
Desired
Result
Optimize
Input
Often Labels
Output

3. Neural Network Inference
Hidden Layers
Raw
Input
Output
No More Loss or Optimize
No More Labels

4. Classification CNN
Conv Blocks(Conv+Pooling) Relu
Reduction of HxW
increased number of filters(K)
FCN layers
Prediction

5. The Magic of CNNs
• CNNs are not just for classification
• Don’t fall into the trap of thinking CNNs are just a
precursor to a set of dense layers, logits and
classes
• CNNs give us rich feature representations of what
we put into them, if we cut off the dense layers and
we use convolution blocks as our outputs

6. Old Style CNN
Reduction of HxW
increase of Filters
Output
All these reduce theConv Block 3x3
Dense Layer
Logits
Dense Layer
Dense Layers
Input
Max PoolingConv Block
Conv Block
Conv Block
Softmax Layer

7. Modified CNN
Reduction of HxW
increased number of filters
Output
Conv Block 3x3
Input
Conv Block
Conv Block
Conv Block
very deep (k) output
eg. lots of filters

8. The paper
"Perceptual Losses for Real-Time Style Transfer and Super-Resolution" by Johnson et.al
http://arxiv.org/abs/1603.08155

9. The SR Network
VGG2
Loss
UpSampling 4x-8x
VGG1
LowRes
input
Optimize
NonTrainable
NormalRes
input
NonTrainable
SR Net
Perceptual Loss

10. The SR Network
VGG2
Loss
UpSampling 4x-8x
VGG1
LowRes
input
Optimize
NonTrainable
NormalRes
input
NonTrainable
SR Net
Perceptual Loss
The Loss Part

11. Perceptual Loss
MSE
ConvNet
for original Image
Output Features not prediction
Perceptual Loss
ConvNet
for Up Sampled Image
Output Features not prediction

12. Up Sampling CNN
Reduction of HxW
increase of Filters
Output
Increase Receptive fieldConv Block 1x1
UpSample Block
ResNet Block
ResNet Block
ResNet Block
ResNet Block
Conv with tanh
activation
UpSample Block
Increasing Pixels
LowRes Input
What are these pixels?
Done at smaller image size

13. Receptive Field
3x3 filter on 3x3 input give 5x5 receptive field

14. Conv Block
input
Conv2D
Batch
Norm
ReLU

15. ResNet Block
Identity
input
Conv
Block
Conv
Block
Sum

16. The SR Network
VGG2
Loss
UpSampling 4x-8x
VGG1
LowRes
input
Optimize
NonTrainable
NormalRes
input
NonTrainable
SR Net
Perceptual Loss
The Loss Part

17. Code Walk through

18. CNN between anything?
Output
Trainable
input CNN

19. Colorization

20. Segmentation

21. Others
• Depth Perception
• De-noising
• Visual Filters
• Audio Clarity
• Audio Filters

22. Summary
• Generative models go far beyond just artist models
• The power of CNN beyond classification
• Perceptual Loss from comparing 2 CNNs
• Generative = image in -> image out
• Try putting a CNN between some data to
manipulate it to get what you want

23. The End
Contact
email: s@maclea.ai
twitter: sam_witteveen
https://github.com/samwit/TensorFlowTalks