Tensorflow-KR 논문읽기모임 44번째 발표자료입니다 영상링크 : https://youtu.be/7UoOFKcyIvM 논문링크 : https://arxiv.org/abs/1704.04861

1. MobileNets:
Efficient Convolutional Neural Networks for
MobileVision Applications
29th October, 2017
PR12 Paper Review
Jinwon Lee
Samsung Electronics

2. MobileNets

3. Key Requirements for Commercial Computer
Vision Usage
• Data-centers(Clouds)
 Rarely safety-critical
 Low power is nice to have
 Real-time is preferable
• Gadgets – Smartphones, Self-driving cars, Drones, etc.
 Usually safety-critical(except smartphones)
 Low power is must-have
 Real-time is required
Slide Credit : Small Deep Neural Networks - Their Advantages, and Their Design by Forrest Iandola

4. What’s the “Right” Neural Network for Use in a
Gadget?
• Desirable Properties
 Sufficiently high accuracy
 Low computational complexity
 Low energy usage
 Small model size
Slide Credit : Small Deep Neural Networks - Their Advantages, and Their Design by Forrest Iandola

5. Why Small Deep Neural Networks?
• Small DNNs train faster on distributed hardware
• Small DNNs are more deployable on embedded processors
• Small DNNs are easily updatable Over-The-Air(OTA)
Slide Credit : Small Deep Neural Networks - Their Advantages, and Their Design by Forrest Iandola

6. Techniques for Small Deep Neural Networks
• Remove Fully-Connected Layers
• Kernel Reduction ( 3x3  1x1 )
• Channel Reduction
• Evenly Spaced Downsampling
• Depthwise Separable Convolutions
• Shuffle Operations
• Distillation & Compression
Slide Credit : Small Deep Neural Networks - Their Advantages, and Their Design by Forrest Iandola

7. Key Idea : Depthwise Separable Convolution!

8. Recap – Convolution Operation
1 1 1 1 0
1 1 1 1 0
0 0 0 1 1
0 1 1 1 0
0 1 1 0 0
0 1 1 0 1
0 1 1 0 0
0 0 1 1 0
0 0 1 1 1
1 1 1 0 0
1 1 1 0 0
0 1 1 1 0
0 0 1 1 1
0 0 1 1 0
0 1 1 0 0
-1 0 0
0 1 0
0 0 -1
0 -1 0
-1 1 -1
0 -1 0
1 0 1
0 1 0
1 0 1
-1 0 -1
0 1 0
0 0 -1
0 -1 0
-1 1 0
0 -1 0
1 0 1
0 -1 0
1 0 1
1 -1 1
0 -1 -1
3 1 0
3 0 1
-2 0 2
0 2 3
=
convolution
Input channel : 3 Output channel : 2# of filters : 2

9. Recap –VGG, Inception-v3
• VGG – use only 3x3 convolution
 Stack of 3x3 conv layers has same effective receptive
field as 5x5 or 7x7 conv layer
 Deeper means more non-linearities
 Fewer parameters: 2 x (3 x 3 x C) vs (5 x 5 x C)
 regularization effect
• Inception-v3
 Factorization of filters

10. Why should we always consider all channels?

11. Standard Convolution
Depthwise convolution
Figures from http://machinethink.net/blog/googles-mobile-net-architecture-on-iphone/

12. Depthwise Separable Convolution
• Depthwise Convolution + Pointwise Convolution(1x1 convolution)
Depthwise convolution Pointwise convolution
Figures from http://machinethink.net/blog/googles-mobile-net-architecture-on-iphone/

13. Standard Convolution vs Depthwise Separable
Convolution

14. Standard Convolution vs Depthwise Separable
Convolution
• Standard convolutions have the computational cost of
 DK x DK x M x N x DF x DF
• Depthwise separable convolutions cost
 DK x DK x M x DF x DF + M x N x DF x DF
• Reduction in computations
 1 / N + 1 / DK
2
 If we use 3x3 depthwise separable convolutions, we get between 8 to 9 times
less computations
DK : width/height of filters
DF : width/height of feature maps
M : number of input channels
N : number of output channels(number of filters)

15. Depthwise Separable Convolutions

16. Model Structure

17. Width Multiplier & Resolution Multiplier
• Width Multiplier –Thinner Models
 For a given layer and width multiplier α, the number of input channels M
becomes αM and the number of output channels N becomes αN – where α
with typical settings of 1, 0.75, 0.6 and 0.25
• Resolution Multiplier – Reduced Representation
 The second hyper-parameter to reduce the computational cost of a neural
network is a resolution multiplier ρ
 0<ρ≤1, which is typically set of implicitly so that input resolution of network
is 224, 192, 160 or 128(ρ = 1, 0.857, 0.714, 0.571)
• Computational cost:
DK x DK x αM x ρDF x ρDF + αM x αN x ρDF x ρDF

18. Width Multiplier & Resolution Multiplier

19. Experiments – Model Choices

20. Model Shrinking Hyperparameters

21. Model Shrinking Hyperparameters

22. Results

23. Results
PlaNet : 52M parameters, 5.74B mult-adds
MobilNet : 13M parameters, 0.58M mult-adds

24. Results

25. Results

26. Tensorflow Implementation
https://github.com/Zehaos/MobileNet/blob/master/nets/mobilenet.py