Learning with side information through modality hallucination, J. Hoffman et al., CVPR2016 http://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Hoffman_Learning_With_Side_CVPR_2016_paper.html

1. Terry Taewoong Um (terry.t.um@gmail.com)
University of Waterloo
Department of Electrical & Computer Engineering
Terry Taewoong Um
LEARNING WITH SIDE INFOR-
MATION THROUGH MODALITY
HALLUCINATION (2016)
1

2. Terry Taewoong Um (terry.t.um@gmail.com)
BEYOND SUPERVISED / UNSUPERVISED
2
supervised learning semi-supervised learning weakly-supervised learning
“Is object localization for free? Weakly-supervised
learning with convolutional neural networks (2015)”, M.
Oquab et al.
“Bayesian Semisupervised Learning with Deep Generative Models (2017)”, J. Gordon
et al.
• Various learning scenarios
• Learning with side information (modality)
(training) (test)

3. Terry Taewoong Um (terry.t.um@gmail.com)
MISSING INPUT DURING TEST
3
(training) (test)
Couch
zero-
padding…?
???

4. Terry Taewoong Um (terry.t.um@gmail.com)
MISSING INPUT DURING TEST
4
(training) (test)
Couch ???
generate

5. Terry Taewoong Um (terry.t.um@gmail.com)
MISSING INPUT DURING TEST
5
(training)
???
(test)
generate
Couch

6. Terry Taewoong Um (terry.t.um@gmail.com)
HALLUCINATION
6
(training) (test)
The red & blue should make similar features :

7. Terry Taewoong Um (terry.t.um@gmail.com)
RELATED WORKS
7
• RGB-D detection : exploit depth images
• Transfer learning and domain adaptation
: transfer the knowledge from a depth image to a RGB image
• Learning using privileged information : Training with a teacher
x : X-ray
x* : Clinician’s interpretation
y : Cancer Y/N
• Distillation : the output from one network is used as the target for a new network.

8. LOSS FUNCTION
8
Hallucination
Classification
Localization

9. LOSS FUNCTION
9
Hallucination
Classification
Localization

10. LOSS FUNCTION
10
Hallucination
Classification
Localization

11. SEVERAL ISSUES
11
Terry Taewoong Um (terry.t.um@gmail.com)
• Training & Initialization
: First train the RGB & D-Net, and copy the D-Net to H-Net
• Which layer to hallucinate? Pool5

12. RESULTS
12
Terry Taewoong Um (terry.t.um@gmail.com)
• With new dataset (Pascal voc 2007)
• With trained dataset (NYUD2)

13. RESULTS
13
Terry Taewoong Um (terry.t.um@gmail.com)
RGB-D-H (O)
RGB (X)
RGB-D-H (X)
RGB (O)

14. SUMMARY
14
Terry Taewoong Um (terry.t.um@gmail.com)
• If you have a missing modality at test time,
(Or if you have additional modality at training time,)
hallucinate!
• Good idea, but not a in-depth understanding…
• How can a RGB image “imagine” its missing depth image?
(Can we visualize
• Is the learned H-net generalizable to new images?
• Is this method effective to other modalities as well?
• Can we propose a domain-specific hallucination architecture?
• We may exploit more information (modalities) at training time than run-time
• Beyond supervised / unsupervised settings….