의료 인공지능 교육 워크샵 (17.09.16) 발표자료 - Generative Adversarial Network에 대한 소개 - Medical imaging 분야에서의 연구 사례 - Our research

1. Copyright © KakaoBrain Corp. All rights reserved.
Generative Adversarial Networks
and Their Applications in Medical Imaging
September 16, 2017
Sanghoon Hong

2. 1. Generative Adversarial Network (GAN)
2. GAN in Medical Imaging + Our Research Project
Contents

3. • Discriminative vs. Generative
Generative Model
CNN classifier GAN
“cat”
an inverse function?

4. • Discriminative vs. Generative
Generative Model
https://duphan.wordpress.com/tag/generative-model/
data distribution
p(x,z) or p(x|z)
Difficult, but important

5. • Data distribution
Generative Model
A slide from “Crash Course on Machine Learning Part II”
주요 특징만 필요
세밀한 특성, 분포를
완전히 알아야함

6. • Applications?
• Structured prediction (e.g., output text)
• Much more robust prediction
• Anomaly detection
• …
Generative Model

7. Generative Model
Slide credit Goodfellow 2016

8. “the coolest idea in ML in the last
twenty years”
- Yann LeCun
Generative Adversarial Network (GAN)

9. Generative Adversarial Network (GAN)
https://github.com/hindupuravinash/the-gan-zoo/

10. • Variational Auto-Encoder
• Data의 generation은 잘 됨
• 그러나 blurry face
Why GANs are different?
“Variational Autoencoder and
Extensions”
by Aaron CouCourville
Sample
interpolation

11. • Why are generated samples blurry?
• Regress to the mean => blurry images
Why GANs are different?
Ledig, C. et al.
Photo-Realistic Single Image Super-Resolution Using a
Generative Adversarial Network.
MSE or Euclidean distance

12. Why GANs are different?
from “Tutorial on Theory and Application of Generative Adversarial Networks” (CVPR17)

13. • Difficult to hand-craft a good perceptual loss
function
• => 그럴듯한지 여부를 판단하는 neural network가
있다면?
Generative Adversarial Network

14. Generative Adversarial Network
Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)

15. Generative Adversarial Network
Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)
real / fake 판단하는 네트워크

16. Generative Adversarial Network
https://ibmathsresources.com/2014/08/27/zenos-paradox-achilles-and-the-tortoise/
DG
Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary
Equilibrium Generative Adversarial Networks.
DG

17. Training GANs
• Discriminator training
Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)

18. Training GANs
• Generator training
Deep Learning for Computer Vision: Generative models and adversarial training (UPC2016)

19. • Adversarial loss (vanilla GAN)
Training GANs
D Real sample은 D(x) -> 1 Generated sample은 D(x) -> 0
G Generated sample도 D(x) -> 1

20. • Iterate these two steps until convergence
• Eventually (we hope) that the generator gets so good that it
is impossible for the discriminator to tell the difference
between real and generated images. Discriminator
accuracy = 0.5
Training GANs
Discriminator
Data
Model
Distribution
Random guess

21. Training GANs
Tutorial on Theory and Application of Generative Adversarial Networks (CVPR17)

22. • In practice?
• Training stability
(D vs. G balance, …)
• Mode collapse
Training GANs
Generative Models II (CIFAR-CRM DLSS 2017) by Aaron Courville
An image from “Generative adversarial networks” by Namju Kim

23. Training GANs
https://github.com/hindupuravinash/the-gan-zoo

24. • Realistic image!
GAN Results & Applications
Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary
Equilibrium Generative Adversarial Networks.
realistic & diverse samples
Nguyen, A. et al. (2016). Plug & Play
Generative Networks: Conditional Iterative
Generation of Images in Latent Space

25. • Representation interpolation
GAN Results & Applications
Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: Boundary
Equilibrium Generative Adversarial Networks.
Memorizing X

26. • Super-resolution
GAN Results & Applications
Ledig, C. et al. (2016) Photo-Realistic Single Image
Super-Resolution Using a Generative Adversarial
Network.

27. GAN Results & Applications
Low-res image
Generated
high-res image
Generated
or not?

28. • Image-to-image translation
GAN Results & Applications
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2016). Image-to-
Image Translation with Conditional Adversarial Networks.

29. GAN Results & Applications

30. GAN Results & Applications

31. Copyright © KakaoBrain Corp. All rights reserved.
GAN in Medical
Imaging

32. • Low-dose CT (noisy) => Routine-dose CT
Literature Review (1) De-noising

33. Literature Review (1) De-noising
loss = (voxel-similarity loss) + adversarial loss

34. Literature Review (1) De-noising
Low-dose Routine-doseGAN-denoised
“Training with an adversarial network allows the generator to better learn the
noise distribution in routine-dose CT and produce more realistic images for
more accurate coronary calcium quantification.”
MSE-denoised

35. • CT => PET image translation
Literature Review (2) - Image-to-image translation

36. Literature Review (2) - Image-to-image translation

37. Literature Review (2) - Image-to-image translation
SL + GAN
A mask with high predicted SUV values (>2.5) (???)

38. Literature Review (2) - Image-to-image translation
input CT
ground-truth PET
generated PET

39. • MRI => CT image translation
Literature Review (2) - Image-to-image translation

40. Literature Review (2) - Image-to-image translation
Supervised training w/ real data
Blurry More realistic

41. Literature Review (2) - Image-to-image translation
Pelvic datasetBrain dataset
Best mean absolute error & peak signal-to-noise ratio

42. • GAN research가 있지만, 아직 결과물의 수준이나 양
이 부족
• 그럼 우리는 무엇을 해볼 수 있을까?
Our Research

43. • Medical image generation with GAN
Our Research
Noise

44. • Applications?
• Unsupervised or semi-supervised training
• Progression forecast & Visualization
Our Research

45. • Medical image generation from scratch?
• Structure + Dynamics + Variations + …
How to Tackle

46. • Medical image generation from scratch?
• Structure + Dynamics + Variations + …
• Might be too difficult
How to Tackle
(GAN은 global structure / counting 같은 것에서 특히 약함)

47. • (First step) image generation w/ structural hints
• Structure + Dynamics + Variations + …
• Image-to-image translation과 유사
How to Tackle

48. • Gaze estimation task
• Model-based synthetic data => GAN => Realistic
data
How to Tackle

49. How to Tackle
Self-regularization
Visual Turing test => 51.7% acc.
State of the art w/o label real data

50. How to Tackle
Self-regularization (minimizing feature dist.)
Local adversarial loss (=PatchGAN)
History of refined images
History 안쓰면 artifacts 발생

51. • Possible hints?
• Model-based synthetic data
• Normal image + conditioning
• Normal image + synthetic or expert-guided label
How to Tackle
Synthetic or guided label
Generator

52. First Trial
Generator
Discriminator
Real normal
Real patient
Image + label
(for controllability)
Generated patient
Our target
- synthetic or supervised label
- Matching image-segmentation pair?
- Realistic image?

53. First Trial
Generator
Discriminator
Image + label
(for controllability)
PatchGAN
DCGAN or
MAD-GAN
Real normal
Real patient
Image matching
Generated patient

54. • Typical GAN issues (training, …)
• Sample quality? (feat. M.D. researchers)
• How to evaluate generated samples
• Practical effectiveness?
• Unsupervised (or semi-supervised)
segmentation/classification in medical domain
Have far to go

55. • One more thing…

56. Visualization of disease progression
Identity
Age

57. • GAN: an interesting & effective way to generate
data
• GAN in Medical Imaging? => 아직 초기 단계
• “Medical Image Generation with GAN”
Conclusion

58. THANK YOU