SUMMER TRAINING REPORT ON BUILDING CONSTRUCTION.docx
Interpretability and informatics of deep learning in medical images3
1. Interpretability and informatics of
deep learning in medical images
Namkug Kim, PhD
Medical Imaging & Intelligent Reality Lab.
Convergence Medicine/Radiology,
University of Ulsan College of Medicine
Asan Medical Center
South Korea
2. Researches with
LG Electronics
Coreline Soft Inc.
Osstem Implant
CGBio
VUNO
Kakaobrain
Conflict of Interests
Stockholder
Coreline Soft, Inc.
AnyMedi
Co-founder
Somansa Inc.
Cybermed Inc.
Clinical Imaging Solution, Inc
AnyMedi, Inc.
Selected Grants as PI
NRF, South Korea
7T용 4D 자기공명유속영상을이용한 심뇌혈관 질환의 in-vivo유동 정량화 SW개발, 2016
4D flow MRI을 이용한 심혈관 질환의 in-vivo유동 연구, 2015-7
자기공명분광영상 및 MRI의 통합 분석 소프트웨어 개발
KEIT, South Korea
DigitalDentistry, 2018-2022
의료영상 인공지능 PACS 과제, 2016-20
3DP 척추 맞춤형 임플란트, 2016-20
3D 프린터 기반 무치악 및 두개악안면결손환자용 수복 보철물 제작, 재건 시스템 개발, 2015-9
근골격계 복구 수술 로봇 개발, 2012-7
영상중재시술 로봇시스템 개발, 2012-7
Spine및 Neurosurgery 수술보조용 항법 시스템 개발, 2001
의료용 3차원 모델 제작 S/W 기술 개발, 정통부, 2000
의료영상재구성에 의한 가상시술 소프트웨어 개발, 중소기업기술혁신개발, 중기청, 2001
KHIDI, South Korea
연구중심병원 육성과제, 2019-2028
인공지능 학습센터 과제, 2018-2023
영상 뇌졸중 예후 예측 및 치료방침 결정 시스템 개발, 2012-8
관동맥 관류 CT 의 자동 진단 프로그램을 활용한 허혈성 질환의 진단과 치료, 2013-6
RP를 이용한 척추나사못 삽입술 계획 프로그램 개발, 2000
Companies Fundings
SiemensGermany, Hyundai Heavy Industry, OsstemImplant, S&G Biotech, Corelinesoft, MidasIT, AnyMedi,
Hitachi Medical,Japan, Kakaobrain
3. Big data : IoT Thermometer
IoT Thermometer
Kinsa : Startup @ USA
Real-time body temperature bigdata@USA
Patients-derived health data
Regional basis
Influenza stats
Kinsa : Realtime vs CDC : 3 week delay
B2B model:
Demands and production
독감 예방 접종 혹은 항생제, 살균제 등의 약
칫솔, 오렌지 쥬스, 수프 등
7
Influenza trends: comparison
with CDC 2.5 y
5. Big data : Facebook
9
Correlation between Facebook usage vs drug addiction
Accuracy : Tobacco(86%), Alcohol(81%), Drug(84%)
6. Healthcare Bigdata
7.5 Exa Byte/day (30% of every data)
생물학적 특성, 건강 이력, 웰빙 상태, 가는 곳, 지출 내역, 수면 상태, 식사와 배
설 / 실험 기록, 의학 영상, 유전정보, 액체생체검사, 심전도 /보험금 청구서, 임
상시험, 처방전, etc
10IBM Healthcare
7. Precision Medicine
11 / 46
• Bigdata
• -Omics
• Genomics, Transcriptomics,
Proteonomics, Metabolomics, …
• Personalized/Precision
Backgrounds
Big Data
NGS
Lab
Imaging
EMR
8. Opportunity
12
8 trillion exam:
Healthcare Industry
2 trillion : wastes in
healthcare industry
Better experience
Imaging :
Unnecessary tests
Lower cost
Oncology:
Variability of Care
Better outcomes
Life sciences:
Failed clinical trials
Government:
Fraud, Waste and Abuse
Value Based Care:
Cost of chronic disease
360 billion : total IT and
healthcare market
opportunity
*IBM Watson
9. Beyond Human-level Performance
• Now, Machines Beat Human in Tasks Once Considered Impossible
5:0
vs Fan Hui
(Oct. 2015)
4:1
vs Sedol Lee
(Mar. 2016)
Modified by KH Jung, PhD
10. Beyond Human-level Performance
• Now, Machines Beat Human in Tasks Once Considered Impossible
TPU Server
used against Lee Sedol
TPU Board used
against Ke Jie
11. Beyond Human-level Performance
• Now, Machines Beat Human in Tasks Once Considered Impossible
Libratus(Jan 30, 2017) DeepStack(Science, Mar 02, 2017)
17. Bio Plausible Neural Network
Mimic human visual recognition
system
Neocognitron, proposed by
Hubel & Wiesel in 1959
Visual primary cortex by cascading
from S-Cell to C-Cell
Each unit connected to a small
subset of other units
Based on what it sees, it decides
what it wants to say
Units must learn to cooperate to
accomplish the task
21From Gallant and van Esses, Simon Thorpe
18. CNN : Major Breakthroughs in Feedforward NN
K. Fukushima Yann Lecun G. Hinton, S. Ruslan
Neocognitron (1979)
• By Kunihiko Fukushima
• First proposed CNN
Convolutional Neural Networks (1989)
• Yann Lecun et.al
• Back propagation for CNN
• Theoretically learn any function
Neocognitron
LeNet-5 architecture
Alex krizhevsky , Hinton
LeNet-5 (1998)
• Convolutional networks
Improved by Yann Lecun et.al
• Classify handwritten digits
D. Rumelhart, G. Hinton, R.
Wiliams
1960 1970 1980 1990 2000 2010 2012
Perceptron
XOR
Problem
Golden Age
1957 1969 1986
F. RosenblattM. Minsky, S. Papert
• Adjustable weights
• Weights are not learned
• XOR problem is not linearly
separble
• Solution to nonlinearity separable problems
• Big computation, local optima and
overfiting
CNN Breakthrough (2012)
• By Alex Krizhevsky et al.
• Winner of ILSVRC2012 by
large marginDark Age (AI winter)
Back propagation (1981)
• Train multiple layers
Multi-layer Perceptron
(1986)
1950
Neocognitron (1959)
• Hubel & Wiesel
• by cascading from S-
Cell to C-Cell
19. Feature Engineering vs Feature Learning
Modified From
Yann LeCun
Knowledge-driven Feature Engineering
Conventional Radiomics
Data-driven Feature Learning
Deep Radiomics
•Feature Learning instead of Conventional Feature Engineering Removes Barriers for
Multi-modal Studies and Data-driven Approaches in Medical Data Analysis
20. Machine Learning vs Deep Learning
— Scale Matters
— Millions to Billions of parameters
— Data Matters
— Regularize using more data
— Productivity Matters
— It’s simple, so we can make tools
Data & Compute
Accuracy Deep Learning
Many previous
methods
Deep learning is most
useful for large problems
Modified by Nvidia DLI
21. Computational map
25
Dense
Few
Millions
#ofVariables
Completeness of Data (Sampling)Sparse
More
• Compute
• Data
• Storage
• Bandwidth
Computationally
Intractable SpaceDeep Learning
Neural Nets
Statistical Analysis
Algorithms, Closed Form Solutions
Expert
Systems
Insufficient
data for analysis
[Un]supervised Learning
Models
Intuition
29. AI Medical Device
33
• Verily@Google: Normal vs Abn, Anti-aging, Life
Prolongation
• IBM: 트루벤 인수등 40B USD
• Apple: GSK, EMR +iPhone Healthcare Platform
• Facebook: Incurable dx, Human cell atlas, 5000M USD
• Zebra Medical Vision
• AI medical imaging Dx : 1st place of investment
• Medical imaging reading cloud service/1 USD
30. Clinical Unmet Needs on Deep Learning
효율적인 데이터 비식별화, Curation, 및 인공지능 기술을 이용한 스마트 레이블링 기술
다양한 장비와 병원마다의 차이를 극복하는 도메인 적응(Domain adaptation) 기술
블랙박스 성격을 완화하기 위한 인공지능 판단 해석(Interpretability) 및 시각화(Visualization) 연
구
의료 데이터가 가지고 있는 불확실성(Uncertainty), 인공지능의 판단이 가지고 있는 불확실성을
평가하는 기술
질환별 편향(imbalance) 문제를 해결하기 위한 전처리 및 인공지능 학습 기술
한번도 보지 못한 새로운 클래스를 미지판단(Novelty)을 통해서 검출해서 추후 의사가 따로 평가
희귀 데이터나 소수의 데이터를 학습하기 위한 One/Multi-shot Learning 기술 개발
반복측정된 영상데이터를 이용하여 딥러닝이 가지고 있는 재현가능성(Reproducibility) 연구
Adversarial Attack에 강인한 인공지능 연구
여러 물리 및 의학 법칙 등을 이용하여 기계학습(Physics Induced Machine Learning)의 효율을
증가하는 연구
34
34. 해석력 강화 모델
38
Patrick Hall et al. (O’REILLY, 2017) Macro Tulio Ribeiro et al. (O’REILLY, 2016)
Surrogate models Local-Interpretable-Model-agnostic Explanations
(Perturbation)
35. 해석력 강화 모델
39
Yin Lou et al, ACM 2012
GAM, Generalized Additive Models Everything should be made as simple as possible, but
not simpler. — Albert Einstein.
• 변수간 상호작용 효과를 배제
• 설명력 제고
• 개별 변수별로 복잡한 구조의
알고리즘을 적용한 후 이를
더하기 형태로 종합
36. Analysis on Deep Learning Methods for Predicting Patient Survival
Basic CNN Modified CNN Multi-layer CNN Residual NetworkFeed-forward NN
37. Interpretability : Machine Operable, Human Readable
Visual attention
Category – feature mapping
Sparsity and diversity
41
51. Uncertainty
Uncertainty of training data
In clinical situation, it is common
Deep Bayesian Modeling
Uncertainty of classification/prediction of Machine Learning
55
52. Novelty (Untrained catergory)
In clinical situation
Novelty is everywhere, especially supervised learning
Rare diseases, but well known to medical doctors
Hard to training
How to determine novel (untrained) category
Unsupervised learning
Semi-unsupervised learning
Normal vs abnormal
Abnormality Detection
56
53. Reproducibility
Test-retest is one of most important issues of biomarker
Multiple scanning within similar date
Evaluate reliability of AI/Deep learning
Chest PA (Nodule)
Nodule Size on Chest PA with YOLO
– 56% : Variability of Size Marker
– Chest PA 50 pairs
DILD CT
57
62. Job Opening @ MI2RL_AMC, Seoul, SouthKorea
Post-doc research fellow, PhD Students, Researchers
116
AMC, UoU
Seoul South Korea
Contact (namkugkim@gmail.com)
63. Collaborators
Radiology
Joon Beom Seo, SangMin LeeA,B, Dong Hyun, Yang, Hyung Jin Won, Ho Sung Kim, Seung
Chai Jung, Ji Eun Park, So Jung Lee,Jeong Hyun Lee, Gilsun Hong
Neurology
Dong-Wha Kang, Chongsik Lee, Jaehong Lee, Sangbeom Jun, Misun Kwon, Beomjun Kim
Cardiology
Jaekwan Song, Jongmin Song, Young-Hak Kim
Emergency Medicine
Dong-Woo Seo
Pathology
Hyunjeong Go, Gyuheon Choi, Gyungyub Gong, Dong Eun Song
Surgery
Beom Seok Ko, JongHun Jeong, Songchuk Kim, Tae-Yon Sung
Internal Medicine
Jeongsik Byeon, Kang Mo Kim
Anesthesiology
Sung-Hoon Kim, Eun Ho Lee