Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Detecting Nuclei from Microscopy Images with Deep Learning

188 visualizaciones

Publicado el

Lecture about identifying cell nuclei from standard microscopy images with deep learning models.

Publicado en: Datos y análisis
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Detecting Nuclei from Microscopy Images with Deep Learning

  1. 1. BIIT Detecting Nuclei from Microscopy Images with Deep Learning (and not only) Nuclei:99% Nuclei:79% Nuclei:99% Nuclei:99% Nuclei:99% Nuclei:99% Nuclei:99% Dmytro Fishman dmytro@ut.ee
  2. 2. https://siteman.wustl.edu/glossary/cdr0000046470/ Biology 101
  3. 3. Cells are building blocks of all living organisms They come in different shapes, carry different functions and have various properties Biology 101
  4. 4. Human cells
  5. 5. Human cells
  6. 6. Neuron Human cells
  7. 7. Neuron Human cells
  8. 8. Neuron Red blood cell Human cells
  9. 9. Neuron Red blood cell Human cells
  10. 10. Neuron White blood cell Red blood cell Human cells
  11. 11. Neuron White blood cell Red blood cell Human cells
  12. 12. Neuron White blood cell Red blood cell Bone cell Human cells
  13. 13. Neuron White blood cell Red blood cell Bone cell Human cells
  14. 14. Neuron White blood cell Red blood cell Bone cell Egg cell Human cells
  15. 15. Neuron White blood cell Red blood cell Bone cell Egg cell Human cells
  16. 16. Neuron White blood cell Red blood cell Bone cell Egg cell Cancer cell Human cells
  17. 17. They all contain one DNA, how come they are so different? Neuron White blood cell Red blood cell Bone cell Egg cell Cancer cell Human cells
  18. 18. …AACCTGTTACAAACCG… DNA in a specific region
  19. 19. …AACCTGTTACAAACCG… Gene 1
  20. 20. …AACCTGTTACAAACCG… Gene 1 Gene 2
  21. 21. …AACCTGTTACAAACCG… Gene 1 Gene 2
  22. 22. …AACCTGTTACAAACCG… Gene 1 Gene 2 White blood cell
  23. 23. …AACCTGTTACAAACCG… White blood cell …AACCTGTTACAAACCG… Bone cell
  24. 24. …AACCTGTTACAAACCG… White blood cell Cancer cell …AACCTGTTACAAACCG…
  25. 25. …AACCTGTTACAAACCG… White blood cell Cancer cell …AACCTGTTACAAACCG… In different cell types different genes are expressed
  26. 26. But we don’t have to measure gene expression in order to say that these cells are different… Neuron White blood cell Red blood cell Bone cell Egg cell Cancer cell
  27. 27. We can look at them
  28. 28. Extraction Producing Microscopy imaging
  29. 29. Extraction Seeding Producing Microscopy imaging
  30. 30. Extraction Seeding Treating Producing Microscopy imaging
  31. 31. Extraction Seeding Treating Staining Producing Microscopy imaging
  32. 32. Extraction Seeding Treating StainingImaging Producing Microscopy imaging
  33. 33. Extraction Seeding Treating StainingImaging Fluorescent Producing Microscopy imaging
  34. 34. Extraction Seeding Treating StainingImaging Fluorescent Histology Producing Microscopy imaging
  35. 35. Extraction Seeding Treating StainingImaging Fluorescent Histology Producing Microscopy imaging
  36. 36. Extraction Seeding Treating StainingImaging Fluorescent Histology Producing Microscopy imaging
  37. 37. Extraction Seeding Treating StainingImaging Fluorescent Histology Brightfield Producing Microscopy imaging
  38. 38. Extraction Seeding Treating StainingImaging Fluorescent Histology Brightfield Producing Microscopy imaging
  39. 39. Extraction Seeding Treating Fluorescent Histology Brightfield Opera Phenix High-Content Screening System Producing Microscopy imaging
  40. 40. Fluorescent Histology Opera Phenix High-Content Screening System Extraction Seeding Treating Brightfield Producing Microscopy imaging
  41. 41. Let’s take a look at the data 1_Load_data.ipynb
  42. 42. We need to be able to detect nuclei for each of this images!
  43. 43. Ok, that is great, but…
  44. 44. Ok, that is great, but…
  45. 45. Mitotic Cells Breast cancer diagnostics Mitotic cell count is one of the key diagnostic markers of the diseaseHistology images Breast cancer is the second most common cancer in the world with an estimated 1.67 million new cancer cases annually
  46. 46. Fluorescent images Treatment Ebola Ebola virus vaccine
  47. 47. Live cell imaging t = 0
  48. 48. Live cell imaging t = 0 t = 1
  49. 49. Brightfield images Live cell imaging t = 0 t = 1 t = 2
  50. 50. Eroom’s law: #drugs discovered per $1billion
  51. 51. Current Best Method for Microscopy Image Analysis?
  52. 52. Thousand man-hours are spent manually looking at images, counting and classifying cells Current Best Method for Microscopy Image Analysis
  53. 53. Automated Microscopy Image Analysis Pipeline
  54. 54. Preprocessed Image filters contrast denoising Original Image (Fluorescent)
  55. 55. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Histogram of pixel brightness
  56. 56. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Histogram of pixel brightness Background Nuclei
  57. 57. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Background Nuclei Magical Threshold
  58. 58. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Magical Threshold Background Nuclei
  59. 59. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Background Nuclei Magical Threshold
  60. 60. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Background Nuclei Magical Threshold
  61. 61. Histogram of pixel brightness Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Histogram of pixel brightness Background Nuclei Magical Threshold
  62. 62. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Thresholding
  63. 63. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection Thresholding
  64. 64. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features Thresholding
  65. 65. nuclei #1: blue, size 29px; Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features Thresholding
  66. 66. nuclei #2: red, size 25px; nuclei #1: blue, size 29px; Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features Thresholding
  67. 67. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Thresholding
  68. 68. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Thresholding
  69. 69. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Thresholding
  70. 70. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Thresholding Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Classical Microscopy Image Analysis Pipeline
  71. 71. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Thresholding Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification We are not going to cover this today
  72. 72. Classical Microscopy Image Analysis Pipeline 2_Tresholding.ipynb
  73. 73. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Can you see a problem? Thresholding
  74. 74. Histology Cell types Image modalities Brightfield Fluorescence Lightning conditionsDifferent magnifications
  75. 75. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Thresholding Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification
  76. 76. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Thresholding
  77. 77. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Can Deep Learning step in? Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification
  78. 78. Approach I
  79. 79. Original Image Ground Truth
  80. 80. Original Image Ground Truth
  81. 81. Patches Ground Truth
  82. 82. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches
  83. 83. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches Nuclei
  84. 84. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches Nuclei Empty
  85. 85. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches Nuclei Empty Empty
  86. 86. Ground Truth Each image (patch) is then assigned a class, depending on the central pixel Patches Nuclei Empty Empty Empty
  87. 87. Ground Truth Then we randomly assign extracted patches into training and validation sets (70/30) Patches Nuclei Empty Empty Empty
  88. 88. Ground TruthPatches Nuclei Empty Empty Empty Then we randomly assign extracted patches into training and validation sets (70/30)
  89. 89. Training set Empty Empty NucleiN E Compare predictions with true labels Validation set Train the network using extracted patches from training set. Predictions generated by the network should be consistent with true labels
  90. 90. Training set Empty Empty Empty NucleiN E Compare predictions with true labels Validation set Train the network using extracted patches from training set. Predictions generated by the network should be consistent with true labels After network has been trained, evaluate its performance on unseen validation set patches Validation set
  91. 91. New Image N E Trained Neural Network Segmentation
  92. 92. New Image N E Trained Neural Network Segmentation
  93. 93. New Image N E Trained Neural Network Segmentation
  94. 94. New Image N E Trained Neural Network Segmentation
  95. 95. New Image N E Trained Neural Network Segmentation
  96. 96. New Image N E Trained Neural Network Segmentation This is a very slow procedure Dmytro Fishman, Ardi Tampuu
  97. 97. Approach II
  98. 98. Convolutional Neural Network Let’s consider the following image
  99. 99. Convolutional Neural Network Let’s consider the following image
  100. 100. Convolutional Neural Network Convolutional layer works as a filter applied to the original image
  101. 101. Convolutional Neural Network Convolutional layer works as a filter applied to the original image There are many filters in the convolutional layer, they detect different patterns 4 filters
  102. 102. Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
  103. 103. Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
  104. 104. Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
  105. 105. Convolutional Neural Network 4 filters Each filter applied to all possible 2x2 patches of the original image produces one output value
  106. 106. Convolutional Neural Network Each filter applied to all possible 2x2 patches of the original image produces one output value Repeat this process for all filters in this layer
  107. 107. Convolutional Neural Network Each filter applied to all possible 2x2 patches of the original image produces one output value Repeat this process for all filters in this layer and the next
  108. 108. Flattening The output of the last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network
  109. 109. Flattening The output of the last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network 0 1 2 7 8 9 This vector is fed into fully connected layer with as many neutrons as possible classes
  110. 110. Flattening The output of the last convolutional layer is flattened into a single vector (like we did with images) Convolutional Neural Network 0 1 2 8 9 This vector is fed into fully connected layer with as many neutrons as possible classes Each neuron outputs probabilities 7
  111. 111. Training Neural Networks (part III) http://scs.ryerson.ca/~aharley/vis/conv/
  112. 112. Encoding Decoding Autoencoder Restored image Original image
  113. 113. Compact representation Encoding Decoding Autoencoder Restored image Original image
  114. 114. Encoding Decoding Autoencoder Binary mask Original image
  115. 115. Encoding Decoding Segnet Architecture Binary mask Original image
  116. 116. Encoding Decoding U-net Architecture Binary mask Original image
  117. 117. Training U-net model to detect nuclei 3_Deep_Learning.ipynb
  118. 118. Splash of colour ResultPredicted maskOriginal image
  119. 119. Approach III: frontiers…
  120. 120. Mask R-CNN He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask r cnn. arXiv preprint arXiv:1703.06870.
  121. 121. Mask R-CNN
  122. 122. Mask R-CNN 1. Proposes bounding boxes for objects (RoI)
  123. 123. Mask R-CNN 1. Proposes bounding boxes for objects (RoI) 2. Filters out bad RoIs
  124. 124. Mask R-CNN 1. Proposes bounding boxes for objects (RoI) 2. Filters out bad RoIs 3. For each RoI builds a mask
  125. 125. Mask R-CNN 1. Proposes bounding boxes for objects (RoI) 2. Filters out bad RoIs 3. For each RoI builds a mask
  126. 126. Mask R-CNN By Daniel Majoral 1. Proposes bounding boxes for objects (RoI) 2. Filters out bad RoIs 3. For each RoI builds a mask
  127. 127. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Can Deep Learning step in? Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification
  128. 128. Original Image (Fluorescent) Preprocessed Image filters contrast denoising Segmentation mask Can Deep Learning step in? Multi-instance mask via objects detection features extraction Relevant features nuclei #2: red, size 25px; nuclei #1: blue, size 29px; nuclei #3: pink, size 22px; nuclei #4: yellow, size 19px; nuclei #5: green, size 18px; nuclei #6: purple, size 16px; nuclei #7: orange, size 14px; Cancer Healthy cells Application Classification Yes!
  129. 129. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III filters contrast denoising
  130. 130. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III E N filters contrast denoising Pixel wise segmentation
  131. 131. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III Autoencoders E N filters contrast denoising Pixel wise segmentation
  132. 132. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III Autoencoders U-net E N filters contrast denoising Pixel wise segmentation
  133. 133. Original Image (Fluorescent) Preprocessed Image Segmentation mask Can Deep Learning step in? Yes! Approach I Approach II Approach III Autoencoders U-net Mask R-CNN E N filters contrast denoising Pixel wise segmentation
  134. 134. E N Almighty Deep Learning
  135. 135. E N VS Almighty Deep Learning Dude in the lab
  136. 136. E N Almighty Deep Learning Dude in the lab VS
  137. 137. Histology Cell types Image modalities Brightfield Fluorescence Lightning conditionsDifferent magnifications
  138. 138. Fluorescent microscopy
  139. 139. Brightfield microscopy
  140. 140. Can we detect nuclei without fluorescent dye?
  141. 141. Sten-Oliver Salumaa
  142. 142. Nuclei:99% Nuclei:79% Nuclei:99% Nuclei:99% Nuclei:99% Nuclei:99% Nuclei:99% Team: Sten-Oliver Salumaa, Daniel Majoral, Dmytro Fishman, Mikhail Papkov, Ardi Tampuu, William Jones, Elizabeth Bell, Ilya Kuzovkin, Tanel Pärnamaa, Leopold Parts, Jaak Vilo, Raul Vicente, Kaupo Palo and Martin Daffertshofer
  143. 143. Something important that we learnt today
  144. 144. Something important that we learnt today Difference in gene expression profile causes cells to be different
  145. 145. Something important that we learnt today Difference in gene expression profile causes cells to be different t = 0 t = 1 t = 2 Effective cell detection can help develop better and cheaper drugs
  146. 146. Something important that we learnt today t = 0 t = 1 t = 2 Thresholding Difference in gene expression profile causes cells to be different Classical image analysis pipeline includes manual parameter tuning Effective cell detection can help develop better and cheaper drugs
  147. 147. Something important that we learnt today t = 0 t = 1 t = 2 Thresholding There are ways in which Deep Learning can make a difference Difference in gene expression profile causes cells to be different Classical image analysis pipeline includes manual parameter tuning Effective cell detection can help develop better and cheaper drugs
  148. 148. Something important that we learnt today t = 0 t = 1 t = 2 Thresholding Classical image analysis pipeline includes manual parameter tuning There are ways in which Deep Learning can make a difference Dude is hard to beat (human is still superior when it comes to cell detection) Difference in gene expression profile causes cells to be different Effective cell detection can help develop better and cheaper drugs
  149. 149. Something important that we learnt today t = 0 t = 1 t = 2 Thresholding There are ways in which Deep Learning can make a difference Dude is hard to beat (human is still superior when it comes to cell detection) and last but not least… Difference in gene expression profile causes cells to be different Classical image analysis pipeline includes manual parameter tuning Effective cell detection can help develop better and cheaper drugs

×