Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Contour-Constrained Superpixels for Image and Video Processing

115 visualizaciones

Publicado el

발표자: 이세호(고려대 박사과정)

발표일: 2017.8.

슈퍼픽셀 알고리즘은 입력 영상을 다수의 의미 있는 영역으로 과분할 하는 기법이다. 입력 영상을 픽셀 단위로 표현할 때와 비교하여, 슈퍼픽셀 단위의 표현은 입력 영상의 단위의 수를 크게 줄이는 장점이 있다. 각 슈퍼픽셀은 객체의 윤곽선을 넘어서는 영역을 포함하지 않는 동시에, 단일 객체만을 담아야 한다. 본 발표에서는 객체의 윤곽선 정보를 고려한 윤곽선 제약 슈퍼픽셀 기법(contour-constrained superpixel algorithm)을 제안한다.

Publicado en: Tecnología
  • Sé el primero en comentar

Contour-Constrained Superpixels for Image and Video Processing

  1. 1. Contour-Constrained Superpixels for Image and Video Processing Se-Ho Lee, Won-Dong Jang, and Chang-Su Kim Korea University CVPR 2017
  2. 2. Overview • Introduction • Contour-constrained superpixels (CCS) • Superpixel labeling • Contour constraint • Hierarchical superpixel refinement • Temporal superpixels • Temporal superpixel labeling • Merging, splitting, and relabeling • Experimental results • Conclusions
  3. 3. • Superpixel • Over-segment input images into a set of meaningful regions • Can reduce the number of image primitives or units greatly • Superpixels should adhere to object contours Introduction
  4. 4. Superpixel • Applications • Preprocessing technique for many vision applications • ex) Saliency detection and image segmentation
  5. 5. Superpixel • Conventional methods • K-means based methods • Assign each pixel to the nearest cluster and update the cluster center iteratively • SLIC (TPAMI’12), LSC (CVPR’15), and MSLIC (CVPR’16) • Coarse-to-fine methods • Change the superpixel label of boundary regions iteratively based on the cost function • From block-level to pixel-level • SEEDS (IJCV’15) and Yao et al. (CVPR’15)
  6. 6. Contour-Constrained Superpixels • Contributions • Introduction of the contour constraint • Extension of the proposed algorithm for video processing • Remarkable performance achievement and improvement of many computer vision algorithms
  7. 7. Contour-Constrained Superpixels • Coarse-to-fine refinement • From block-level to pixel-level • Hierarchical block structure
  8. 8. Contour-Constrained Superpixels • Superpixel labeling • Hierarchical block structure • From block to pixel levels • Divide only inhomogeneous regions • Can maintain relatively regular and compact shape
  9. 9. Contour-Constrained Superpixels • Superpixel labeling • Change the label of region (block or pixel) 𝑅𝑖 from 𝑙 𝑅𝑖 to 𝑙 𝑅𝑗 of neighboring region 𝑅𝑗 ∈ 𝒩𝑅 𝑖 • Change the label of region by minimizing the cost function • 𝐸 𝑖, 𝑗 = 𝐸D 𝑖, 𝑗 + 𝛾𝐸L 𝑖, 𝑗 + 𝜂𝐸I 𝑖, 𝑗 × 𝐸C 𝑖, 𝑗 • To preserve the topology of superpixels • Check the boundary region 𝑅𝑖 is simple point Initial Level 1 Level 2 Level 3 Level 4
  10. 10. Superpixel Labeling • Feature distance from superpixel centroid • 𝐸D 𝑖, 𝑗 = 𝐜 𝑅𝑖 − 𝐜 𝑆𝑙 𝑅 𝑗 2 + 𝐩 𝑅𝑖 − 𝐩 𝑆𝑙 𝑅 𝑗 2 • Color distance term makes the color of each superpixel to be regular • Spatial distance term imposes the superpixels to be distributed compactly Color distance Spatial distance
  11. 11. Superpixel Labeling • Boundary length cost • 𝐸L 𝑖, 𝑗 = 𝜆 𝑅𝑖, 𝑙 𝑅𝑗 − 𝜆 𝑅𝑖, 𝑙 𝑅𝑖 • 𝜆 𝑅𝑖, 𝑘 : total number of boundary regions when the superpixel label of 𝑅𝑖 is 𝑘 • Minimize the boundary length of superpixels • Make the superpixels distributed compactly
  12. 12. Superpixel Labeling • Inter-region color cost • 𝐸I 𝑖, 𝑗 = max 0, 𝐜 𝑅𝑖 − 𝐜 𝑅𝑗 2 − 𝜅 𝑆𝑙 𝑅 𝑗 • Neighboring regions with dissimilar color information → Assign different superpixel labels • Internal difference • Measure the amount of texture information of superpixels • 𝜅 𝑆𝑙 𝑅 𝑖 = max 𝑅 𝑚,𝑅 𝑛∈𝑆𝑙 𝑅 𝑖 , 𝑅 𝑛∈𝒩 𝑅 𝑚 𝐜 𝑅 𝑚 − 𝐜 𝑅 𝑛 2 • Maximum color difference between neighboring regions within 𝑆𝑙 𝑅 𝑖
  13. 13. Superpixel Labeling • Contour constraint 𝐸C 𝑖, 𝑗 • Amplify the cost function when there is an object contour between two regions • Use holistically-nested edge detection (HED)* • Deep learning based edge detection method 𝐸 𝑖, 𝑗 = 𝐸D 𝑖, 𝑗 + 𝛾𝐸L 𝑖, 𝑗 + 𝜂𝐸I 𝑖, 𝑗 × 𝐸C 𝑖, 𝑗 * S. Xie and Z. Tu, Holistically-nested edge detection. In ICCV, pages 1395-1403, 2015.
  14. 14. Contour Constraint • Contour pattern set extraction • Use 200 training images in the BSDS500 dataset • Denote patch centered at each contour pixel as a contour pattern • Select top 1,000 frequently occurring patterns
  15. 15. Contour Constraint • Contour pattern set • Only consider the patterns which divide the patch into two regions • 1,000 patterns cover 90.5% of the patches in the training dataset
  16. 16. Contour Constraint • Contour pattern matching • Get thin binary contour map • Perform non-maximum suppression and then do thresholding • Compute Hamming distances from the contour patterns and select the best matching pattern
  17. 17. Contour Constraint • Contour probability 𝜙 𝑢, 𝑣 • Difficult to estimate the existence of object contours only considering the thin contour map • Measure the proportion of patches whose matching patterns separate 𝑢 from 𝑣 • Consider patches containing both pixels 𝑢 and 𝑣
  18. 18. Contour Constraint • Determining contour constraint 𝐸C 𝑖, 𝑗 • Contour probability 𝜓 𝑅𝑖, 𝑅𝑗 between regions • Find the maximum contour probability between the pixels in 𝑅𝑖 and 𝑅𝑗 • 𝜓 𝑅𝑖, 𝑅𝑗 = max 𝑢∈𝑅 𝑖,𝑣∈𝑅 𝑗 𝜙 𝑢, 𝑣 • 𝐸C 𝑖, 𝑗 = exp 𝛽 × 𝜓 𝑅𝑖, 𝑅𝑗
  19. 19. Hierarchical Superpixel Refinement • Hierarchical block structure • Divide only inhomogeneous regions into four blocks • Dissimilarity function • 𝜃 𝑅𝑖 = max 𝑢,𝑣∈𝑅 𝑖,𝑣∈𝒩𝑢 𝐜 𝑢 − 𝐜 𝑣 2 + exp 𝛽 × max 𝑢,𝑣∈𝑅 𝑖,𝑣∈𝒩𝑢 𝜙 𝑢, 𝑣 • Divide the region 𝑅𝑖 if 𝜃 𝑅𝑖 > 𝜏div
  20. 20. Contour-Constrained Superpixels • Algorithm
  21. 21. Temporal Superpixels • Superpixels for videos • Get temporally consistent superpixels Input SLIC CCS
  22. 22. Temporal Superpixels • Initialization • To make temporally consistent superpixels • Estimate optical flows from 𝐼 𝑡−1 to 𝐼 𝑡 • Transfer the label of each superpixel by employing the average optical flow of the superpixel • Do not assign any labels to occluded or disoccluded pixels • Occluded pixel: a pixel mapped from multiple superpixels • Disoccluded pixel: a pixel mapped from no superpixel
  23. 23. Temporal Superpixels • Temporal superpixel labeling • Performed at the pixel level only • Based on the energy function • 𝐸 𝑖, 𝑗 = 𝐸D 𝑖, 𝑗 + 𝛾𝐸L 𝑖, 𝑗 + 𝜂𝐸I 𝑖, 𝑗 × 𝐸T 𝑖, 𝑗 • 𝐸T 𝑖, 𝑗 : temporal contour constraint
  24. 24. Temporal Superpixel Labeling • Temporal Contour constraint • Make superpixels temporally consistent • Make superpixels compatible with image contours • 𝐸T 𝑖, 𝑗, 𝑡 = 𝐸C 𝑖, 𝑗, 𝑡 × 𝜌 𝑖, 𝑗, 𝑡 • Relaxation factor 𝜌 𝑖, 𝑗, 𝑡 • 𝜌 𝑖, 𝑗, 𝑡 = ൞ 1 1+exp −𝜁×ℎ 𝑅𝑖 𝑡 , if 𝑙 𝑅𝑗 𝑡 ∈ ℒ 𝑖 𝑡 1, otherwise • ℒ 𝑖 𝑡 is the set of labels that are mapped to 𝑅𝑖 𝑡 from 𝐼 𝑡−1 • Relax 𝐸T 𝑖, 𝑗, 𝑡 when 𝑅𝑖 𝑡 does not contain object contour
  25. 25. Merging, Splitting, and Relabeling • Merging and splitting • Prevent irregular superpixel size • When Τ𝐴 𝑘 𝑡 ҧ𝐴 is larger than 𝜏spl → split • Consider the biggest eigenvector of the spatial distribution • When Τ𝐴 𝑘 𝑡 ҧ𝐴 is smaller than 𝜏mer → merge • Merge to the nearest superpixel
  26. 26. Merging, Splitting, and Relabeling • Relabeling • Avoid incorrect labeling • Because of occlusion or illumination variation • Define color consistency 𝐶 𝑘 • 𝐶 𝑘 = 𝐜1:𝑡−1 𝑆 𝑘 − 𝐜 𝑡 𝑆 𝑘 2 • If 𝐶 𝑘 is larger than 𝜏rel → relabel
  27. 27. Temporal Superpixels • Algorithm
  28. 28. Experimental Results • Superpixel results • On BSDS500 dataset • CCS-wo-CC denotes proposed algorithm without the contour constraint ASA ↑ BR ↑ UE ↓
  29. 29. Experimental Results • Temporal superpixel results • On SegTrack dataset SA2D ↑ BR2D ↑ UE2D ↓ SA3D ↑ BR3D ↑ UE3D ↓
  30. 30. Experimental Results • Visual comparison of superpixel methods Input SEEDS LSC Proposed
  31. 31. Experimental Results • Visual comparison of temporal superpixel methods TCS TSP Proposed
  32. 32. Experimental Results • Run-times comparison • Superpixel algorithms • Temporal superpixel algorithms Turbopixels SLIC ERS LSC SEEDS MSLIC CCS Times(s) 8.09 0.26 1.52 0.34 0.06 0.36 0.97 sGBH SLIC TCS TSP CCS Times(s) 5.71 0.08 7.83 2.39 1.70
  33. 33. Experimental Results • Application to video object segmentation • To superpixel-based video object segmentation method* • Use CCS instead of SLIC as a preprocessing • Intersection over union (IoU) is increased from 0.532 to 0.571 * W.-D. Jang and C.-S. Kim, Semi-supervised video object segmentation using multiple random walkers. In BMVC, 2016.
  34. 34. Experimental Results • Application to video saliency detection • Postprocessing • Average the saliency values of the pixels in all frames, constituting each superpixel • Applied to HS* and DHSNet** * Q. Yan, L. Xu, J. Shi, and J. Jia, Hierarchical saliency detection. In CVPR, 2013. ** N. Liu and J. Han, DHSNet: Deep hierarchical saliency network for salient object detection. In CVPR, 2016.
  35. 35. Experimental Results
  36. 36. Conclusions • Contour-constrained superpixels (CCS) • Perform hierarchical refinement from block to pixel levels • Based on the contour constraint • Temporal superpixels • CCS algorithm for video processing • Use optical flow to obtain temporally consistent superpixels • Experimental results • CCS outperforms the state-of-the-art superpixel methods • Can be applied to object segmentation and saliency detection
  37. 37. Thank You Q & A