3D Scene Change Detection with 3D and 2D Information
•
1 recomendación•1,456 vistas
This document discusses change detection in 3D scenes using 3D and 2D information. It presents four chapters: 1) 3D keypoints based 3D-2D matching to estimate camera poses, 2) 3D-2D based change detection in images, 3) 3D-3D based change detection between point clouds, and 4) conclusions. The key goals are to detect potential changed areas efficiently using local 3D features and camera poses from 3D-2D matching, and to estimate accurate changes by registering two point clouds and calculating the scale ratio between them in 3D-3D change detection. Evaluation shows the 3D-2D method can detect changes fast while 3D-3D provides higher accuracy.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (19)
Similar a 3D Scene Change Detection with 3D and 2D Information
Similar a 3D Scene Change Detection with 3D and 2D Information (20)
Último
Último (20)
3D Scene Change Detection with 3D and 2D Information
- 1. 1 Change Detection of 3D Scene with 3D and 2D Information for Environment Checking PhD Candidate: Baowei Lin August 12th, 2013
- 2. 1. Introduction Research Motivation Change Detection 2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation 3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation 4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation 5. Conclusions 2
- 3. 1. Introduction Research Motivation Change Detection 2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation 3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation 4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation 5. Conclusions 3
- 4. 4 Changes should be alerted at these areas.
- 6. 6 • Impossible to check manually Wide range Huge number of blocks • Important to check automatically
- 7. 7 • Impractical to check by fixed cameras
- 8. 8 • possible to check by hand-held devices
- 10. 10 Estimating accurate changes. Sub-goal 2: offline Finding potential change area. online Sub-goal 1:
- 11. 1. Introduction Research Motivation Change Detection 2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation 3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation 4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation 5. Conclusions 11
- 12. A change is a difference of objects in the scene at time A and at time B. 12 Time A Time B
- 13. 13 3D point cloud Training images (2D images)
- 14. 1. 2D-2D Method 14 Need Fixed camera input output Original image Change image Changed area 2D-2D
- 15. input output 2. 3D-2D Method 15 Detection is fast but not accurate Original point cloud Change image Changed area 3D-2D
- 16. input output 3. 3D-3D Method 16 Detection is accurate but slow Original point cloud Change point cloud Changed area 3D-3D
- 17. 17 Potential changed areas 3D information Camera poses Online system Offline system 3D-2D based change detection 3D-2D based camera pose estimation 3D-3D based change detection Chapter 2 Chapter 3 Chapter 4 3D-3D3D-2D
- 18. 1. Introduction Research Motivation Change Detection 2. 3D Keypoints Based 3D-2D Matching Background 3D Keypoints Detection Evaluation 3. 3D-2D Based Change Detection Background Image Based Change Detection Evaluation 4. 3D-3D Based Change Detection Background Scale Estimation of a Single Point Cloud Scale Ratio Estimation of Two Point Clouds Evaluation 5. Conclusions 18
- 19. 19 3D interesting points 2D interesting points Camera pose=[R,t] 3D point cloud 2D training images
- 20. 20 2D-2D 3D-3D SIFT[Lowe 2004], SURF [Bay 2006], etc. spin image[Johnson 1998], NARF[Steder 2010], etc. detector descriptor detector descriptor
- 21. 21 3D point cloud 2D training images 3D detector and 2D detector can not be corresponded.
- 22. 22 Image patch Point distribution Can not match 2D image 3D point cloud
- 23. 23 • Detect keypoints correctly • Describe keypoints appropriately 2D image 3D point cloud
- 24. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 24
- 25. Detected 2D interesting points 25 Feature matching Obviously, the SIFT features could be used in 3D keypoints detection and description.
- 26. Point cloud 26 P1 P2 P3 P4 P5 P6Camera position 3D keypoint Projected 3D points 2D images number threshold used for 3D keypoints decision. the points which can appear on multiple training images Back face points are not used for computation 3D keypoints th_v
- 27. 27 th_v = 1 #3D keypoints ≅10,000 27 training images 105,779 3D points th_v = 7 #3D keypoints ≅ 1,000 Reconstructed 3D points #3D points ≅30,000 Too many for real time calculating Smaller number and good distribution oursoriginal
- 28. 28 2D SIFT keypoints and descriptors 3D keypoint& descriptor -Keep all 2D descriptors Accurate but slow Description methods: -Average and Median SIFT features are different when view directions are different.
- 29. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 29
- 30. 30 P1 P2 P3 P4 P5 P6 3D point cloud Ground truth Camera positions …… Training images
- 31. 31 P1 P2 P3 P4 P5 P6 P6 ’ Camera pose estimation 3D keypoints generation
- 32. 32 P1 P2 P3 P4 P5 P6 P6 ’ P6 ’ =[R ’ |t ’] P6 =[R | t ]
- 33. TranslationerrorRotationerror[rad] 33 1. Our method is accurate 2. th_v does not affect the result th_v is used for 3D keypoints selection 2 degrees Dataset: 27 training images Image resolution:2256x1504 3D points number:105,779 3D scene size:40x25x5cm Bounding box size:10.6x5.7x1.4 0.24cm
- 34. 34 3D point cloud Query image project 3D points
- 35. 35
- 36. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 36
- 37. 37 Potential changed areas 3D information Camera poses Online system Offline system 3D-2D based change detection 3D-2D based camera pose estimation 3D-3D based change detection
- 38. 38 Our method: 1. Use local feature instead of color 2. Detect any shape of object Using laser range finder [Goncalves 2010, Ryle 2011 and Neuman 2011]. Not for wide area targets. Not applicable for our round shape or natural scenes. Matching 3D line segments [Eden 2008]. Using color differences [Sato 2006, Pollard 2007 and Taneja 2011] Not stable for illumination changes.
- 39. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 39
- 40. 40 1. Find the nearest image Query image Nearest image 2. Find changed area Nearest image Query image changed area 3. Visualization Project 3D points onto changed area
- 41. 1st Nearest Query image 41 P1 P2 P3 P4 P5 3D keypoints generation Need fixed camera Smallest distance Ground truth …… Training images 2nd 3rd P
- 42. 42 the 1st nearest image Query image Points: 2D keypoints Blue: correspondence Red: no correspondence Blue: correspondence Red: no correspondence Non-change area Uncovered area is the changed area Estimated changed area
- 43. 3D point cloud Visualized 3D points 43 change area projection Detected result Projected 3D points
- 44. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 44
- 45. 3D point cloud Query image 45 Quantitative results visualization Changed 3D points Changed area
- 46. Results for different thresholds 46 Set as:0, 5, 10, 20, 30, 50, 70 and 90 pixels 0 5 10 20 30 50 70 90 Image resolution:2256x1504 The number of Image: 54 The number of 3D points: 190,845
- 47. TP rate= True Positive Ground Positive FP rate= Ground Negative False Positive 47 Receiver operating characteristic (ROC) plot threshold = 30 threshold = 30 Ground truth is set manually We expect the 1st nearest image perform better than others, but the best result is the 2nd nearest image. Good performance Bad performance
- 48. 48It is the parameter left for users. 1st 2nd Query image Detection results
- 49. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 49
- 50. 50 Potential changed areas 3D information Camera poses Online system Offline system 3D-2D based change detection 3D-2D based camera pose estimation 3D-3D based change detection
- 51. Different size scale because of the character of Structure- from-Motion (SfM) 51 3D-3D registration is actually, the scale registration 3D point cloud 3D point cloud 3D point cloud Change points registration Point clouds of same scene with different size 3D point cloud 3D point cloud
- 52. 52 Iterative closest point (ICP) based alignment [Besl 1991]. -Need simple scenes -Need initial pose and scale -Not robust to clutters, occlusions and missing part spin images [Johnson 1998], NARF [Steder 2010], shape context [Belongie 2002], etc. Feature based alignment -Need appropriate neighborhood size 3D SIFT [Scovanner 2007], 3D SURF [Knopp 2010], etc. -Not robust to clutters, occlusions and missing part Easy data Different data Fixed scale Adaptive scale
- 53. 53 1. Scale estimation 2. Scale Ratio estimation Keyscale1=0.5 Keyscale2=0.1 Scale ratio=5
- 54. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 54
- 55. Bunny point cloud Width=0.001 55 Similar to each other Different to each other Width=0.1 Width=1.0 3D keypoints Spin images the minimum of similarity between spin images when the width changes. Similar to each other Keyscale Robust to clutters, occlusions and missing part Calculate similarity of collected spin images
- 56. 56 Decide which set of spin images are different to each other by using Contribution rate. PCA Robust to order of extracted spin images. Robust to detail
- 57. 57minimum 1 5 10 15 similaritysimilarity d w Similar to each other Different Similar to each other
- 58. minimum is not unique Finding them is not stable 58 minimumminimum similarity w
- 59. 59 Bunny point cloud Finding minimum is not stable similarity w
- 60. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 60
- 61. Point clouds 61 Register two plots to get scale ratio Scale ratio ICP similarity plots
- 62. Overlapping parts Original bunny curves 5 times larger bunny curves 62
- 63. Original bunny curves 5 times larger bunny curves 63
- 64. Scale ratio t Displaced Original bunny curves 5 times larger bunny curves 64
- 66. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 66
- 67. 67 Original point cloud The number of points:207,583 Scene size: 35m x 5m
- 68. overlapping area 68 Created 1st point cloud Created 2nd point cloud estimate scale ratio Original point cloud
- 69. 69 The method provides perfect result when the overlap rate is larger than 70%. Ground truth = 1
- 70. Small blocks point clouds 70 Changed block
- 71. Introduction ◦ Research Motivation ◦ Change Detection 3D Keypoints Based 3D-2D Matching ◦ Background ◦ 3D Keypoints Detection ◦ Evaluation 3D-2D Based Change Detection ◦ Background ◦ Image Based Change Detection ◦ Evaluation 3D-3D Based Change Detection ◦ Background ◦ Scale Estimation of a Single Point Cloud ◦ Scale Ratio Estimation of Two Point Clouds ◦ Evaluation Conclusions 71
- 72. 72 Potential changed areas 3D information Camera poses Online system Offline system 3D-2D based change detection 3D-2D based camera pose estimation 3D-3D based change detection
- 73. 73 Future work: 1. Improve computation speed and detection accuracy for online system. -current computation time: 20 seconds per image 2. Optimize algorithm to operate with huge size data for offline system. -current computation time: 10 minutes for 100,000 points