Open data on buildings with satellite imagery processing - Jubal Harpster, Bing Maps Team, Microsoft.

2. What are we going to cover?
• Project Description
• Motivation
• Solution Overview
• Extracting Buildings for OpenStreetMap
• What’s next
• Conclusions

3. Building Extraction Project Overview
Why is Microsoft investing in this?
• Consistent side by side Loss against competitors for Base
Map
• Lack of Building footprints is #1 contributor
• Metrics show users like building polygons on base maps
• Reverse geocoding scenarios
• Cortana geofencing: “What building am I in?”
• OpenStreetMap is the largest source of buildings data by
far
• ~315M and growing by 1000s per day
• “Free” source of data if we follow the guidelines

5. Open Source at Microsoft
 Microsoft is the largest Open
Source contributor to GitHub
 2,620 public repos
 10,966 contributors
 902 teams
 The Open Source shift at MS makes
Opening data on GitHub easy

6. Map Tiles
• Images from Bing Satellite Layer
• Image map tiles are extracted from the world’s
Mercator projection using quadkeys
• Images are of 256x256 pixels size, jpeg
compressed
• The maximal level of detail available is 23
• Going one level deeper =>
• 2x better resolution -> more details
• 4x more data to process and store
• Detail level 19 properties:
• Resolution is 0.3m/pixel (~1 foot/pixel)
• The whole US is covered
• ~2.7 billion tiles in mainland US
Level 18 Level 19 Level 20

7. Scale is Huge
• Extracting from ~2.7 billion L19 tiles in mainland US in 1
month, need to process 1000 images per second
• 60TB of imagery for the US
• Toolkits & Platforms that makes this possible:
• CNTK – Microsoft Open Source Cognitive Toolkit:
• A unified deep-learning toolkit
• Philly Cluster:
• Internal GPU compute cluster for to large-scale DNN training
• COSMOS (aka AzureDB):
• A Microsoft internal Big Data platform service
• Our platform of choice for storing massive datasets

8. Creating the Training Set
• Once the tiles are isolated, training pairs are created
• Label masks are created in Æther:
• For each training tile find the overlapping label polygons by sending a query request to spatial DB
• Render building polygon on the label tile images – converts lat/long coordinates into the pixel
domain
• Creating uncertainty/weight areas around building enables better training
• Training loss function can be modified to account for errors around building edges
since the labels aren’t perfect
Text
Training pair
Uncertainty area

9. Postprocessing – Creating Building Polygons
• The process of converting building pixel predictions into the building polygons
defined by latitude/longitude pair vertices
• Processing stages included:
• Finetuning and de-noising raw DNN building predictions
• Conversion into representative polygons that approximate the building shapes
• Subtracting known source satellite image offsets in the resulting polygons
• Executed Azure Æther because of easy integration with Python image libraries
Raw predictions
Finetuning
Noise Removal Creating polygons
Storing into a textual stream
WKT formatParcels utilization

10. The Final Pipeline – 64 GPUs
Buildings Extraction in the United
States
• Support for autoscaling resources
• Parametrization of different jobs
• Auto restart with no data loss
Imagery data : United States = 60
TB;
Final Buildings Processing time:
United States = 2 weeks

11. Errors – Pacific county, WA
• Beach and waves identified as
buildings
• We didn’t have enough negative
examples in the training set
• Proper buildings were not
detected
• The imagery is of lower or
different resolution
• There were not enough positive
images in the training set

12. Seattle Downtown
• Boats & piers recognized as buildings
• We didn’t have enough negative examples
in the training set in coastal areas
• Parts of highway recognized as buildings
• Need to add road intersection
postprocessing
• Shadows can cause problems
• In high density areas buildings can put
shadows on other buildings
• Need more high-density area training
samples

13. Other issues rendering false positives
• We need to spend more time
working on the training set
• Snow spots and clouds can
generate false positive predictions

14. What else is possible?
We are only beginning
Possible data sets include:
• National Level Road Networks
• Buildings for additional countries
• Water & coastline extractions
• High accuracy lanes & road markings
• Off Nadir (oblique) segmentation

15. Roads – Extracted in days,
not months or weeks

16. Paved Attribute coverage
Some countries have a significant number of
unpaved roads
 In Australia OSM has a sparse attribute coverage:
 320K km of road missing surface=
 790K km of road correctly tagged
 Applied computer vision to detect the type:
 Classifier that identifies if an image contains paved road
 Classifier Precision/Recall = 92%/90% (per tile)
 Voting logic brings this to 98%/99% measured in km
Unpaved
Paved
PavedPaved
Unpaved
Unpaved
UnpavedUnpavedUnpavedUnpaved
Unpaved Paved

17. Building Extraction - Canada
• Canada spans on 12.5 billion L19 tiles
(US spans across 2.7 billion tiles)
• Optimization:
• L12 tiles intersecting:
• Expanded populated places and
neighborhoods by 10km
• Expanded roads by 2km
• This method could be validated on US
• Create the evaluation sets
• Iterate improvements targeting the error
space
Camera Id Canada US old US new
222/225 0.04% 1.30% 1.36%
217 0.00% 76.34% 0.79%
138 80.42% 10.87% 88.34%
235 3.89% 0.15% 0.22%
No image 15.64% 10.95% 9.25%
Model trained on
High focus
imagery
Low focus
imagery
Tiles camera distribution

18. Water & Coastlines
• L16 or L17 tiles extraction
• 64x or 256x smaller data space compared
to buildings extraction
• We can use the same deep net model –
or simplify
• No polygonization necessary

19. Oblique Imagery
• Taken from the flying plane/drone
• 4 directions
• Coverage is limited to top US cities
• Potential for 3D reconstruction
• 45 different shots of the same location
• Could help semantic understanding
• Building heights

20. Oblique Road Horizontal Signalization
• Image resolution 10300x7700
• 6cm/px close areas , 8.5 cm/px far
areas
• Compared to 32cm/px – building
extraction
• Turn restriction and lane
detection
• Apply Deep Neural Nets
• Lane Markings
• Car, bike, bus, pedestrian, parking, etc
Bottom road Upper road

21. Open up all the building data – for free
 We released all the extracted
buildings on GitHub
 Amplifies the value of the data by
making it Open
 Adheres to OSM community
guidelines
 Encourages additional contributions
 To date over 75K downloads of the
building data

22. Applications & Summary
• Population Estimations
• Neighborhood delineations
• Telecommunications – Mobile Signal Targeting
• Urban Planning
• Transportation & Land use Planning
The value of the data are amplified when Open
Everywhere we have training data we can apply deep
learning