Hadoop Image Processing for Disaster Relief Andrew Levine TexelTek

1. OPEN CLOUD CONSORTIUM IMAGE
PROCESSING FOR DISASTER RELIEF
Image Cache and Image Delta

2. GOALS FOR PROCESSING MAP IMAGERY
• Make imagery available for Disaster Relief
workers over the web
• Provide a mechanism for large scale image
processing Satellite/Map Imagery
• Provide image deltas for temporally different
and geospaJally idenJcal image sets

3. • Source imagery can be very large
– New image formats can be ~2G
– Compare image sets easily
• New data daily
– NASA E01 mission tasking
for fires and floods
• Pass over areas about every 3rd day
• High availability for results
HaiJ image:
18,878px by 34,782px
TOOLS AND THE PROCESSING PLATFORM MOTIVATION FOR CLOUD IMPLEMENTATION

4. USE CASES FOR THIS FRAMEWORK
• Disasters
– Fires
– Floods
– Earthquakes
– DeforestaJon
– Drought
– War/Refugees
– Tornados
• Other Processing
– Medical Imagery
– Anomaly
DetecJon
– Full MoJon Video
– Tracking
– Digital Cinema

5. TOOLS AND THE PROCESSING PLATFORM
• OCCTestbed pla^orm
– Resources for processing large data
– Testbed of mulJple clouds
– UIC cloud is 32 nodes
• Quad Core, 16GB RAM, GigE, HDFS on 256GB
• Apache Hadoop: MapReduce and HBase
– Algorithm adheres to MapReduce framework
– hcp://hadoop.apache.org/
• OCC Image Processing tools (open source)
– hcp://code.google.com/p/matsu‐project/
– Image comparison

6. INTERFACING WITH RESULTS
• Open GeospaJal ConsorJum Web Map Service
– Images available through OGCWMS open specificaJon
– hcp://www.opengeospaJal.org/
• OCC WMS Servlet (open source)
– hcp://code.google.com/p/matsu‐project/
• Various Map Viewing Tools
– OpenLayers, Google Maps, others

7. ARAL SEA 1989 AND 2008

8. ZOOM LEVELS / BOUNDS
Zoom Level 1: 4 images Zoom Level 2: 16 images
Zoom Level 3: 64 images Zoom Level 4: 256 images

9. Mapper Input Key: Bounding Box
Mapper Input Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper resizes and/or cuts up the original
image into pieces to output Bounding Boxes
(minx = ‐135.0 miny = 45.0 maxx = ‐112.5 maxy = 67.5)
Step 1: Input to Mapper
Step 2: Processing in Mapper Step 3: Mapper Output
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Build Tile Cache in the Cloud ‐ Mapper

10. Reducer Key Input: Bounding Box
(minx = ‐45.0 miny = ‐2.8125 maxx = ‐43.59375 maxy = ‐2.109375)
Reducer Value Input:
Step 1: Input to Reducer
…
Step 2: Reducer Output
Assemble Images based on bounding box
• Output to HBase
• Builds up Layers
for WMS for
various datasets
Build Tile Cache in the Cloud ‐ Reducer

11. Mapper Input Key: Bounding Box
Mapper Input Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper resizes and/or cuts up the original
image into pieces to output Bounding Boxes
(minx = ‐135.0 miny = 45.0 maxx = ‐112.5 maxy = 67.5)
Step 1: Input to Mapper
Step 2: Processing in Mapper Step 3: Mapper Output
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
Mapper Output Key: Bounding Box
Mapper Output Value:
+ Timestamp
+ Timestamp
+ Timestamp
+ Timestamp
+ Timestamp
+ Timestamp
+ Timestamp
+ Timestamp
+ Timestamp
Image Processing in the Cloud ‐ Mapper

12. Reducer Key Input: Bounding Box
(minx = ‐45.0 miny = ‐2.8125 maxx = ‐43.59375 maxy = ‐2.109375)
Reducer Value Input:
Step 1: Input to Reducer
… …
Step 2: Process difference in Reducer
Assemble Images based on Jmestamps and compared Result is a delta of the two Images
Step 3: Reducer Output
All images go to different map layers set of images for display in WMS
Timestamp 1
Set
Timestamp 2
Set
Delta Set
Image Processing in the Cloud ‐ Reducer

13. GULF OIL SPILL
Day 115 Day 128 Delta

14. SAMPLES / FLOODS IN PAKISTAN
2010 day 197 2010 day 263
Delta

15. SAMPLES / FLOODS IN PAKISTAN
2010 day 197 2010 day 263 Delta

16. HBASE TABLES
• OGC WMS Query translates to HBase scheme
– Layers, Styles, ProjecJon, Size
• Table name: WMS Layer
– Row ID: Bounding Box of image
‐Column Family: Style Name and ProjecJon
‐Column Qualifier: Width x Height
‐Value: Buffered Image