1. Distributed
bulk rendering
on the Amazon cloud
Dr. Raphael Volz
nogago GmbH, Gilching, Germany
State of the map, Girona, July 9, 2010
2. Another 6 years into the project, the
planet will have 5 TB (compressed).
Hypothesis
July 2010
10 GB
July 2016
5 TB
Distributed bulk rendering on the Amazon cloud
3. OSM is power law distributed with
top 10% covering only 1 ‰ of planet
Observation
Top 100 6°x6° areas
Size (Bytes GZ-compressed)
# 1 / 1 GB
West 6 South 48
# 2 / 0,6 GB
West -84 South 30
6x6 splits by size
Distributed bulk rendering on the Amazon cloud
4. Lower barriers of entry and increase the
number of eyes !
How to achieve the 5 TB planet
Distributed bulk rendering on the Amazon cloud
5. Ride the smartphone wave !
Smartphones in use (in Mio.)
Source: Nogago analysis of Juniper Research, Gartner Spring 2009 forecasts
Note: Assumption: Devices sold are in use for two years
Distributed bulk rendering on the Amazon cloud
6. For example nogago outdoor is a smartphone app
that accompanies your outdoor activities
Plan Share
activities experiences
• View maps and
• Download guides anytime • View tracks
Maps and offline • View photos
• Choose from • Record and (geo-referenced)
400.000 tracks follow Tracks • Share with
• Plan new family, friends
tracks Enjoy the
or the world !
Outdoors !
Distributed bulk rendering on the Amazon cloud
7. Our customers will download offline maps
for regions of choice
Raster map
POI DB
Ways DB
Max.
200 MB
per 1°x1°
Distributed bulk rendering on the Amazon cloud
8. Generating offline maps takes
significant time and space
Using the OSM map rendering tool chain…
Get
Load into Render
XML planet PNG
Database Maps
osm.bz2
wget osm2pgsql mapnik
… takes space and time …
Zoom Level 15 1 Billion 5 621
(1:14000 @eq.) Tiles TB Days
Zoom Level 18 68 Billion 350 109
(1:1700 @eq.) Tiles TB Years
Assumptions: 16 KB per Tile on average, 20 Tiles per Second to render on modern hardware
Distributed bulk rendering on the Amazon cloud
9. Basic approach to parallelization using
Amazon cloud
Render
Image Nogago.com
Ubuntu 10.04 Start Msg. Msg. Shut off
Apache Instance started finished instance
Postgres 8.4 User Amazon EC2
osm2pgsql Data
osmosis
mapnik Render Image
TimKay‘s aws Get
planet
Load into Render Store
Database Maps
mapnik-OSM osm.bz2
on S3
Nogago tools
Amazon EC2
runs images
Rarely
Changing
Amazon S3
Content
stores files
Total: 3 GB Amazon S3
Note: UD = User Data, thanks to Eric Hammond (http://ec2-run-user-data.notlong.com)
Distributed bulk rendering on the Amazon cloud
10. To make things fit consider corner cases
Example:
1 x1
to be rendered
at zoom level 10
1,352 x 1,352
to be loaded
into database
190% overhead
at zoom level 10
Data to load into Postgres Area to be rendered
(depends on zoom levels) by mapnik
Distributed bulk rendering on the Amazon cloud
11. Optimization: Do not always get planet
Prepare chunks periodically
• Two step approach, first:
– Download planet, periodically 1 Day
to process
– Split into chunks and copy
on S3
– Store chunks on EC2
• Then, many rendering images
– Get appropriate chunk from S3 Max.
– Load (parts of) chunk into Postgres 3 hours
per
– Render from Postgres 1 x1
Distributed bulk rendering on the Amazon cloud
12. What have we learned ?
Rendering the world on
zoom level 15
in 28 hours instead of 621 days
( 2 ‰ of the time)
By distributing bulk rendering
on the Amazon cloud
Distributed bulk rendering on the Amazon cloud
13. EU FP7 Thank
project partner you !
sought for
mapping-related
project proposal
Distributed bulk rendering on the Amazon
cloud
