Talk given by Oliver Norton and Tim Jenks at AWS for the Gaming Industry November 2012 Terrabyte geometry processing to build a 3D interactive model of the real world from GIS data. https://aws.amazon.com/campaigns/aws-for-gaming-london/

1. apps on maps...

2. Building the World with Elastic
Map Reduce
Oliver Norton, Technical Director
Tim Jenks, Technical Lead

8. appsonmaps.com
AS3 SDK – Apps in Browser
JS API – Embed on your website
IOS SDK – Mobile, coming Q4 2012

9. social media

10. social commerce

11. traffic updates

12. journey planning

13. worldflightclub.com
flying like a bird
World Flight Club on YouTube

14. flipping the bird!
Flipper on YouTube

15. photographic maps
Photographic-based maps ….

19. layered data
We fuse layered data to procedurally
generate our maps
(using AWS’ Elastic Map Reduce)

26. streamed
All built & served from off-the-shelf Amazon Web Service
infrastructure

27. pipeline

28. data size
Over 2TB Data
Terrain:
GB (10m, ¼ million Km2)
US (10m, 40x GB)
Buildings :
GB (full coverage)
US (120 cities)
Roads:
GB (¼ million miles)
US (4 million miles)
Processing this can start to be expensive $$$

30. before
• Limited scalability -> 60 desktop spec machines
• Multi-TB SAN with a £10k/year maintenance cost
• In house build that needed maintaining
• 10mbit/sec symmetric internet to upload TBs of data
• 3 developers knew how to run builds
• Electric costs -> who knows…

31. now
On Amazon Elastic Map Reduce

32. now
• Scalability -> 800 m1.large instances
• Off shelf tech that’s discoverable (hadoop, MRJob)
• Maintenance reduced
• Data is already in cloud (source, and destination)
• More predictable costs (and happier costs, with spot
pricing)
• DevOps benefits: Now any engineer can write and run
jobs, not just 3

33. pipeline

34. mrjob
• MRjob from Yelp
• http://github.com/Yelp/mrjob

35. 800 machines in 20lines
class MyMapReduceJob(MRJob):
def mapper_init(self):
self.__mapper = # wire up mapper
def mapper(self, key, line):
# perform map work
for key, value in self.__mapper.map(line, None):
yield str(key), value
def reducer_init(self):
self.__reducer = # wire up reducer
def reducer(self, key, values):
# perform reduce work
result = self.__reducer.reduce(key, values)
if result:
yield key, "built successfully"
else:
yield key, "failed"
if __name__ == '__main__':
MyMapReduceJob.run()

36. amazon emr
Processing Complexity X Data Size

37. thanks
appsonmaps.com
Twitter: @eeGeo