The Weather of the Century
•
7 recomendaciones•2,006 vistas
The weather is everywhere and always. That makes for a lot of data. This talk will walk you through how you can use MongoDB to store and analyze worldwide weather data from the entire 20th century in a graphical application. We’ll discuss loading and indexing terabytes of data in a sharded cluster, and optimizing the schema design for interactive exploration. MongoDB also natively supports geospatial indexing and querying, and it integrates easily with open source visualization tools. You'll earn high-performance techniques for querying and retrieving geospatial data, and how to create a rich visual representation of global weather data using Python, Monary, and Matplotlib.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a The Weather of the Century
Similar a The Weather of the Century (20)
Más de MongoDB
Más de MongoDB (20)
Último
Último (20)
The Weather of the Century
- 1. “The Weather of the Century”:! Data Visualization With MongoDB And Python A. Jesse Jiryu Davis Senior Engineer, MongoDB @jessejiryudavis
- 2. Serious MongoDB Talk Database
- 4. This Talk
- 5. Where’s the data from?
- 6. Where’s the data from?
- 7. How Much Is There? • 2.5 billion documents • 4 TB (1.6k per document) • “Medium data”
- 8. What Does It Look Like? 0303725053947282013060322517+40779-073969FM-15+0048KNYC V0309999C00005030485MN0080475N5+02115+02005100975 ADDAA101000095AU100001015AW1105GA1025+016765999GA2045+024385999 GA3075+030485999GD11991+0167659GD22991+0243859GD33991+0304859... { "st" : "u725053", "ts" : ISODate("2013-06-03T22:51:00Z"), "airTemperature" : { "value" : 21.1, "quality" : "5" }, "atmosphericPressure" : { "value" : 1009.7, "quality" : "5" } } Station Identifier (»NYC Central Park«)
- 9. {! ts: ISODate("1991-01-01T00:00:00Z"),! position: {! type: "Point",! coordinates: [! -94.6,! 39.117! ]! },! airTemperature: {! value: 27,! quality: "1"! }! }! GeoJSON
- 10. Visualization
- 11. Visualization Pipeline MongoDB PyMongo Python NumPy Matplotlib dicts SciPy
- 12. {! ts: ISODate("1991-01-01T00:00:00Z"),! position: {! type: "Point",! coordinates: [! -94.6,! 39.117! ]! },! airTemperature: {! value: 45,! quality: "1"! }! }!
- 13. import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)!
- 15. # NumPy column access syntax.! lons = arrays[:, 0]! lats = arrays[:, 1]! temps = arrays[:, 2]!
- 16. from scipy import griddata! from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! Magic!! ! pyplot.contour(xs, ys, zs)! Also magic!!
- 18. from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)!
- 19. Triangulation
- 20. Triangulation
- 22. Barycentric Interpolation What temperature? 53 48 54 Weighted Average 51.1
- 24. Interpolation
- 25. Interpolation
- 26. Contours
- 27. Contours
- 35. Not terrifically fast import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)!
- 36. MongoDB-to-NumPy Performance • Querying: 109k documents per second • (On localhost) • Can we go faster? • Enter “Monary”
- 37. Monary by David Beach MongoDB PyMongo Python NumPy Matplotlib dicts MongoDB Monary NumPy Matplotlib
- 38. import monary! ! data = []! connection = monary.Monary()! ! arrays = monary_connection.query(! db='my_database',! coll='collection',! query=query,! fields=[! 'position.coordinates.0',! 'position.coordinates.1',! 'airTemperature.value'],! types=[! 'float32',! 'float32',! 'float32'])!
- 39. Monary • PyMongo: 109k documents per second • Monary: 817k documents per second
- 40. Visualization
- 41. • Author: David Beach • Contributors from MongoDB, Inc.: Kyle Suarez Matt Cotter Anna Herlihy • Mentors: A. Jesse Jiryu Davis Jason Carey Monary
- 42. Recent features: • Easy installation • Nested field access • Aggregation • Python 3 Monary
- 43. • Insert, update, remove • SSL and authentication mechanisms • Improved API and logging • parallelCollectionScan Monary Future:
- 44. ! • MongoDB • Python • Monary • NumPy • SciPy • Matplotlib
- 45. Thanks
- 46. Thank you A. Jesse Jiryu Davis Senior Python Engineer, MongoDB #MongoDBWorld
- 47. Presents 1. http://bit.ly/century-links 2. October MongoDB certification exams! price *= 0.8 Code “MongoDBBoston20” university.mongodb.com 3. Ask The Experts!!