Field Notes from Expeditions in the Cloud-(Matt Wood, Amazon Web Services)
•
2 recomendaciones•1,720 vistas
Presentation at Spark Summit 2015
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Field Notes from Expeditions in the Cloud-(Matt Wood, Amazon Web Services)
Similar a Field Notes from Expeditions in the Cloud-(Matt Wood, Amazon Web Services) (20)
Más de Spark Summit
Más de Spark Summit (20)
Último
Último (20)
Field Notes from Expeditions in the Cloud-(Matt Wood, Amazon Web Services)
- 1. FIELD NOTES DR. MATT WOOD GM, PRODUCT S TRATEGY FROM EXPEDITIONS IN THE CLOUD
- 3. THANK YOU
- 6. “WHAT IS BIG DATA?”
- 9. CONSTRAINTS
- 10. BOXED IN
- 11. CONSTRAINTS
- 12. PRODUCTIVITY Quick provisioning Resource fit Designed for iteration
- 13. Canonical data stores Streaming data Task clusters
- 14. SPARK WITH AMAZON EMR Production workloads on AWS Security event streaming Machine learning & ad targeting Web analytics Revenue forecasting Ad targeting & recommendations PersonalizationPredictive marketing App search
- 15. S3 MLlib on Spark Clicks, videos, interactives
- 16. S3 MLlib on Spark Clicks, videos, interactives S3 Customer segments
- 17. S3 MLlib on Spark Clicks, videos, interactives S3 Customer segments Forecasting Ad sales Map user to ads
- 18. S3 MLlib on Spark Clicks, videos, interactives S3 Customer segments Forecasting Ad sales Map user to ads Recommendation Spark cluster Clicks, videos, interactives
- 19. S3 MLlib on Spark Clicks, videos, interactives S3 Customer segments Forecasting Ad sales Map user to ads Recommendation Spark cluster Clicks, videos, interactives Kafka Every click
- 21. Node.js app Kinesis Click stream Elastic Beanstalk
- 22. Node.js app Kinesis Click stream Elastic Beanstalk Spark Streaming EMR cluster 5 minute windows 100MB S3 JSON and CSV
- 23. Node.js app Kinesis Click stream Elastic Beanstalk Spark Streaming EMR cluster 5 minute windows 100MB S3 JSON and CSV EMRElasticSearch Redshift
- 24. Node.js app Kinesis Click stream Elastic Beanstalk Spark Streaming EMR cluster 5 minute windows 100MB S3 JSON and CSV EMRElasticSearch Redshift
- 25. S3 Ad impressions & clicks 24/7 Spark cluster Impression RDD
- 26. S3 Ad impressions & clicks 24/7 Spark cluster Impression RDD Interactive dashboard Revenue forecast
- 27. S3 Ad impressions & clicks 24/7 Spark cluster Impression RDD Interactive dashboard Revenue forecast Click stream logs Batch Spark clusters Redshift
- 28. S3 Ad impressions & clicks 24/7 Spark cluster Impression RDD Interactive dashboard Revenue forecast Click stream logs Batch Spark clusters Redshift Data exploration and testing
- 29. PRODUCTIVITY Quick provisioning Designed for iteration Resource fit
- 30. SPARK ON EMR New Provision and scale managed Spark clusters, as first class citizens
- 31. RAPID PROVISIONING OF ELASTIC CLUSTERS Provision new clusters in minutes High memory, high CPU, high IO instances Access cluster instances directly Clusters run within a VPC Add or remove capacity on running clusters
- 32. DIRECT ACCESS TO DATA ON S3 Access objects directly on Amazon S3 Server-side and client-side encryption with customer controlled keys Multiple clusters can access canonical data in S3 No need to copy or manage the data on the cluster Mix S3 and HDFS on a cluster
- 33. INTEGRATION WITH THE SPOT MARKET Bid on under utilized capacity on EC2 “Name your price” clusters Very low cost at high scale Lowest cost for time insensitive workloads Also on-demand and reserve capacity pricing
- 34. SPARK ON EMR New No additional cost. Available today. aws.amazon.com/emr/spark