NoSQL like there is No Tomorrow
•Descargar como PPTX, PDF•
0 recomendaciones•1,253 vistas
The AWS NoSQL team shares the design philosophy behind DynamoDB and lessons learned in building for massive scale.
Recomendados
Recomendados
Más contenido relacionado
Destacado
Destacado (20)
Similar a NoSQL like there is No Tomorrow
Similar a NoSQL like there is No Tomorrow (20)
Más de Amazon Web Services
Más de Amazon Web Services (20)
Último
Último (20)
NoSQL like there is No Tomorrow
- 1. NoSQL like there is NoTomorrow Khawaja Engineering Lead for NoSQL, AWS @swami_79 @ksshams
- 2. NASA JPL has visited every planet in the @ksshams solar system ... except Pluto
- 3. 25 Gbps!
- 7. @ksshams
- 8. let’s start with a trilogy … @swami_79 @ksshams
- 9. episode 1 once upon a time... (in 2000) @swami_79 @ksshams
- 10. a half mile away... (Seattle) @swami_79 @ksshams
- 11. amazon.com - a rapidly growing Internet based retail business relied on relational databases @swami_79 @ksshams
- 12. we had 1000s of independent services @swami_79 @ksshams
- 13. each service managed its own state in Relational Databases @swami_79 @ksshams
- 14. Relational Databases are pretty seductive @swami_79 @ksshams
- 15. first of all... SQL!! @swami_79 @ksshams
- 16. so it is easier to query.. @swami_79 @ksshams
- 17. easier to learn @swami_79 @ksshams
- 18. They are as versatile as a swiss army knife key-value access complex queries analytics transactions @swami_79 @ksshams
- 19. Relational Databases are very similar to Swiss Army Knives @swami_79 @ksshams
- 20. sometimes.. swiss army knifes.. can be more than what you bargained for @swami_79 @ksshams
- 21. partitioning easy re-partitioning HARD.. @swami_79 @ksshams
- 22. so we bought bigger boxes... @swami_79 @ksshams
- 23. benchmark new hardware migrate to new hardware Q4 was hard-work at Amazon repartition databases pray ... @swami_79 @ksshams
- 24. Relational Databases have availability challenges.. @swami_79 @ksshams
- 25. episode 2 then.. (in 2005) @swami_79 @ksshams
- 26. amazon dynamo predecessor to dynamoDB replicated DHT with consistent hashing optimistic replication “sloppy quorum” anti-entropy mechanism object versioning specialist tool : •limited querying capabilities •simpler consistency @swami_79 @ksshams
- 27. dynamo had many benefits • higher availability • we traded it off for consistency • incremental scalability • no more repartitioning • no need to architect apps for peak • just add boxes • simpler querying model ==>> predictable performance @swami_79 @ksshams
- 28. but dynamo was not perfect... lacked strong consistency @swami_79 @ksshams
- 29. but dynamo was not perfect... scaling was easier, but... @swami_79 @ksshams
- 30. but dynamo was not perfect... steep learning curve @swami_79 @ksshams
- 31. but dynamo was not perfect... dynamo was a library ==>> not a service... @swami_79 @ksshams
- 32. episode 3 then.. (in 2012) @swami_79 @ksshams
- 33. Managed NoSQL Database ADMIN DynamoDB Built for Scale Fast & Predictable Performance @swami_79 @ksshams
- 34. DynamoDB “Even though we have years of experience with large, complex NoSQL architectures, we are happy to be finally out of the business of managing it ourselves.” - Don MacAskill, CEO @swami_79 @ksshams
- 35. DynamoDB Goals and Philosophies durability and availability scale is our problem easy to use scale in rps consistent and low latencies @swami_79 @ksshams
- 36. durability is key… @swami_79 @ksshams
- 37. availability is key… @swami_79 @ksshams
- 38. scale is our problem, not yours.. @swami_79 @ksshams
- 39. Fault Tolerant Design Infrastructure Fails - deal with it! Planning for failures is not easy How do you ensure your recovery strategies work correctly? @swami_79 @ksshams
- 40. Byzantine General Problem @swami_79 @ksshams
- 41. A simple 2-way replication system of a traditional database… Writes @swami_79 @ksshams
- 42. S is dead, need to trigger new replica P is dead, need to promote myself P S @swami_79 @ksshams
- 43. Improved Replication: Quorum Writes Quorum: Successful write on a majority @swami_79 @ksshams
- 44. Easy? Writes from client X Reads and Writes from client Y Classic Split Brain Issue in Replicated systems leading to lost writes! @swami_79 @ksshams
- 45. Building correct distributed systems is not straight forward.. Handle replica failures Handle partial failures of replicas Handle concurrent failures Ensure there isn’t a parallel quorum @swami_79 @ksshams
- 46. Trends in the World Of Databases @swami_79 @ksshams
- 47. A decade ago, it was all about the DBAs @swami_79 @ksshams
- 48. Last 5 years have been about self service. @swami_79 @ksshams
- 49. Today is about managed services. @swami_79 @ksshams
- 50. Plan for Success … Plan for Scale @swami_79 @ksshams
Notas del editor
- This story will be a little biased because Swami and I are from the NoSQL generation and we really don’t like relational databases
- http://centerfordentalimplants.info/wp-content/uploads/2012/10/young-woman-holding-cheeks-in-surprise.jpg
- more people already know it, etc
- http://www.toolstop.co.uk/components/com_virtuemart/shop_image/product/399926c941f80748cd41a85a57f8c163.jpg
- http://www.toolstop.co.uk/components/com_virtuemart/shop_image/product/399926c941f80748cd41a85a57f8c163.jpg Swami
- http://www.swissknifeshop.com/media/catalog/product/cache/1/image/5e06319eda06f020e43594a9c230972d/W/G/WG16999.jpg They seriously don’t scale well and if you’re not careful and try to use all the features at once, you can get cut in the process
- it was like playing minesweeper
- RDBMs focus way too much on consistency - and compromise availability And seriously - are they really all that consistent? ever had slaves fall behind? or how about a recent bug we had where all slaves got updates, but the master didn’t take the update Like Swiss army knives - there is waaaay too much functionality in the knives. Let’s move it to the app (easier to debug and change). Developers often use transactions when they shouldn’t - very annoying - it is like bringing a swiss army knife through a TSA checkpoint. Just foolish
- Khawaja
- Swami:
- K:
- required action from developers Q4 scaling: While not as scary as phase 1, still was work Need benchmarking, patching, adding hardware, rebalancing clusters..
- developers still had to learn to set it up if you are building a recommendation system, you want to be learning k-clustering and collaborative filtering algorithms rather than focusing on lamport’s papers
- Swami: like s3 - and it required developers to be admins. this is the really key point - our developers wanted a service, not a product. If you look at mongodb, that is what they are doing today.
- K
- </K>
- 2 dc writes on disk continuous replication we should brag about availability
- multi-region service! Regional service spans multiple availability zones All data is continuously replicated to multiple AZ’s
- plan for success, plan for scalability