(PFC402) Bigger, Faster: Performance Tips for High Speed and High Volume Applications | AWS re:Invent 2014
•
4 recomendaciones•1,305 vistas
This expert level session covers best practices and tips on how to reduce latency to the absolute minimum when dealing with high volume, high speed datasets, using Amazon DynamoDB. We take a deep dive into the design patterns and access patterns geared to provide low latency at very high throughput. We cover some ways in which customers have achieved low latencies and have a customer speak about their experience of using DynamoDB at scale.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (11)
Destacado
Destacado (20)
Similar a (PFC402) Bigger, Faster: Performance Tips for High Speed and High Volume Applications | AWS re:Invent 2014
Similar a (PFC402) Bigger, Faster: Performance Tips for High Speed and High Volume Applications | AWS re:Invent 2014 (20)
Más de Amazon Web Services
Más de Amazon Web Services (20)
Último
Último (20)
(PFC402) Bigger, Faster: Performance Tips for High Speed and High Volume Applications | AWS re:Invent 2014
- 1. November 13, 2014 | Las Vegas, NV Ben Clay, Amazon DynamoDB Brett McCleary, Precision Exams
- 5. •Independent throughput or storage scaling •Supports both document and key-valuedata models
- 9. Example Schema: WebstoreOrders Hash Key (string) CustomerID Range Key (string) Timestamp Attribute (map) Item ID: quantity map Attribute (number) Customer ID
- 10. Item #1 #2 … Partition #1 Partition #2 Partition #3 Item #N •One item, one partition •Placement based on key
- 11. Item #1 #2 … Partition #1 Partition #2 Partition #3 Provisioned Throughput 1000 WPS 1000 WPS 1000 WPS
- 12. Table Partition 1 Partition 3 Partition 2 … Client Client Client Partition 4 Partition 6 Partition 5 Client Client Client
- 14. Client Machine Application OS SDK DynamoDB
- 15. Client Machine Application OS SDK DynamoDB
- 16. Client Machine Application OS SDK DynamoDB
- 17. 0 250 500 750 1000 1250 1500 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Throughput Time Provisioned Consumed
- 18. Table Partition 1 Partition 3 Partition 2 Client Client Client Partition 4 Client Table Partition 1 Client Client Table grows over time, skew becomes noticeable
- 22. WebstoreOrders Hash Customer ID Range Timestamp Attrib Items ordered Attrib Order ID
- 23. Orders Partition 1 Partition 3 Partition 2 … Client Client Client Partition 4 Partition 6 Partition 5 Client Client Client
- 25. WebstoreOrders Hash Customer ID Range Timestamp Attrib Items ordered Attrib Order ID Attrib Household ID Household Index Hash HouseholdID Range Timestamp Attrib Items ordered Attrib Order ID Attrib Customer ID Indexing
- 26. Table Item #1 Item #3 Item #2 … Index Item #1 Item #3 Item #2 … Read Clients Read + Write Clients
- 27. Dayof Order Index Hash Dayof order Range Order ID Attrib Items ordered Attrib Customer ID Attrib Timestamp WebstoreOrders Hash Customer ID Range Timestamp Attrib Items ordered Attrib Order ID Attrib Day of order Indexing
- 28. Day of order index Table Partition 1 Partition 3 Partition 2 … Index Partition 1 Partition 3 Partition 2 … Client Client Client New orders
- 29. Day of order index Table Partition 1 Partition 3 Partition 2 Index Partition 1 Partition 3 Partition 2 Client Client Client New orders … …
- 31. Alternate Approach: Scanning H: Alice R: Oct 2 Partition #1 Partition #2 H: Alice R: Nov 11 H: Alice R: Dec 25 H: Bob R: Oct 20 H: Bob R: Nov 12 H: Bob R: Dec 23 P1 P2 P3 P4 P5 P6 P7 P8 P9 Scan
- 32. •Delete old items from the client side H: Alice R: Oct 2 Partition #1 Partition #2 H: Alice R: Nov 11 H: Alice R: Dec 25 H: Bob R: Oct 20 H: Bob R: Nov 12 H: Bob R: Dec 23 •Takeaway: Table growth can impact throughput per key •Important when:Accumulating infrequently-read data
- 34. •Controlling table growth with deletes works but… •Deleting items from client = 2x write cost! •Can we achieve cheaper deletes AND scans? H: Alice R: Oct 2 Partition #1 Partition #2 H: Alice R: Nov 11 H: Alice R: Dec 25 H: Bob R: Oct 20 H: Bob R: Nov 12 H: Bob R: Dec 23 Scan for last month
- 35. H: Alice R: Dec 23 Dec Table Hash: Bob R: Dec 25 H: Alice R: Oct 2 Oct Table Hash: Bob R: Oct 20 •Takeaway: Time series data chunks very well •Important when: Big, growingtime series tables H: Alice R: Nov 11 Nov Table Hash: Bob R: Nov 12 Scan last month
- 37. Orders Table Item #1 Item #2 Item #3
- 39. Orders Table Item #1 Item #2 … EMR Fleet 2. Export 1. Scan Item #3
- 40. Orders Table Item #1 Item #2 Stream Processor Fleet 1.Update recordsYou need near-realtimedata 2. Export
- 44. “It is not the strongest of the species that survives, nor the most intelligent, but the one most adaptable to change.” --Charles Darwin
- 46. web02 web02 web n Web Tier app01 app02 app n Application Tier
- 48. rpt01 web02 web02 web n Web Tier app01 app02 app n Application Tier Data warehouse process sat01 sat02 sat n
- 53. rpt01 web02 web02 web n Web Tier app01 app02 app n Application Tier dw01 Amazon DynamoDB
- 57. TestPacket Answer Record Hash Key (string) Test Packet ID Attribute (string) Answer JSON
- 58. TestPacket Response Record Hash Key (string) Test Packet ID Range Key (string) Test PacketResponse ID Attribute (string) Create Timestamp Attribute (string) PostDate Attribute (string) Response JSON
- 61. { "testPacketId":11193654, "answerJson": { "SQ22545":{"responses":{"010":"Y"},"awardedPts":1}, "SQ22546":{"responses":{"040":"Y"},"awardedPts":1}, "21137":{"responses":{"030":"Y"},"awardedPts":0} ... } }
- 63. { "testPacketId":11193654, "testPacketResponseId":"SQ22545", "createdTimeStamp":"1412609315419", "postDate":"1412609315419", "responseJson":{"i":"26492","f":"N","t":0,"r":{"010":"Y"}} }
- 66. tst01 tst02 Amazon DynamoDB
- 69. Client Client Client Grinder Client Client Client Amazon DynamoDB
- 74. 0 2 4 6 8 10 12 300 600 1200 2200 5000 Latency (ms) Provisioned Capacity (units) P90 Client Latency (ms) Avg. Server Latency (ms) Avg. Client Latency (ms)