(SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
•
24 recomendaciones•9,028 vistas
Amazon DynamoDB is a fully managed, highly scalable distributed database service. In this technical talk, we show you how to use DynamoDB to build high-scale applications like social gaming, chat, and voting. We show you how to use building blocks such as secondary indexes, conditional writes, consistent reads, and batch operations to build the higher-level functionality such as multi-item atomic writes and join queries. We also discuss best practices such as index projections, item sharding, and parallel scan for maximum scalability.
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
Similar a (SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014 (7)
Más de Amazon Web Services
Más de Amazon Web Services (20)
Último
Último (20)
(SDD407) Amazon DynamoDB: Data Modeling and Scaling Best Practices | AWS re:Invent 2014
- 1. November 13, 2014 | Las Vegas, NV David Yanacek, Amazon DynamoDB
- 7. Product A Product B Shopper ProductCatalogTable
- 8. Product A Product B Shopper SELECT Id, Description, ... FROM ProductCatalog WHERE Id=“Product A” ProductCatalogTable
- 9. Need to scale for the sale Administrator (You) ProductCatalogTable
- 10. Administrator (You) Provision 5000 read capacity units ProductCatalogTable
- 11. Administrator (You) 2500 read capacity units (each) Provision 5000 read capacity units Partition 1 Partition 2 ProductCatalogTable
- 12. Administrator (You) (no sharing) Partition 1 Partition 2 ProductCatalogTable Provision 5000 read capacity units
- 13. Partition 1 Partition 2 ProductCatalogTable
- 14. Administrator (You) ProductCatalogTable Provision 100,000 read capacity units Partition 1 2000 RCU Partition K 2000 RCU Partition M 2000 RCU Partition 50 2000 RCU
- 15. Product A Product B Shoppers Partition 1 2000 RCU Partition K 2000 RCU Partition M 2000 RCU Partition 50 2000 RCU ProductCatalogTable
- 16. Partition 1 2000 RCU Partition K 2000 RCU Partition M 2000 RCU Partition 50 2000 RCU Product A Product B Shoppers ProductCatalogTable
- 17. Partition 1 2000 RCU Partition K 2000 RCU Partition M 2000 RCU Partition 50 2000 RCU Product A Product B Shoppers ProductCatalogTable
- 18. Partition 1 2000 RCU Partition K 2000 RCU Partition M 2000 RCU Partition 50 2000 RCU Product A Product B Shoppers ProductCatalogTable
- 20. Size Throughput
- 21. #표푓푃푎푟푡푖푡푖표푛푠= 푇푎푏푙푒푆푖푧푒푖푛푏푦푡푒푠 10퐺퐵 (푓표푟푠푖푧푒)
- 22. #표푓푃푎푟푡푖푡푖표푛푠 (푓표푟푡ℎ푟표푢푔ℎ푝푢푡) = 푅퐶푈푓표푟푟푒푎푑푠 3000푅퐶푈 + 푊퐶푈푓표푟푤푟푖푡푒푠 1000푊퐶푈
- 24. 100,000푅퐶푈 50푃푎푟푡푖푡푖표푛푠 ≈ퟐퟎퟎퟎ푟푒푎푑푐푎푝푎푐푖푡푦푢푛푖푡푠푝푒푟푝푎푟푡푖푡푖표푛 Partition 1 2000 RCU Partition K 2000 RCU Partition M 2000 RCU Partition 50 2000 RCU ProductCatalogTable
- 25. Space Time
- 28. 0 400 800 1200 1600 Capacity Units Time Provisioned Consumed “save up” unused capacity consume saved up capacity
- 29. 0 400 800 1200 1600 Capacity Units Time Provisioned Consumed Attempted Burst capacity: 300 seconds (1200 ×300 = 3600 CU) Throttled requests
- 31. Partition 1 2000 RCU Partition K 2000 RCU Partition M 2000 RCU Partition 50 2000 RCU Product A Product B Shoppers ProductCatalogTable
- 33. Requests Per Second Item Primary Key Request Distribution Per Hash Key DynamoDB Requests
- 34. Requests Per Second Item Primary Key Request Distribution Per Hash Key DynamoDB Requests Cache Hits
- 35. Partition 1 Partition 2 ProductCatalogTable User DynamoDB User SELECT Id, Description, ... FROM ProductCatalog WHERE Id="POPULAR_PRODUCT"
- 37. Substitute indexes for filters
- 39. GameId Date Host Opponent Status d9bl3 2014-10-02 David Alice DONE 72f49 2014-09-30 Alice Bob PENDING o2pnb 2014-10-08 Bob Carol IN_PROGRESS b932s 2014-10-03 Carol Bob PENDING ef9ca 2014-10-03 David Bob IN_PROGRESS Games Table
- 40. GameId Date Host Opponent Status d9bl3 2014-10-02 David Alice DONE 72f49 2014-09-30 Alice Bob PENDING o2pnb 2014-10-08 Bob Carol IN_PROGRESS b932s 2014-10-03 Carol Bob PENDING ef9ca 2014-10-03 David Bob IN_PROGRESS Games Table
- 43. Secondary Index Bob Opponent Status GameId Date Host Alice DONE d9bl3 2014-10-02 David Carol IN_PROGRESS o2pnb 2014-10-08 Bob Bob IN_PROGRESS ef9ca 2014-10-03 David Bob PENDING 72f49 2014-10-03 Alice Bob PENDING b932s 2014-09-30 Carol
- 44. Secondary Index Opponent Status GameId Date Host Alice DONE d9bl3 2014-10-02 David Carol IN_PROGRESS o2pnb 2014-10-08 Bob Bob IN_PROGRESS ef9ca 2014-10-03 David Bob PENDING 72f49 2014-10-03 Alice Bob PENDING b932s 2014-09-30 Carol Bob
- 45. Secondary Index Opponent Status GameId Date Host Alice DONE d9bl3 2014-10-02 David Carol IN_PROGRESS o2pnb 2014-10-08 Bob Bob IN_PROGRESS ef9ca 2014-10-03 David Bob PENDING 72f49 2014-10-03 Alice Bob PENDING b932s 2014-09-30 Carol Bob SELECT * FROM Game WHERE Opponent='Bob' AND Status='PENDING'
- 46. Bob SELECT * FROM Game WHERE Opponent='Bob' AND Status='PENDING' ORDER BY Date? Opponent Status GameId Date Host Alice DONE d9bl3 2014-10-02 David Carol IN_PROGRESS o2pnb 2014-10-08 Bob Bob IN_PROGRESS ef9ca 2014-10-03 David Bob PENDING 72f49 2014-10-03 Alice Bob PENDING b932s 2014-09-30 Carol Secondary Index
- 47. SELECT * FROM Game WHERE Opponent='Bob' AND Status='IN_PROGRESS' ORDER BY Date? (hash) (range) (?)
- 48. Secondary Index Opponent Date GameId Status Host Alice 2014-10-02 d9bl3 DONE David Carol 2014-10-08 o2pnb IN_PROGRESS Bob Bob 2014-09-30 72f49 PENDING Alice Bob 2014-10-03 b932s PENDING Carol Bob 2014-10-03 ef9ca IN_PROGRESS David Bob
- 49. Secondary Index Bob Opponent Date GameId Status Host Alice 2014-10-02 d9bl3 DONE David Carol 2014-10-08 o2pnb IN_PROGRESS Bob Bob 2014-09-30 72f49 PENDING Alice Bob 2014-10-03 b932s PENDING Carol Bob 2014-10-03 ef9ca IN_PROGRESS David SELECT * FROM Game WHERE Opponent='Bob' ORDER BY Date DESC FILTER ON Status='PENDING' (filtered out)
- 51. Bob
- 52. Bob
- 54. StatusDate DONE_2014-10-02 IN_PROGRESS_2014-10-08 IN_PROGRESS_2014-10-03 PENDING_2014-09-30 PENDING_2014-10-03 Status DONE IN_PROGRESS IN_PROGRESS PENDING PENDING Date 2014-10-02 2014-10-08 2014-10-03 2014-10-03 2014-09-30
- 55. Secondary Index Opponent StatusDate GameId Host Alice DONE_2014-10-02 d9bl3 David Carol IN_PROGRESS_2014-10-08 o2pnb Bob Bob IN_PROGRESS_2014-10-03 ef9ca David Bob PENDING_2014-09-30 72f49 Alice Bob PENDING_2014-10-03 b932s Carol
- 56. Opponent StatusDate GameId Host Alice DONE_2014-10-02 d9bl3 David Carol IN_PROGRESS_2014-10-08 o2pnb Bob Bob IN_PROGRESS_2014-10-03 ef9ca David Bob PENDING_2014-09-30 72f49 Alice Bob PENDING_2014-10-03 b932s Carol Secondary Index Bob SELECT * FROM Game WHERE Opponent='Bob' AND StatusDate STARTS_WITH 'PENDING'
- 57. Status + Date
- 60. Messages Table Messages app
- 61. David Recipient Date Sender Message David 2014-10-02 Bob … … 48 more messages for David… David 2014-10-03 Alice … Alice 2014-09-28 Bob … Alice 2014-10-01 Carol … (Many more messages) MessagesTable
- 62. Recipient Date Sender Message David 2014-10-02 Bob … … 48 more messages for David… David 2014-10-03 Alice … Alice 2014-09-28 Bob … Alice 2014-10-01 Carol … (Many more messages) David MessagesTable SELECT * FROM Messages WHERE Recipient='David' LIMIT 50 ORDER BY Date DESC
- 63. Recipient Date Sender Message David 2014-10-02 Bob … … 48 more messages for David… David 2014-10-03 Alice … Alice 2014-09-28 Bob … Alice 2014-10-01 Carol … (Many more messages) David MessagesTable Cost of query? SELECT * FROM Messages WHERE Recipient='David' LIMIT 50 ORDER BY Date DESC
- 64. Recipient Date Sender Message David 2014-10-02 Bob … … 48 more messages for David… David 2014-10-03 Alice … Alice 2014-09-28 Bob … Alice 2014-10-01 Carol … (Many more messages) MessagesTable Average item size = 256 KB Large message bodies Attachments? David SELECT * FROM Messages WHERE Recipient='David' LIMIT 50 ORDER BY Date DESC
- 65. Recipient Date Sender Message David 2014-10-02 Bob … … 48 more messages for David… David 2014-10-03 Alice … Alice 2014-09-28 Bob … Alice 2014-10-01 Carol … (Many more messages) David MessagesTable 50 items ×256 KB each SELECT * FROM Messages WHERE Recipient='David' LIMIT 50 ORDER BY Date DESC
- 66. Items evaluated by query Average item size Conversion ratio Eventually consistent reads
- 67. Messages Table Inbox Table Messages app
- 68. Recipient Date Sender Subject MsgId David 2014-10-02 Bob Hi!… afed David 2014-10-03 Alice RE: The… 3kf8 Alice 2014-09-28 Bob FW:Ok… 9d2b Alice 2014-10-01 Carol Hi!... ct7r Inbox Table Messages Table MsgId Body 9d2b … 3kf8 … ct7r … afed … (Bulk message data) (Message metadata, indexed by recipient)
- 69. Recipient Date Sender Subject MsgId David 2014-10-02 Bob Hi!… afed David 2014-10-03 Alice RE: The… 3kf8 Alice 2014-09-28 Bob FW:Ok… 9d2b Alice 2014-10-01 Carol Hi!... ct7r Inbox Table Messages Table MsgId Body 9d2b … 3kf8 … ct7r … afed … David 1.Query Inbox: 1 RCU (50 sequential items at 128 bytes)
- 70. Recipient Date Sender Subject MsgId David 2014-10-02 Bob Hi!… afed David 2014-10-03 Alice RE: The… 3kf8 Alice 2014-09-28 Bob FW:Ok… 9d2b Alice 2014-10-01 Carol Hi!... ct7r Inbox Table Messages Table MsgId Body 9d2b … 3kf8 … ct7r … afed … David 1.Query Inbox: 1 RCU 2.BatchGetItem Messages: 1600 RCU (50 separate items at 256 KB) (50 sequential items at 128 bytes)
- 71. Messages Table Inbox Table David PutItem { MsgId: 123, Body: ... } PutItem { Recipient: Steve, Date: 2014-10-23, MsgId: 123, ... }
- 72. Messages Table Inbox Table David PutItem { MsgId: 123, Body: ... } PutItem { Recipient: Steve, Date: 2014-10-23, MsgId: 123, ... }
- 73. David PutItem { MsgId: 123, Body: ..., Recipient: Steve, Sender: David, Date: 2014-10-23, ... } Inbox Global secondary index Messages Table
- 75. Inbox Global secondary index Messages Table David Outbox Global secondary index Coming soon:
- 76. Inbox Messages
- 77. De-normalize schemas Design for retries
- 79. User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Friends Table HighScoresTable User Friend Bob Alice Bob Chuck Alice Bob Alice Chuck Alice Erin
- 80. User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Friends Table HighScoresTable User Friend Bob Alice Bob Chuck Alice Bob Alice Chuck Alice Erin Alice
- 81. User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Friends Table HighScoresTable User Friend Bob Alice Bob Chuck Alice Bob Alice Chuck Alice Erin 1. Query 2. BatchGetItem Alice
- 83. User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 Table FriendScoresTable HighScoresTable
- 84. User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 User H.S. Friend Bob 20 Chuck Bob 1000 Alice Alice 20 Chuck Alice 470 Bob Alice 580 Erin Table Local Secondary Index FriendScoresTable HighScoresTable
- 85. User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 User HighScore Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 User H.S. Friend Bob 20 Chuck Bob 1000 Alice Alice 20 Chuck Alice 470 Bob Alice 580 Erin Table Local Secondary Index FriendScoresTable HighScoresTable Query FROM FriendScores-Index WHERE User=Alice, DESC, LIMIT 10 Alice
- 86. Scores Friends
- 87. User HighScore Chuck 30 Alice 1000 Bob 470 Dave 850 Erin 580 User Friend H.S. Bob Alice 1000 Bob Chuck 30 Alice Bob 470 Alice Chuck 30 Alice Erin 580 FriendScoresTable HighScoresTable 1. UpdateItem 2. UpdateItem(s) Chuck
- 88. User HighScore Chuck 30 Alice 1000 Bob 470 Dave 850 Erin 580 User Friend H.S. Bob Alice 1000 Bob Chuck 30 Alice Bob 470 Alice Chuck 20 Alice Erin 580 FriendScoresTable HighScoresTable 1. UpdateItem 2. UpdateItem(s) Chuck
- 89. User H.S. Updating Chuck 20 Alice 1000 Bob 470 Dave 850 Erin 580 User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 FriendScoresTable HighScoresTable truewhile updating the high score
- 90. User H.S. Updating Chuck 30 true Alice 1000 Bob 470 Dave 850 Erin 580 User Friend H.S. Bob Alice 1000 Bob Chuck 20 Alice Bob 470 Alice Chuck 20 Alice Erin 580 FriendScoresTable HighScoresTable 1. UpdateItem Chuck
- 91. User H.S. Updating Chuck 30 true Alice 1000 Bob 470 Dave 850 Erin 580 User Friend H.S. Bob Alice 1000 Bob Chuck 30 Alice Bob 470 Alice Chuck 20 Alice Erin 580 FriendScoresTable HighScoresTable Chuck 2. UpdateItem(s)
- 92. User H.S. Updating Chuck 30 true Alice 1000 Bob 470 Dave 850 Erin 580 User Friend H.S. Bob Alice 1000 Bob Chuck 30 Alice Bob 470 Alice Chuck 20 Alice Erin 580 FriendScoresTable HighScoresTable Chuck 3. GetItem
- 93. User H.S. Updating Chuck 30 true Alice 1000 Bob 470 Dave 850 Erin 580 User Friend H.S. Bob Alice 1000 Bob Chuck 30 Alice Bob 470 Alice Chuck 30 Alice Erin 580 FriendScoresTable HighScoresTable Chuck 4. UpdateItem(s)
- 94. User H.S. Updating Chuck 30 Alice 1000 Bob 470 Dave 850 Erin 580 User Friend H.S. Bob Alice 1000 Bob Chuck 30 Alice Bob 470 Alice Chuck 30 Alice Erin 580 FriendScoresTable HighScoresTable 5. UpdateItem Chuck
- 96. PutItemRequestacceptBob; PutItemRequestacknowledgeChuck; Transaction t = txManager.newTransaction(); t.putItem(acceptBob); t.putItem(acknowledgeChuck); t.commit();
- 97. Friends Table Transactions Table Transaction Images Table Transaction Client Bob
- 100. Your app FriendScoresTable HighScoresTable
- 101. Your app FriendScoresTable HighScoresTable
- 102. Your Kinesis-enabled app Your app FriendScoresTable HighScoresDynamoDB Stream HighScoresTable
- 103. User H.S. Chuck 20 Alice 100 HighScoresTable Chuck Seq User Old New 1 Chuck 10 20 2 Alice 80 90 3 Alice 90 100 HighScoresStream
- 104. User H.S. Chuck 30 Alice 100 HighScoresTable Chuck Seq User Old New 1 Chuck 10 20 2 Alice 80 90 3 Alice 90 100 HighScoresStream
- 105. User H.S. Chuck 30 Alice 100 HighScoresTable Chuck Seq User Old New 1 Chuck 10 20 2 Alice 80 90 3 Alice 90 100 4 Chuck 20 30 HighScoresStream
- 106. User H.S. Chuck 30 Alice 100 HighScoresTable Chuck Seq User Old New 1 Chuck 10 20 2 Alice 80 90 3 Alice 90 100 4 Chuck 20 30 HighScoresStream
- 107. User H.S. Chuck 30 Alice 100 HighScoresTable User Friend H.S. Bob Chuck 20 Alice Chuck 20 FriendScoresTable Friend User Chuck Alice Chuck Bob Table Global Secondary Index Seq User Old New 1 Chuck 10 20 2 Alice 80 90 3 Alice 90 100 4 Chuck 20 30 HighScoresStream Your Kinesis-enabled app
- 108. User H.S. Chuck 30 Alice 100 HighScoresTable User Friend H.S. Bob Chuck 20 Alice Chuck 20 FriendScoresTable Friend User Chuck Alice Chuck Bob Table Global Secondary Index Seq User Old New 1 Chuck 10 20 2 Alice 80 90 3 Alice 90 100 4 Chuck 20 30 HighScoresStream Your Kinesis-enabled app
- 109. User H.S. Chuck 30 Alice 100 HighScoresTable User Friend H.S. Bob Chuck 30 Alice Chuck 30 FriendScoresTable Friend User Chuck Alice Chuck Bob Table Global Secondary Index Seq User Old New 1 Chuck 10 20 2 Alice 80 90 3 Alice 90 100 4 Chuck 20 30 HighScoresStream Your Kinesis-enabled app
- 110. User H.S. Chuck 30 Alice 100 HighScores Table User Friend H.S. Bob Chuck 30 Alice Chuck 30 FriendScores Table Friend User Chuck Alice Chuck Bob Table Global Secondary Index Seq User Old New 1 Chuck 10 20 2 Alice 80 90 3 Alice 90 100 4 Chuck 20 30 HighScoresStream Your Kinesis-enabled app Streams Checkpoint Table
- 112. Shard write-heavy items Design for retries
- 113. Candidate A Votes: 20 Candidate B Votes: 30 Voter Votes Table
- 114. Candidate A Votes: 21 Candidate B Votes: 30 Votes Table UPDATE Votes SET Votes = Votes + 1 WHERE Id=“Candidate A” Voter (aka atomic increment)
- 115. Need to scale for the election Administrator (You) Votes Table
- 116. Administrator (You) Provision 200,000 write capacity units Partition 1 1000 WCU Partition K 1000 WCU Partition M 1000 WCU Partition N 1000 WCU Votes Table
- 117. Candidate A Candidate B Voters Partition 1 1000 WCU Partition K 1000 WCU Partition M 1000 WCU Partition N 1000 WCU Votes Table
- 118. Partition 1 1000 WCU Partition K 1000 WCU Partition M 1000 WCU Partition N 1000 WCU Votes Table Candidate A Candidate B Voters
- 119. Candidate A Candidate A_2 Candidate A_1 Candidate A_3 Candidate A_4 Candidate A_7 Candidate A_6 Candidate A_8 Candidate A_5
- 120. Candidate A_2 Candidate B_1 Candidate B_2 Candidate B_3 Candidate B_5 Candidate B_4 Candidate B_7 Candidate B_6 Candidate A_1 Candidate A_3 Candidate A_4 Candidate A_7 Candidate B_8 Candidate A_6 Candidate A_8 Candidate A_5 Voter Votes Table
- 121. Candidate A_2 Candidate B_1 Candidate B_2 Candidate B_3 Candidate B_5 Candidate B_4 Candidate B_7 Candidate B_6 Candidate A_1 Candidate A_3 Candidate A_4 Candidate A_7 Candidate B_8 UpdateItem: “CandidateA_” + rand(0, 10) ADD 1 to Votes Candidate A_6 Candidate A_8 Candidate A_5 Voter Votes Table
- 122. Votes Table Candidate A_2 Candidate B_1 Candidate B_2 Candidate B_3 Candidate B_5 Candidate B_4 Candidate B_7 Candidate B_6 Candidate A_1 Candidate A_3 Candidate A_4 Candidate A_5 Candidate A_6 Candidate A_8 Candidate A_7 Candidate B_8 Periodic Process Candidate A Total: 2.5M 1. Sum 2. Store Voter
- 125. UserId Candidate Date Alice A 2013-10-02 Bob B 2013-10-02 Eve B 2013-10-02 Chuck A 2013-10-02 RawVotesTable Voter
- 126. UserId Candidate Date Alice A 2013-10-02 Bob B 2013-10-02 Eve B 2013-10-02 Chuck A 2013-10-02 RawVotesTable Segment Votes A_1 23 B_2 12 B_1 14 A_2 25 AggregateVotesTable Voter 1. Record vote and de-dupe, retry 2. Increment candidate counter
- 127. UserId Candidate Date Alice A 2013-10-02 Bob B 2013-10-02 Eve B 2013-10-02 Chuck A 2013-10-02 RawVotesTable Segment Votes A_1 23 B_2 12 B_1 14 A_2 25 AggregateVotesTable Voter
- 128. AggregateVotes Table Voters RawVotesTable Voting app
- 129. AggregateVotes Table Amazon Redshift Amazon EMR Your Kinesis-enabled app Voters RawVotesTable Voting app RawVotes DynamoDB Stream
- 130. AggregateVotes Table Amazon Redshift Amazon EMR Your Kinesis-enabled app Voters RawVotesTable Voting app RawVotes DynamoDB Stream
- 131. AggregateVotes Table Amazon Redshift Amazon EMR Your Kinesis-enabled app Voters RawVotesTable Voting app RawVotes DynamoDB Stream
- 132. AggregateVotes Table Amazon Redshift Amazon EMR Your Kinesis-enabled app Voters RawVotesTable Voting app RawVotes DynamoDB Stream
- 133. AggregateVotes Table Amazon Redshift Amazon EMR Your Kinesis-enabled app Voters RawVotesTable Voting app RawVotes DynamoDB Stream
- 138. Status + Date
- 139. Inbox Messages
- 140. Scores Friends