(BDT203) From Zero to NoSQL Hero: Amazon DynamoDB Tutorial | AWS re:Invent 2014
•
21 recomendaciones•7,008 vistas
Got data? Interested in learning about NoSQL? In this session, we take you from not knowing anything about Amazon DynamoDB to being able to build an advanced application on top of DynamoDB. We start with an overview of the service, basic fundamental concepts, and then dive right in to a hands-on follow along tutorial in which you: create your own table, make queries, add secondary indexes to existing tables, query against the secondary indexes, modify your indexes, as well as detect changes to your data in DynamoDB to build all kinds of analytics and complex event processing apps. You can walk in a novice with DynamoDB, but rest assured, you will walk out as a NoSQL expert ready to tackle large distributed systems problems with your database problems addressed with DynamoDB.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (8)
Similar a (BDT203) From Zero to NoSQL Hero: Amazon DynamoDB Tutorial | AWS re:Invent 2014
Similar a (BDT203) From Zero to NoSQL Hero: Amazon DynamoDB Tutorial | AWS re:Invent 2014 (20)
Más de Amazon Web Services
Más de Amazon Web Services (20)
Último
Último (20)
(BDT203) From Zero to NoSQL Hero: Amazon DynamoDB Tutorial | AWS re:Invent 2014
- 1. November 12, 2014 | Las Vegas, NV David Yanacek, Amazon DynamoDB Jason Lambert, Here by Nokia
- 4. App/Web Tier Client TierRDBMS
- 5. App/Web Tier Client TierRDBMS
- 6. App/Web Tier Client TierData Tier Search Cache Blob Store RDBMS NoSQL Data Warehouse
- 7. Data Tier Search Cache Blob Store RDBMS NoSQL Data Warehouse logging analytics key/value simple query rich search hot reads complex queries and transactions
- 8. Data Tier Amazon DynamoDB Amazon RDS Amazon ElastiCache Amazon S3 Amazon Redshift Amazon CloudSearch logging analytics key/value simple query rich search hot reads complex queries and transactions
- 10. “DynamoDB has scaled effortlessly to match our company's explosive growth, doesn't burden our operations staff, and integrates beautifully with our other AWS assets”. “I love how DynamoDB enables us to provision our desired throughput, and achieve low latency and seamless scale, even with our constantly growing workloads.”
- 12. RDBMS Poor Availability Limited Scalability High Cost
- 13. lack of strong operational consistency every engineer needs to learn distributed systems complexity
- 14. Non-Relational Fast & Predictable Performance Seamless Scalability Easy Administration
- 15. database service automated operationspredictable performance fast development always durable low latencycost effective =
- 16. table
- 19. WRITES Continuously replicated to 3 zones Quorum acknowledgment Persisted to disk (custom SSD) READS Strongly or eventually consistent No trade-off in latency
- 20. “Our previous NoSQL database required almost a full time administrator to run. Now AWS takes care of it.” “Being optimized at AdRollmeans we spend more every month on snacks than we do on DynamoDB–and almost nothing on an ops team” Save MoneyReduce Effort
- 21. Weatherbugmobile app Lightning detection & alerting for 40M users/month Developed and tested in weeks, at “1/20th of the cost of the traditional DB approach” Super Bowl promotion Millions of interactions over a relatively short period of time Built the app in 3 days, from design to production-ready
- 22. database service automated operationspredictable performance fast development always durable low latencycost effective =
- 24. https://www.example.com/maps/Pike+Place+Market,+Seattle,+ WA/47.6206924,-122.3492979/@47.6151852,- 122.3516502,16z/data=!3m1!4b1!4m8!4m7!1m5!1m1!1s0x54906ab2c151d6cb:0x60441d5f633e9f1c!2m2!1d- 122.3420567!2d47.6101359!1m0
- 25. https://__________________________Place+Market,+Seattle,+ WA/47.6206924,-122.3492979/@47.6151852,- 122.3516502,16z/data=!3m1!4b1!4m8!4m7!1m5!1m1!1s0x54906ab2c151d6cb:0x60441d5f633e9f1c!2m2!1d- 122.3420567!2d47.6101359!1m0 https://short.example.com/c3b87de1
- 26. client Amazon DynamoDB Web Service Elastic Load Balancing URL Shortener App Target Website example.com website
- 27. client example.com website Web Service Elastic Load Balancing GET /u/d83 HTTP 1.1 301 Moved Permanently Location: http://www.example.com/maps/... Amazon DynamoDB
- 28. client example.com website Web Service Elastic Load Balancing GET /maps/Pike+Place... Amazon DynamoDB
- 29. ShortUrl RedirectTo CreatedBy CreateDate example.com/u/d83 <long url> David 2014-09-09 example.com/u/abc <long url> Alice 2013-08-17 example.com/u/123 <long url> 2013-12-21 example.com/u/ea2 <long url> Bob 2014-03-30 Url Table Table
- 30. ShortUrl RedirectTo CreatedBy CreateDate example.com/u/d83 <long url> David 2014-09-09 example.com/u/abc <long url> Alice 2013-08-17 example.com/u/123 <long url> 2013-12-21 example.com/u/ea2 <long url> Bob 2014-03-30 Url Table Item
- 31. ShortUrl RedirectTo CreatedBy CreateDate example.com/u/d83 <long url> David 2014-09-09 example.com/u/abc <long url> Alice 2013-08-17 example.com/u/123 <long url> 2013-12-21 example.com/u/ea2 <long url> Bob 2014-03-30 Url Table Attributes (schema-less)
- 32. ShortUrl RedirectTo CreatedBy CreateDate example.com/u/d83 <long url> David 2014-09-09 example.com/u/abc <long url> Alice 2013-08-17 example.com/u/123 <long url> 2013-12-21 example.com/u/ea2 <long url> Bob 2014-03-30 Url Table Primary Key (Hash Index)
- 34. String Set String Number Map Not pictured: •Binary •List •Null •Number Set •Binary Set Boolean
- 35. 64 KB 400 KB
- 37. dynamodb.putItem({ TableName:'Url', Item:{ ShortUrl:"short.example.com/4ua93", RedirectTo:"long.example.com/maps/Pike+Place...", IsLive:true, Tags:["map","directions"], Metadata:{ CreatedBy:"David", CreateDate:"2014-19-21T08:49", TargetViews:100 }, Stats:{ LastVisited:"2014-10-30T10:31", PageViews:326 } } },function(err,data){ if(err)print(err);// an error occurred elseprint(data);// successful response }); Request Response (empty)
- 38. dynamodb.getItem({ TableName:'Url', Key:{ "ShortUrl":"short.example.com/4ua93" }, ConsistentRead:true },function(err,data){ if(err)print(err); elseprint(data); }); Optional read-your-writes consistency
- 39. dynamodb.scan({ TableName:'Url', ProjectionExpression:'ShortUrl, ' +'RedirectTo, Stats.LastVisited', Limit:2 }).eachPage(function(err,data){ if(err)print(err); elseif(data)print(data); }); SQL-like expressions Paginated (max 1 MB per page)
- 40. dynamodb.scan({ TableName:'Url', FilterExpression: 'begins_with(RedirectTo, :domain) '+ 'AND contains(Tags, :tag)'+ 'AND ('+ ' Stats.PageViews> Metadata.TargetViews'+ ' OR '+ ' Stats.StarCount> Metadata.TargetStars'+ ')', ExpressionAttributeValues:{ ':domain':'long.example.com/', ':tag':'directions' } }).eachPage(function(err,data){ if(err)print(err); elseif(data)print(data); }); Nested conditions Compare existing attributes Filter Expressions and Improved Conditions!
- 45. dynamodb.updateItem({ TableName:'Url', Key:{ "ShortUrl":"short.example.com/4ua93" }, UpdateExpression: 'SET Usage.Views= Usage.Views+ :increment_by, ' +'Usage.LastVisited= :now', ExpressionAttributeValues:{ ':increment_by':1, ':now':newDate().toISOString() }, ReturnValues:'ALL_NEW', ConditionExpression:'attribute_exists(ShortUrl)' },function(err,data){ if(err)print(err); elseprint(data); }); Increment views by 1
- 46. dynamodb.updateItem({ TableName:'Url', Key:{ "ShortUrl":"short.example.com/4ua93" }, UpdateExpression: 'SET Usage.Views= Usage.Views+ :increment_by, ' +'Usage.LastVisited= :now', ExpressionAttributeValues:{ ':increment_by':1, ':now':newDate().toISOString() }, ReturnValues:'ALL_NEW', ConditionExpression:'attribute_exists(ShortUrl)' },function(err,data){ if(err)print(err); elseprint(data); }); Writes are also free reads!
- 51. Images Table
- 52. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Hash and range primary key schema (composite key)
- 53. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Range: Index is sorted by range key, can do range queries Hash: Equality queries only
- 54. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… Bob Query for Bob’s Images Query FROM Images WHERE User=Bob
- 57. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… User Date Image Bob 2013-09-05 cf2e2 Bob 2013-10-01 aed4c Bob 2013-10-08 f93bae Alice 2013-09-12 ca61a Table ByDate Local Secondary Index Local secondary index on Date
- 58. Images Table User Image Date Link Bob aed4c 2013-10-01 s3://… Bob cf2e2 2013-09-05 s3://… Bob f93bae 2013-10-08 s3://… Alice ca61a 2013-09-12 s3://… User Date Image Bob 2013-09-05 cf2e2 Bob 2013-10-01 aed4c Bob 2013-10-08 f93bae Alice 2013-09-12 ca61a Table ByDate Local Secondary Index Query for Bob’s two most recent images Query FROM Image-Index WHERE User=Bob, DESC, LIMIT 2
- 60. Images Table ImageTags Table
- 61. ImageTags Table Image User aed4c Alice aed4c Bob f93bae Bob
- 62. ImageTags Table Image User aed4c Alice aed4c Bob f93bae Bob Hash and range primary key schema
- 63. Image User aed4c Alice aed4c Bob f93bae Bob ImageTags Table Bob Query FROM ImageTags WHERE Image=aed4c
- 64. ImageTags Table Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Bob Tag Alice in Image f93bae
- 67. and/or range keyfor your table •Even moreflexible query patterns Distributed transaction managed for you by DynamoDB
- 68. ImageTags Table Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table
- 69. ImageTags Table Global secondary index on User, Image User Image Bob aed4c Bob f93bae Alice aed4c Alice f93bae ByUser Global Secondary Index Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Alternate hash and range keys
- 70. ImageTags Table Alice Query for images tagged Alice User Image Bob aed4c Bob f93bae Alice aed4c Alice f93bae ByUser Global Secondary Index Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Query FROM ImageTags-Index WHERE User=Alice
- 74. { Id: abecd, Players : [ Alice, Bob ], State : STARTED, Turn : Bob, Top-Right : O } Game Item
- 75. Amazon DynamoDB Alice Bob
- 76. Amazon DynamoDB Alice Bob Update: Top-Right : O Turn : Bob
- 77. Amazon DynamoDB Alice Bob Update: Top-Left : X Turn : Alice
- 78. Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3)
- 79. Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3)
- 80. Alice Bob (1) Amazon DynamoDB Bob (2) Bob (3)
- 81. Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Bob, Top-Right : O
- 82. Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Update: Turn : Alice Mid : X State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Low-Right : X
- 83. Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Update: Turn : Alice Mid : X State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X, Mid: X, Low-Right: X Update: Turn : Alice Low-Right : X
- 85. Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Bob (1) Amazon DynamoDB Bob (2) Bob (3) Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null
- 86. Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Bob, Top-Right : O Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null
- 87. Bob (1) Amazon DynamoDB Bob (2) Bob (3) State : STARTED, Turn : Alice, Top-Right : O, Top-Left : X Update: Turn : Alice Top-Left : X Expect: Turn : Bob Top-Left : null Update: Turn : Alice Mid : X Expect: Turn : Bob Mid : null Update: Turn : Alice Low-Right : X Expect: Turn : Bob Low-Right : null
- 88. Amazon DynamoDB Bob Alice Update: Turn : Alice Top-Right : X Expect: Turn : Bob Top-Right : null Update: Turn : Bob Top-Left : X Expect: Turn : Alice Top-Left : null
- 89. { Id : abecd, Players : [ Alice, Bob ], State : STARTED, Turn : Bob, Top-Right: O, Version : 2 }
- 91. Bob (1) Bob (2) Time
- 92. Bob (1) Bob (2) Read Version: 1 Turn: Bob Version: 1 Turn: Bob Time
- 93. Bob (1) Bob (2) Read Read Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 1 Turn: Bob Time
- 94. Bob (1) Bob (2) Time Read Read Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X
- 95. Bob (1) Bob (2) Read Read Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X Put If V=1 Time
- 96. Bob (1) Bob (2) Read Read Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 1 Turn: Bob Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X Version: 2 Turn: Alice Top-Right: X Put If V=1 Put If V=1 Time
- 97. @DynamoDBTable(tableName="TicTacToeGame") publicclassGame { privateString id; // ... other attributes privateLong version; @DynamoDBHashKey publicString getId(){...} @DynamoDBVersionAttribute publicLong getVersion(){...} }
- 98. @DynamoDBTable(tableName="TicTacToeGame") publicclassGame { privateString id; // ... other attributes privateLong version; @DynamoDBHashKey publicString getId(){...} @DynamoDBVersionAttribute publicLong getVersion(){...} }
- 99. // 1. Load Game Game game=dynamoDBMapper.load(Game.class,"cde3f"); // 2. See if the move is valid // 3. Update the Game object // 4. Save the game dynamoDBMapper.save(game);
- 103. Amazon Redshift Amazon EMR DynamoDB Amazon RDS Amazon CloudSearch dynamoDBMapper.save(game); transaction.commit();
- 108. records, tracking and analytics
- 109. high performance run time data retrieval
- 110. –Preparing a new wave of data to hit the service, tripling our coverage and data count
- 112. •Chop the big data stream files into small files •Store those billions of files on a big CDN file system
- 113. http://navteq-maps.ovi.com.edgesuite.net/144/0206/23/87/53/206238753_r10_f5_x0_y2.jpg We split the world into chunks –lots of those chunks have no streets We couldn’t implement our own tracking and analytics $$$$$$$$$$$$
- 114. Future flexible API
- 115. Keeps an index of all large files Stores large files Origin Service
- 118. RDS
- 120. 406311934325785de138c33-fa48-4418- 8ffa-54cc05e72adabigFile8098.big A server can look up a DynamoDB record by key (~10ms), find the s3 location and use the Amazon S3 API to read file chunks (~60ms)
- 124. ~50% 80%