Learn about the thought and decisions that went into designing and building DynamoDB. We'll talk about its roots and how we can deliver the performance and throughput you enjoy today. We’ll also show you how to model data, maintain maximum throughput, and drive analytics against the data with DynamoDB. Finally, you'll hear from some of our customers on how they've built large-scale applications on DynamoDB and about the lessons they've learned along the way.
31. The DynamoDB Uniform Workload
DynamoDB divides table data in to multiple partitions
Data is distributed primarily by primary key
Provisioned throughput is divided evenly across partitions
32. The DynamoDB Uniform Workload
To achieve and maintain full provisioned throughput
for a table, spread the workload evenly
across primary keys
33. Non-uniform workloads
Some requests might be throttled,
even at high levels of provisioned throughput
38. date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00
39. Table
date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00
40. date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00 Item
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00
41. date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00 Attribute
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00
42. Items are indexed by primary key
Single hash keys and composite range keys
43. Hash key
date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00
44. Range key
date = 2012-05-16-09-00-
id = 100 10 total = 25.00
date = 2012-05-15-15-00-
id = 101 11 total = 35.00
date = 2012-05-16-12-00-
id = 101 10 total = 100.00
date = 2012-03-20-18-23-
id = 102 10 total = 20.00
54. Distinct values for hash keys
Hash key elements should have a high
number of distinct values
55. Lots of unique user IDs: workload well distributed
user_id = first_name = last_name =
mza Matt Wood
user_id = first_name = last_name =
jeffbarr Jeff Barr
user_id = first_name = last_name =
werner Werner Vogels
user_id = first_name = last_name =
mattfox Matt Fox
... ... ...
56. Limited response codes: workload poorly distributed
status = date =
200 2012-04-01-00-00-01
status = date =
404 2012-04-01-00-00-01
status date =
404 2012-04-01-00-00-01
status = date =
404 2012-04-01-00-00-01
60. What we’ll cover
faбrik overview
Getting more out of DynamoDB with python/boto
– More throughput / provisioned capacity
– Across more endpoints / table
– More reliably and controllably
60
61. Frank McCloud: “He wants more, don't you, Rocco?”
Johnny Rocco: “Yeah. That's it. More. That's right! I want more!”
James Temple: “Will you ever get enough?”
Frank McCloud: “Will you, Rocco?”
Johnny Rocco: “Well, I never have. No, I guess I won't.”
61
62. Takeaways
Messaging infrastructure is cool (again)
Old dogs have tricks you can apply
– The Internet is your friend
– BUT: much good computer science was done prior
– HENCE: not so readily findable
Boto is great – clone and contribute!
62
63. NYT Mission
Enhance society by creating, collecting and distributing high quality
news, information and entertainment
- Distributing: publish / subscribe
- Collecting: gather / analyze
- High Quality: fast, reliable, accurate
63
64. faбrik
Asynchronous Messaging Framework
For client devices as well as our apps
Enabled by:
– Websockets
– Robust message handling software
– Amazon Web Services
Focusing on simple, common services
64
65. Typical Web Architecture
Clients interact with front-end via load balancers
Front-end makes requests to back-end on behalf of client
Bottlenecks abound
Information transfer is initiated by client
65
68. faбrik Web Architecture
Clients interact with the nearest “App Buddy” front-end
The “App Buddy” is connected to the “Bad Rabbit” backbone
The “Bad Rabbit” backbone is clustered regionally and federated globally
NYT content producers connect directly to the backbone
Information flow is bidirectional and event-driven
68
69. faбrik Information Flow
Client
Client Client
NYT Globally distributed
Internal “faбrik ” layer
69
80. So why DynamoDB?
faбrik services are reliable but stateless (mostly)
A happy faбrik has short queues (measurable by the way)
So persist everything as rapidly as possible (enter DynamoDB)
Plus we want to gather & analyze
– Pulse: Map / Reduce, rapid cycle
– Longitudinal analysis
– Complex Event Processing in parallel (maybe)
Note: the faбrik is asynchronous and facilitates parallelization
80
81. DynamoDB requirements
Store all messages crossing each ‘virtual host’
Note: think of a ‘virtual host’ as a horizontal band of related, reliable
services/endpoints across zones/instances in a region
Store log messages for all application and system instances
Facilitate ‘burst’ loads as well as steady state
Support gather / analyze for all of the above
Generational storage: DDB to S3 to Glacier (with some weeding)
Fairly allocate resources among many competing endpoints
81
83. More conventional wisdom…
“In addition to simple retries, we recommend using an exponential backoff algorithm
for better flow control. The concept behind exponential backoff is to use progressively
longer waits between retries for consecutive error responses. For example, up to 50
milliseconds before the first retry, up to 100 milliseconds before the second, up to 2400
milliseconds before third, and so on. However, after a minute, if the request has not
succeeded, the problem might be the request size exceeding your provisioned
throughput, and not the request rate. Set the maximum number of retries to stop around
one minute. If the request is not successful, investigate your provisioned throughput
options.” [i.e. increase provisioned throughput – hmmmm…]
83
84. So…
We would have to provision for peaks
Exponential backoff would give us about a 1 minute buffer
But! The faбrik does buffering and we can monitor queue lengths
Plus we have asynchronous event scheduling/handling facilities built in…
84
85. First strategy
With node.js, asynchronously blast all requests at dynamo, reschedule exponentially
based on backpressure
This worked pretty well!
– * Dynamo would deliver about 3 times stated capacity in bursts
– Nothing got lost
– Converged reasonably onto table capacity
But…
– Problems exerting backpressure on the faбrik from node.js… hence requests could get scheduled
WAY into el futuro… and WAY out of order
– Competition among endpoints was ‘unfair’ and fostered convergence problems
85
86. Current strategy
Be smarter, look for similar patterns and tested solutions, plus select tools that give
the right level of control
Old dog:
– “I remember when TCP was new and throughput was not very high…”
(time passes)
– “The ‘ThroughputExceeded’ backpressure from DynamoDB is sort of like TCP backpressure…”
(more time passes)
– “Perhaps we could leverage that thought by applying the research and practices that have
improved TCP etc. to our use of DynamoDB.”
(time for a nap)
86
87. Current strategy
Be smarter, look for similar patterns and tested solutions, plus select tools that give
the right level of control
Token Bucket (circa 1986) for traffic shaping
“…an algorithm used in packet switched computer networks and telecommunications networks to
check that data transmissions conform to defined limits on bandwidth and burstiness.” – Wikipedia
Additive Increase/Multiplicative Decrease (AIMD)
“…a feedback control algorithm best known for its use in TCP congestion control…combines linear
growth of the congestion window with an exponential reduction when congestion takes place…flows
will eventually converge to use equal amounts of a contended link.” – Wikipedia
Explicit Congestion Notification (ECN) etc. etc.
87
88. Current strategy
Be smarter, look for similar patterns and tested solutions, plus select reliable tools
that give the right level of control
Tools:
– Use python to get a more mature and lower level event-driven interface (pika) to RabbitMQ –
easier to exert backpressure on the message source
– Use boto to get a mature interface to DynamoDB that can be easily ‘tweaked’ to give better
information about backpressure from DynamoDB (ThroughputExceeded exception)
– Use python’s concurrent futures to easily add asynchronous capability to boto, making use of
boto’s connection pooling
88