The Data Mullet: From all SQL to No SQL back to Some SQL

The Data Mullet
From All SQL to No SQL back to Some SQL
Alexis Lê-Quôc @alq

This Talk
• A (mostly) DIRTy Architecture for...
• A new application (datadoghq.com) on a limited budget
• Running on a public cloud
• Focussing on data stores.

Servers
Monitoring
IaaS, PaaS Usage Analytics
Perf. Management
Apps
Hosting
CDNs Asset Management
SDLC
Ops team Dev team

Dev & Ops “collaborate”

Concretely, what does Datadog
do?

Watching real
time feeds
Looking for patterns
Constant telemetry
Real-time
Bursty batches
Share

Data Taxonomy
Metrics
Unique visitors
Load
Transaction duration
...
Events
Conversations
Alerts
Build & Deploys
...

Unit of scale
• 1 source, typically a server
• 100 metrics
• Every 15 s
• 24,000 points per hour
• ~3 bytes per point
• 100 KB/hour, 850 MB/year
• Events
• whenever they occur
• Highest resolution: 1s
• Small payload + metadata

ACID, BASE & DIRT
• ACID
• http://en.wikipedia.org/wiki/ACID
• BASE
• http://en.wikipedia.org/wiki/Eventual_consistency
• DIRT (Bryan Cantrill at Surge 2010)
• http://dtrace.org/resources/bmc/DIRT.pdf

The Consequences of DIRT?
Latency
• Data consumed by people (and machines)
• Low end-to-end latency (5-15s)
• Psycho-physiological Factor
• Same order of magnitude as email/SMS*
http://citeseerx.ist.psu.edu/viewdoc/* download?doi=10.1.1.76.2465&rep=rep1&type=pdf

Concurrency
• Concurrent events & data points show up in sync
• Access Patterns?
• All recent data, e.g. last 24 hours

Tolerance to noise
• Not a System of Record
• “Real-time” decisions
• Drop (some) individual data points rather be late
• Applies to metrics, not events

Noise but no Latency Latency but no Noise
Cross here? Or here?

Storage
• Business Cycles
• Retention Policy > Business Cycle
• E.g. retail, education 12 months
• Elastic Storage
• !CAPEX

Latency
• Datadog, a data exploration app for people
• Looking for patterns
• Ideal: 300 ms round-trip
• Access patterns for long-term data?
• Storage trade-off: precompute oft-used properties
• Run-time Trade-off: want longer timespan, get lower resolution
• != RRD

Aggregate
Constant data influx
Large data sets

Aggregate
Large data sets
Watch & Share
Real-time updates
On-the-fly data analysis

Watch & Share
Real-time updates
Aggregate
Large data sets
Look for Patterns
On-demand visualization
Background data analysis

Watch & Share
Real-time updates
Aggregate
Constant BASE
data DIRT
influx
Large data sets
Look for Patterns

Watch Real-time DIRT
& Share
updates
Aggregate
Constant BASE
data DIRT
influx
Large data sets
Look for Patterns

& Share
updates
Aggregate
Constant BASE
data DIRT
influx
Large data sets
Look for On-demand BASE
Patterns
visualization

& Share
updates
Aggregate
Constant BASE
data DIRT
influx
Large data sets
Look for On-demand BASE
Patterns
visualization
Datadog = DIRT + BASE + a tiny bit of ACID

How It All Fits Together

The Mullet
All SQL in front, NoSQL party in the back

Actual Stack

Choices, choices
• 5 axes
• Volume of Data
• Latency
• Ops: wake-up-in-the-middle-of-the-night factor
• Dev: community & tools
• Cost as in “a function of X”

Durable, Large-Scale Storage
• Postgres
• Mongo
• Cassandra
• (Riak)
• SciDB

• Postgres
• Itemized data points in a time series are useless
• BLOB management not fun
• Mongo
• Cassandra
• (Riak)
• SciDB

• Postgres
• Mongo
• SciDB
• Cassandra
• (Riak)

• Postgres
• Mongo
• Durability in question in 2010
• SciDB
• Cassandra
• (Riak)

• Postgres
• Mongo
• SciDB
• Very very early
• Cassandra
• (Riak)

• Postgres
• Mongo
• SciDB
• Our pick: Cassandra
• (Riak)

Cassandra: Volume of Data
• 100s of hosts, 150TB at FB in 2010
• Easy to distribute data, durable quorum writes

Cassandra: Latency
• < 10ms on writes
• reads more variable (on EC2)*
* More on this in a bit

Cassandra: Ops
• Release Engineering too aggressive
• ~10 releases since 1/2011 on 0.7 branch
• Good resilience to node loss in the later 0.7 versions
• Annoying idiosyncrasies (cassandra.yaml, predictability of disk use)

Cassandra: Dev
• Bizarre nomenclature (rows, columns... families?)
• Cumbersome data access
• Limited Semantics when used to SQL
• Good libraries

Cassandra: Cost
• Ops time
• I/O limits raised by increasing number of nodes
• Thereby increasing costs,

Riak
• Prototyped out of spite for Cassandra 0.7[0123]
• We ♡ Erlang
• Great folks
• But Cassandra pain subsided, priorities shifted.
• git merge datadog/riak did not happen

In-memory DB
• We started with Redis
• Then we stopped looking :)

Redis
• Volume of Data
• Limited by available RAM, easy partitioning in our case
• Latency
• << 5 ms, dominated by network
• Ops
• Low-maintenance, stable, predictable, replicated, boringly rock-solid
• Dev
• Brilliant, clear docs, simple protocol, oft-used native data structures
• Cost
• ~ cost of RAM on EC2

General-purpose data store
• We ♡ SQL
• Oracle
• Postgres

Oracle in numbers
• base license 47.5
• clustered db 23
• replication 10
• partitioning 11.5
• analytics 23
• in-mem cache 23
• total: $138,000

Oracle in numbers
• base license 47.5
• clustered db 23
• replication 10
• partitioning 11.5
• analytics 23
• in-mem cache 23
• total: $138,000
• for 2 cores
• + 22% annual support
• Just in licenses...

General-purpose data store
• Oracle
• Postgres

Postgres
• Volume of Data
• High GBs, Low TBs
• Latency
• 10-100 ms after EXPLAIN ANALYZE
• Ops
• Low-maintenance, stable, predictable, replicated, boringly rock-solid
• Dev
• Well understood by (a certain class of) engineers
• Cost, a function of storage latency

Not forgetting...
• VoltDB
• RAM-based, potentially a match for our DIRTy parts
• Stored procedures, an acquired taste
• Home-grow data stores (soon)
• Rainbird
• ...

The Data Mullet
• All open-source, good if you’re ready to dive in code
• $0 CAPEX
• All OPEX on EC2

The Data Mullet on EC2
Structural Weakness: I/O latency at moderate throughputs

One “bad” cassandra query

Clogging the I/O pipes on EC2
Maximum Average Wait: up to 670 ms
Maximum Service Time: up to 5 ms
While writing 100 MB/s
and reading 30 MB/s

Average wait in ms
Transfer per seconds
Consumer HD: ~75 tps
SSD: 1-30 Ktps
DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
03:35:02 PM dev8-80 380 24000 5.7 62 47 130 1.3 47
03:35:02 PM dev8-96 370 24000 5.6 63 46 120 1.2 45
03:35:02 PM dev8-112 380 24000 5.5 63 46 120 1.2 46
03:35:02 PM dev8-128 380 24000 7.2 63 56 150 1.3 50
Average service time in
ms
Read throughput in sector/s
Total: 46 MB/s
Another “Bad” Query

Cassandra: I/O Mitigation
• More nodes, more RAM, more partitions, more parallelism
• $$$

Postgres: I/O Mitigation
• Scale up to a point
• Replicate
• Move to bare Metal => $$$
• A well-trodden but difficult path

Better yet...
• Less dependency on low-latency, durable storage
• Move more data to RAM (Redis)
• Archive immutable data
• S3/Cloudfront?

A digression:
Your Very Own Chaos Monkey
• Instances go bye-bye
• https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/741224
• Instances go bye-bye, take 2 (high load)
• https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708920

Takeaway
• By mixing and matching open-source SQL (PG) and NoSQL (Redis,
Cassandra) Datadog has been able to quickly & simply get up-and-running
with “$0” down payment on infrastructure.

http://datadoghq.com
@datadoghq

The Data Mullet: From all SQL to No SQL back to Some SQL

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (19)

Destacado

Destacado (19)

Similar a The Data Mullet: From all SQL to No SQL back to Some SQL

Similar a The Data Mullet: From all SQL to No SQL back to Some SQL (20)

Más de Datadog

Más de Datadog (15)

Último

Último (20)

The Data Mullet: From all SQL to No SQL back to Some SQL