1. Events & Metrics
The Lifeblood Of Webops
Alexis Lê-Quôc (Product Guy) at Datadog
NYCBUG
July 6th, 2011

2. I <3 BSD
‣OpenBSD user since 2.8 (pf)
‣Love the documentation
‣m0n0wall/pfSense
‣ZFS-envy

3. What I’m going to talk about
‣Briefly we do and for whom
‣Where we started
‣The kind of data we deal with
‣How it fits altogether
‣A few things we learned along the way
‣Q+A

5. SaaS Platform for Dev & Ops
‣Aggregation
‣Correlation
‣Collaboration
What we do?

6. Where We Started

7. The Mess
Usage Analytics Too many data streams,
too many silos
IAAS / PAAS
Issue Resolution
t
ics
Servers and Devices
s igh
ices
tric ins
metr
g
e Too many choices to
billin
m m
cho
et
ric s make, too often
s
?!? change
Dev team
changes !? s
ic choices
metr
Ops team Applications
s
t ric ch
an
me ts ge
me
even Only getting worse as
ev en ts s
tri
ad
s + fe
ice edb SaaS Silos multiply
cs
vic
o ack
ch
e
me
s
s
tric
choice
tri
me
cs
Cap. Planning SDLC support
Monitoring
Hosting Asset Mgmt
CDNs
Separate Dev and Ops
teams, looking at separate
data streams

8. WHERE WE STARTED
Discourages exploration

9. Very Specific View

10. Different View
Same Reality

11. Dev Interdiction
Part 1

12. Where We Are

13. In Action
https://app.datad0g.com/dash/host/8#/date_range/1309383780732-1309988580732

14. Welcome developers Context Matters
‣Graphite ‣Ganglia Event API
‣statsd
Large Datasets Data Exploration
‣OpenTSDB ‣d3, protovis
TRENDS
Visible through Datadog and others

15. Sides Of A Coin

16. Events Metrics
User comments Unique visitors
Alert Load
Build Transaction duration
Batch job etc.

17. etc.
Aggregation

18. Taxonomy

19. Atomicity
Concistency
Isolation
Durability
e.g. SQL DBs
CLASSICS
http://en.wikipedia.org/wiki/Eventual_consistency

20. Atomicity Basically
Concistency Available
Isolation Soft-state
Durability Eventual
consistency
e.g. SQL DBs
e.g. DNS
CLASSICS
http://en.wikipedia.org/wiki/Eventual_consistency

21. Data
Intensive
Real
Time
e.g. real-time web
NEW COMER
Brian Cantrill: http://dtrace.org/resources/bmc/DIRT.pdf

22. Aggregation
Constant data influx
Large data sets
Correlation
On-demand visualization
Background data analysis
Collaboration
Real-time updates
On-the-fly data analysis

23. Aggregation
SE
Constant data influx
BA
Large data sets
Correlation
On-demand visualization
Background data analysis
Collaboration
Real-time updates
On-the-fly data analysis

24. Aggregation
SE
T
Constant data influx
IR
BA
D
Large data sets
Correlation
On-demand visualization
Background data analysis
Collaboration
Real-time updates
On-the-fly data analysis

25. Aggregation
SE
T
Constant data influx
IR
BA
D
Large data sets
Correlation
SE
On-demand visualization
BA
Background data analysis
Collaboration
Real-time updates
On-the-fly data analysis

26. Aggregation
SE
T
Constant data influx
IR
BA
D
Large data sets
Correlation
SE
On-demand visualization
BA
Background data analysis
Collaboration
T
Real-time updates
IR
D
On-the-fly data analysis

27. Aggregation
SE
T
Constant data influx
IR
BA
D
Large data sets
Correlation
SE
On-demand visualization
BA
Background data analysis
Collaboration
T
Real-time updates
IR
D
On-the-fly data analysis
Datadog = DIRT + BASE + a tiny bit of ACID

28. How It All Fits Together
http://www.flickr.com/photos/tom-margie/1253798184/

29. Architecture
Simplified

30. Architecture
Simplified
SE
BA

31. Architecture
Simplified
SE
T
IR
BA
D

32. Architecture
Simplified
SE
ID
T
IR
C
BA
A
D

33. The Environment

34. 4 Dimensions
Compute
Storage
Network
Management

35. ON-PREMISE TRAITS
http://www.flickr.com/photos/theplanetdotcom/4879419788/sizes/l/in/photostream/

36. Compute
Fast
Inelastic
ON-PREMISE TRAITS
http://www.flickr.com/photos/theplanetdotcom/4879419788/sizes/l/in/photostream/

37. Compute
Fast
Inelastic
Storage
Fast
Centralized
Redundant
ON-PREMISE TRAITS
http://www.flickr.com/photos/theplanetdotcom/4879419788/sizes/l/in/photostream/

38. Compute Network
Fast Fast
Inelastic Localized
Storage
Fast
Centralized
Redundant
ON-PREMISE TRAITS
http://www.flickr.com/photos/theplanetdotcom/4879419788/sizes/l/in/photostream/

39. Compute Network
Fast Fast
Inelastic Localized
Storage
Fast Management
Centralized People-based
Redundant Full access
ON-PREMISE TRAITS
http://www.flickr.com/photos/theplanetdotcom/4879419788/sizes/l/in/photostream/

40. CLOUD TRAITS

41. Compute
Slow
Elastic
CLOUD TRAITS

42. Compute
Slow
Elastic
Storage
Slow
Jittery
Maybe durable
Low memory
CLOUD TRAITS

43. Compute Network
Slow “Fast”
Elastic Geo-distributed
Storage
Slow
Jittery
Maybe durable
Low memory
CLOUD TRAITS

44. Compute Network
Slow “Fast”
Elastic Geo-distributed
Storage
Slow
Jittery Management
Maybe durable No bare-metal
Low memory “Magic” API
CLOUD TRAITS

45. What We Have
Found

46. Network

47. Network
Layer 2: Virtual Domain
Layer 3: Crude Edge Filtering
Layer 7: Crude Load Balancing
DNS
CDN

48. Network
Layer 2: Virtual Domain
!
Layer 3: Crude Edge Filtering
ks
or
Layer 7: Crude Load Balancing
DNS
W
It
CDN

49. Storage

50. Latency
BASE
Amazon S3
BASE
Apache Cassandra
ACID
PostgreSQL
DIRT
Redis
Capacity
Storage

51. Latency
BASE
y
nc
Amazon S3
te
La
t
BASE
pu
y
gh
er
Apache Cassandra
ou
ACID tt
hr
Ji
dt
PostgreSQL
i te
Lim
DIRT
y
or
em
Redis
Capacity
m
w
Lo
Storage

52. Low Memory
http://aws.amazon.com/ec2/#instance

53. Jittery, Limited Throughput
Network Block Storage (EBS)
https://app.datad0g.com/dash/dash/1032#/date_range/1308608717016-1309213517016

54. Average wait in ms
DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
03:35:02 PM dev8-80 375.95 23614.08 5.70 62.83 47.21 125.58 1.26 47.34
03:35:02 PM dev8-96 373.63 23749.65 5.64 63.58 45.55 121.91 1.22 45.72
03:35:02 PM dev8-112 375.28 23693.47 5.52 63.15 45.52 121.22 1.23 46.31
03:35:02 PM dev8-128 375.31 23721.57 7.19 63.22 56.00 148.96 1.34 50.35
Read throughput in sector/s Average service
Total: 368Mb/s time in ms
Limited Throughput In Numbers
RAID 0 EBS Volumes, m1.large instances

55. Some Tricks

56. Software RAID
RAID 0
Offsite backups
Some Tricks

57. Software RAID Limited by slowest
RAID 0 volume
Offsite backups
Some Tricks

58. Software RAID Limited by slowest
RAID 0 volume
Offsite backups
Streaming replication
S3 backups
Some Tricks

59. Software RAID Limited by slowest
RAID 0 volume
Offsite backups
Ephemeral volumes
And Offsite backups
Streaming replication
S3 backups
Some Tricks

60. Software RAID Limited by slowest
RAID 0 volume
Offsite backups
Ephemeral volumes
And Offsite backups Complexity
Recovery Time Objective
Streaming replication Recovery Point Objective
S3 backups
Some Tricks

61. Software RAID Limited by slowest
RAID 0 volume
Offsite backups
Ephemeral volumes
And Offsite backups Complexity
Recovery Time Objective
Streaming replication Recovery Point Objective
S3 backups
Database Service
MySQL/Oracle RDS
Some Tricks

62. Software RAID Limited by slowest
RAID 0 volume
Offsite backups
Ephemeral volumes
And Offsite backups Complexity
Recovery Time Objective
Streaming replication Recovery Point Objective
S3 backups
Database Service Trust
MySQL/Oracle RDS RDS Outage 2 months ago
Some Tricks

63. Network Block Storage
Is The Dark Side

64. Network Block Storage
Is The Dark Side
Bait For Enterprise
Customers

65. Network Block Storage
Is The Dark Side
Bait For Enterprise
Customers
Hard Problem For
Cloud Providers

66. Don’t rely on networked block storage
Small data sets only if you have to
Don’t trust data-at-rest
Copy, replicate, back up
Do use S3 if you can
Object semantics a limitation
Slow but durable
Some Do’s And Don’t

67. Compute

68. “Performance”
Scale up Shard
ACID
Nodes
BASE DIRT Add more
Nodes Nodes
Number
Compute

69. Don’t rely on scale-ups
Low memory a hard limit for DBs
Noisy neighbors
Individual performance poor and jittery
Scale out
First scale up
Then Shard
Parallelize across machines
Vector-processing via GPUs
Some Do’s And Don’t

70. Management

71. An API for everything
Compute
Storage
Network
Management

72. Questions!
http://datadoghq.com
twitter: @alq