Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2l2Rr6L. Doug Daniels discusses the cloud-based platform they have built at DataDog and how it differs from a traditional datacenter-based analytics stack. He walks through the decisions they have made at each layer, covers the pros and cons of these decisions and discusses the tooling they have built. Filmed at qconsf.com. Doug Daniels is a Director of Engineering at Datadog, where he works on high-scale data systems for monitoring, data science, and analytics. Prior to joining Datadog, he was CTO at Mortar Data and an architect and developer at Wireless Generation, where he designed data systems to serve more than 4 million students in 49 states.

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
datadog-cloud

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. The Evolution of a Data Project

5. The Evolution of a Data Project
Python
script

6. The Evolution of a Data Project
Python
script
SQL on
live DB

7. The Evolution of a Data Project
Python
script
SQL on
reporting DB
SQL on
live DB

8. The Evolution of a Data Project
Python
script
SQL on
reporting DB
SQL on
live DB
Terrible
confusion

9. The Evolution of a Data Project
Python
script
SQL on
reporting DB
SQL on
live DB
Terrible
confusion
Hadoop / Spark
cluster

10. What needs fixing
image: Pexels

11. What needs fixing
image: Pexels
• One cluster: data lock-in.

12. What needs fixing
image: Pexels
• One cluster: data lock-in.
• Want cluster time? You have to wait.

13. What needs fixing
image: Pexels
• One cluster: data lock-in.
• Want cluster time? You have to wait.
• Clusters are underutilized and EXPENSIVE

14. Elastic Big Data
Platform @ Datadog
Doug Daniels
Director, Engineering

15. WHOM
What’s our big data platform do?
Data Engineers
Data Scientists

16. WHOM
What’s our big data platform do?
Data Engineers
Data Scientists
do
WHAT
App features
Statistical Analysis/ML
Ad-hoc investigation

17. WHOM
What’s our big data platform do?
Data Engineers
Data Scientists
do
WHAT
App features
Statistical Analysis/ML
Ad-hoc investigation
WITH
Spark
Hadoop (Pig)
Python (Luigi)
with

18. Exploring the platform
COPIOUS
TOOLING
CLOUD
STORAGE
ELASTIC
COMPUTE

20. CLOUD STORAGE

21. What do we store?

22. 150 Integrations
…and more

23. What’s time series data?
timestamp 1447020511
metric system.cpu.idle
value 98.16687
tags host:i-xyz,
role:cassandra, …

24. We collect
over a trillion
of these per day
…and growing!

25. Where to put the petabytes?
Amazon S3.
Amazon S3

26. How data gets to S3
116
- Buffer
- Sort + Dedupe
- Upload
GO
- Partition + Sort
- Write Parquet
- Update Metastore
LUIGI/SPARK/PIG
HIVE METASTORE
Internal Format
AMAZON S3
Parquet Metadata
Kafka

27. Isn’t this a job for HDFS?

28. What we don’t love about HDFS

29. What we don’t love about HDFS
• Causes the “one cluster” problem

30. What we don’t love about HDFS
• Causes the “one cluster” problem
• Come for the storage, get stuck with the servers

31. What we don’t love about HDFS
• Causes the “one cluster” problem
• Come for the storage, get stuck with the servers
• No Java? No data!

32. S3 is flexible!
• Read data from as many clusters as you want

33. S3 is flexible!
• Read data from as many clusters as you want
• Store unlimited stuff(*) with no management
* Accepting laws of physics and your credit card limit

34. S3 is flexible!
• Read data from as many clusters as you want
• Store unlimited stuff(*) with no management
• Rock solid: durability (99.999999999), availability (99.99)
* Accepting laws of physics and your credit card limit

35. S3 is flexible!
• Read data from as many clusters as you want
• Store unlimited stuff(*) with no management
• Rock solid: durability (99.999999999), availability (99.99)
• Access from any programming language
* Accepting laws of physics and your credit card limit

36. Decouple data and compute
(BREAK THE RULES!)

37. Breaking the rules is fine.
In benchmarks: S3 is ~2X slower than HDFS

38. Breaking the rules is fine.
In benchmarks: S3 is ~2X slower than HDFS

39. It’s not all roses

40. Listing is slooooow
(A CAUTIONARY TALE)

41. How to fix slow listing
Bigger filesParallelize it

42. HDFS
No way to quickly move data
Task
Intermediate Final
write atomic move

43. HDFS
No way to quickly move data
Task
Intermediate Final
write atomic move
S3 Task
write

44. No way to quickly move data
• Say goodbye to speculative execution

45. No way to quickly move data
• Say goodbye to speculative execution
• Say hello to better task timeouts

46. But really: We 💜S3
This is a great system.
✓ Data accessible from many clusters
✓ Storage is easy to manage
✓ It’s a multi-language paradise up in here

47. ELASTIC
COMPUTE
CLOUD
STORAGE

48. One cluster to
compute it all
TRADITIONALLY

49. Instead, we run many, many clusters
• New cluster for every
automated job
• 10–20 clusters at a time
• Median lifetime: 2hrs

50. Why so many clusters?

51. Total isolation
We know what’s happening and why

52. No more waiting on loaded clusters
• Tailor each cluster to the work you want to do
• Scale up when you need results faster
• Data scientists and data engineers don’t have to wait
🕐🕓🕥

53. Pick the best hardware for each job
for CPU-bound jobs
r3
if you don’t care (cheap!)
== ~30% savings over general purpose hardware
c3
for memory-bound jobs
m1.xlarge

54. 100% spot-instance
clusters, all the time.*
* (ok, most of the time)

55. 100% spot-instance
clusters, all the time.*
* (ok, most of the time)
Ridiculous
savings!
Disappearing
clusters!

56. How we do spot clusters
• Bid the on-demand price,
pay the spot price
In the big data platform

57. How we do spot clusters
• Bid the on-demand price,
pay the spot price
• Fallback to on-demand
instances if you can’t get spot
In the big data platform

58. How we do spot clusters
• Bid the on-demand price,
pay the spot price
• Fallback to on-demand
instances if you can’t get spot
• Monitor everything: jobs,
clusters, spot market
In the big data platform

59. How we do spot clusters
• Bid the on-demand price,
pay the spot price
• Fallback to on-demand
instances if you can’t get spot
• Monitor everything: jobs,
clusters, spot market
• ☞ Save up to 80% off the
on-demand price
In the big data platform

60. Switch hardware when the market gets volatile
Monitor the spot price

61. We like this strategy a lot!
Cluster is oversubscribed; everyone waiting in line to do their work
Lots of expensive hardware sits idle when everyone’s gone
✓ No waiting for the cluster you need
✓ No waste from hardware sitting idle
✓ Spot clusters are affordable enough to use everywhere

62. What’s challenging,
though?

63. Many things that disappear.

64. ELASTIC
COMPUTE
CLOUD
STORAGE
COPIOUS
TOOLING

65. Web and APIs
Platform as a service
CLI
Jobs, Clusters, Schedules, Users, Code, Monitoring, Logs, and more

66. Big Data Platform Architecture
DATA Amazon S3

67. Big Data Platform Architecture
DATA Amazon S3
CLUSTER EMR

68. Big Data Platform Architecture
DATA Amazon S3
CLUSTER EMR
WORKER Pig Workers Spark Workers Luigi Workers

69. Big Data Platform Architecture
DATA Amazon S3
CLUSTER EMR
WORKER Pig Workers Spark Workers Luigi Workers
STORAGE Metadata DB Queueing Logs

70. Big Data Platform Architecture
DATA Amazon S3
CLUSTER EMR
WEB Web API
WORKER Pig Workers Spark Workers Luigi Workers
STORAGE Metadata DB Queueing Logs

71. Big Data Platform Architecture
DATA Amazon S3
CLUSTER EMR
WEB Web API
WORKER Pig Workers Spark Workers Luigi Workers
USER CLI API Clients Job Scheduler
STORAGE Metadata DB Queueing Logs

72. Big Data Platform Architecture
DATA Amazon S3
CLUSTER EMR
WEB Web API
WORKER Pig Workers Spark Workers Luigi Workers
USER CLI API Clients Job Scheduler
STORAGE Metadata DB Queueing Logs
Datadog
Monitoring

73. How to find the right cluster
when they disappear?

74. Cluster tagging
for discovery
#anomaly
-detection
#monitor-report

75. How to monitor many
disappearing clusters?

76. Dashboards Monitors
Dynamic Monitoring on Tags
anomaly-detection
cluster_tags: anomaly-detection

77. How to debug problems
when the cluster’s gone?

78. Debugging In a Post-Cluster World

79. Debugging In a Post-Cluster World
Send all logs to S3
• HDFS
• YARN
• Pig
• Spark

80. Debugging In a Post-Cluster World
Visualize the pipeline
• Lipstick for Pig
• Spark History Server
• Luigi task flow
Send all logs to S3
• HDFS
• YARN
• Pig
• Spark

81. Debugging In a Post-Cluster World
Visualize the pipeline
• Lipstick for Pig
• Spark History Server
• Luigi task flow
Preserve historical
monitoring data
Keep history, by tag, after
the cluster disappears
Send all logs to S3
• HDFS
• YARN
• Pig
• Spark

82. How to handle
certain cluster failure
in your jobs?

83. Automatic cleanup and restart
Luigi: design for failure.
A B

84. Automatic cleanup and restart
Luigi: design for failure.
B

85. Automatic cleanup and restart
Luigi: design for failure.
❌

86. Automatic cleanup and restart
Luigi: design for failure.

87. ELASTIC
COMPUTE
CLOUD
STORAGE
COPIOUS
TOOLING

88. Recommendations
for Cloud Big Data

89. Recommendations
for Cloud Big Data
• Use S3 for permanent data, not HDFS

90. Recommendations
for Cloud Big Data
• Use S3 for permanent data, not HDFS
• Start from EMR if building yourself

91. Recommendations
for Cloud Big Data
• Use S3 for permanent data, not HDFS
• Start from EMR if building yourself
• Look into a PaaS: Netflix Genie, Qubole, Databricks

92. Recommendations
for Cloud Big Data
• Use S3 for permanent data, not HDFS
• Start from EMR if building yourself
• Look into a PaaS: Netflix Genie, Qubole, Databricks
• Tag your clusters for dynamic monitoring

93. Recommendations
for Cloud Big Data
• Use S3 for permanent data, not HDFS
• Start from EMR if building yourself
• Look into a PaaS: Netflix Genie, Qubole, Databricks
• Tag your clusters for dynamic monitoring
• Design for failure with a workflow tool (Luigi, Airflow)

94. Thanks!
Want to work with us on Spark, Hadoop,
Kafka, Parquet, and more?
jobs.datadoghq.com
DM me @ddaniels888 or doug@datadoghq.com

95. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
datadog-cloud