Keystone processes over 700 billion events per day (1 peta byte) with at-least once processing semantics in the cloud. We will explore in detail how we leverage Kafka, Samza, Docker, and Linux at scale to implement a multi-tenant pipeline in AWS cloud within a year. We will also share our plans on offering a Stream Processing as a Service for all of Netflix use.

1. A NETFLIX ORIGINAL SERVICE
Streaming Data Pipeline @ scale in the Cloud
Monal Daxini

2. Monal Daxini
Real Time Data Infrastructure
Senior Software Engineer, Netflix
https://www.linkedin.com/in/monaldaxini
@monaldax
#Netflix #Keystone

3. Netflix Is a Data Driven Company
Content
Product
Marketing
Finance
Business Development
Talent
Infrastructure
←CultureofAnalytics→

4. Years ago...

5. In the Old Days ...
EMR
Event
Producers

6. About a year ago ...

7. Chukwa / Suro + Real-Time Branch
Event
Producer
Druid
Stream
Consumers
EMR
Consumer
Kafka
Suro Router
Event
Producer
Suro
Kafka
Suro
Proxy

8. ● Support at-least-once processing
● Scale, Multi-tenancy, Ease of Operations
● Enable future value adds - Stream Processing As a Service
● Replace dormant open source software - Chukwa
Why a new pipeline?

9. Goal - Migrate Events to a new Pipeline in flight,
while not losing more that 0.1% of them

10. Jan 2016

11. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Consumer
Kafka
Control Plane
Event
Producer
KSProxy

12. Global Launch - Jan 6, 2016

13. Over 75M Members
190 Countries
125M hours/day → 11B hours / quarter
14,269 years / day → 1,255,707 years / quarter
1000+ devices
37% of Internet traffic at peak

14. 700 billion unique events ingested / day
1 trillion unique events ingested / day - Dec 2015
1+ trillion events processed every day
Keystone Scale

15. 11 million events ingested / sec @ peak
24 GB / sec @ peak
Upto 10MB payload / Avg 3K
1.3 Petabyte / day
Keystone Scale

16. 99.99% + Availability / Four 9s
Keystone Scale

17. Want to know more...
Netflix Tech Blog - Pipeline Evolution

18. Event flow
Keystone Pipeline As a Service

19. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

20. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

21. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

22. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

23. Events & Producers

24. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

25. Event Payload is Immutable
At-least-once semantics*
* Once the event makes it to Kafka, there are disaster scenarios where this breaks.

26. Injected Event Metadata
● GUID
● Timestamp
● Host
● App
`

27. Custom Extensible Wire Protocol
● Backwards and forwards compatibility
● Support multiple serialization formats
○ JSON, AVRO, Protobuf in the works
● Additional metadata
● Efficient - 10 bytes overhead per message (hundreds of bytes to 10MB)

28. Netflix Kafka Producer
● Configurable - topic to Kafka clusters routing
● Sticky partitioner
● Prefer event drop than disrupt producer app
● Best effort delivery, ack = 1
● Buffer size tuning based on traffic

29. Kafka Clusters

30. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

31. ● Pioneer Tax
● Started with 0.7, went live with 0.8.2
● Done moving to 0.9 &
● VPC in progress
● Work closely with Confluent to get patches through - OSS contribution
Kafka in the Cloud

32. ● No dynamic topic creation
● Two copies
● Rack / Zone aware partition assignment
● Enable unclean leader election
Fronting Kafka Topics

33. ● 4000 + d2.xl brokers for fronting & consumer clusters
● 120 Zookeeper nodes (24 ensembles)
○ Independent zookeeper cluster per Kafka cluster
Scale - Kafka (prod)

34. ● 24 island clusters, 8 per region
○ 3 ASGs per cluster, 1 ASG per zone
○ 24 warm standby 3 node failover clusters
● 3 router offset checkpoint cluster, 1 per region
Scale - Kafka (prod)

35. Kafka Cluster Size -Tips
● Per Cluster Stay under 10k partitions & 200 brokers
● Leave approx. 40% free disk space on each broker

36. Kafka Kong
At least once
a week

37. Samza
RouterFronting
KafkaEvent
Producer
X

38. Fronting Kafka Failover
Self Service Tool

39. Fronting Kafka Failover

40. Fronting Kafka Failover

41. Kafka Auditor
Open sourcing on the road map

42. Kafka Auditor - One per cluster
● A service deployable on single or multiple instances
● Broker monitoring
● Consumer monitoring
● Heart-beat & Continuous message latency
● On-demand Broker performance testing

43. Kafka Metadata Visualization
Open sourcing on the road map

46. Want to know more...
Netflix Tech Blog - Kafka in Keystone Pipeline

47. Routing Service

48. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Control Plane

49. Routing Infrastructure
+
Checkpointing
Cluster
+ 0.9.1Go
C language

50. Router Job Manager
(Control Plane)
EC2 Instances
Zookeeper
(Instance Id assignment)
Job
Job
Job
ksnode
Checkpointing Cluster
ASG

51. Samza Job Deployment
● Multiple Samza jobs for one Kafka source topic
● Each job processes messages for one sink
● Each job processes partitions only from one topic
● One checkpoint topic per Kafka source topic

52. POWERFULL!
1 checkpoint topic per sink, & source topic
for many Samza Jobs

53. Immutable Config in Running Job

54. Custom Go
Executor
./runJob
Logs
Snapshots
Attach Volumes
./runJob
./runJob
Reconcile Loop - 1 min
Health Check
What’s running in ksnode?
Zookeeper
(Instance Id assignment)

55. Logs
ZFS Volume
Snapshots
Custom Go
Executor
.
/runJo
b
.
/runJo
b
.
/runJo
b
Go Tools Server
Client Tools
Stream Logs
Browse through
rotated logs by date
Ksnode Tooling

56. Yes! You inferred right!
No Mesos & No Yarn

57. Samza Alterations
ver 0.9.1

58. Using ThreadJobFactory in production
job.factory.class=org.apache.samza.job.local.ThreadJobFactory

59. SAMZA-41 - static partition range assignment
job.systemstreampartition.matcher.class=
org.apache.samza.system.RegexSystemStreamPartitionMatcher
job.systemstreampartition.matcher.config.ranges=[8-10]
^8&|^9$|^10$
you need

60. SAMZA-41 - static partition range assignment
Simplify...
job.systemstreampartition.matcher.class=
org.apache.samza.system.RangeSystemStreamPartitionMatcher
job.systemstreampartition.matcher.config.ranges=6-10

61. Prefetch Buffer - When is it going to OOM?
● Default count based per Samza container
○ (50,000 / # partitions) per topic
○ systems.source.samza.fetch.threshold=50000
● Cannot avoid OOM - variable message size

62. SAMZA-775- size based Prefetch buffer
● How much of heap should I use for prefetching?
○ systems.source.samza.fetch.threshold.bytes=200000000 (200MB)
○ per system / stream / partition
○ if > 0 precedence over systems.source.samza.fetch.threshold

63. SAMZA-775- size based Prefetch buffer
● Value of systems.source.samza.fetch.threshold.bytes based on
○ Incoming traffic Bps into source Kafka
○ 60 seconds of buffer with region failover traffic
○ Samza in memory data structures (2 x message size)

64. SAMZA-775- size based Prefetch buffer
● How does it perform?
○ Per message overhead within 0.02% of computed heuristics in the patch
○ Actual footprint of systems.source.samza.fetch.threshold.bytes is 10-15% at
the most in worst case.
■ Example: If set to 200MB, worst case observed 230MB

65. SAMZA-655 & SAMZA-540
● Backported from 0.10
○ environment variable configuration rewriter
■ Pass config from RDS to executor to Docker to Samza Job
○ expose latency related metrics in OffsetManager
■ checkpointed offset guage

66. ● 14,000+ docker containers (samza jobs)
● 1,400+ AWS C3-4XL instances
Scale - Routing Service

67. More Info - Samza Meetup (10/2015)
Samza ver 0.9.1 Contributions

68. Metrics & Monitoring

69. Keystone
Stream
Consumers
Samza
Router
EMR
Fronting
Kafka
Consumer
Kafka
Control Plane
Event
Producer
KSProxy

70. Customer Facing per topic end-to-end dashboard

71. Dev facing infrastructure end-to-end dashboard

72. Scaling Avenues
1 trillion / day
Now What?

73. ● Exposed cost attribution per event producers & topic
○ E.g. one producer reduced throughput by 600%
● Automation - frees up additional resources
Scaling Up by Scaling Down
Oxymoron?

74. Culture

75. Not DevOps, but move towards NoOps
You build it! You run it!

76. Team has
● No dedicated product or project managers
● No separate devops or operations team
We build and run what you saw today!

77. ● This does not mean we are constantly overworked
○ we make wise and simple choices and
○ lean towards automation & self-healing systems
We build and run what you saw today!

78. Looking into the future?

79. ● Data thruway
○ Support for schemas - registry, discovery, validation
● Self Service Tooling
Future steps

80. Stream Processing As a Service (SPaaS)
Multi-tenant polyglot support for stream processing engines

81. Big Data Systems - streaming
Data Pipeline & Stream processing - Keystone - Samza / Flink (poc)
Playback & edge Operations insight - Mantis
Stream Processing - Spark Streaming
* Metrics & monitoring - Atlas *

82. Apache Beam
○ Portable API layer for building sophisticated data processing applications
○ Unified model API over bounded and unbounded data sources
○ Google lineage - Dataflow model
SPaaS - “Beam Me Up, Scotty ! "

83. SPaaS
SPaaS UI
Container
Runtime
Beam API
Bounded / Unbounded
Data Source
Dockerized Job
1. Create 2. Submit 3. Launch
Runner
Mantis / Flink /
Spark
Running
Job
1. Submit
DSL Job
Job Dashboard

84. Apache Beam + Apache Flink
SPaaS - init( )

85. More brain food...
Netflix OSS
Samza Meetup Presentation
Netflix Tech Blog
Spark Summit 2015 Talk