Netflix manages petabyte-scale Apache Cassandra databases in the cloud through declarative infrastructure and tooling. They provision new Cassandra clusters quickly using robust AMIs and pre-flight checks. Existing clusters are kept running through declarative control planes that monitor desired states and automatically remedy issues. Netflix migrates software and hardware seamlessly through immutable infrastructure, incremental backups to S3, and parallel data transfers. They apply lessons from failures like accidentally deleting real backup data through over-automation.

1. How Netflix manages
petabyte scale
Apache Cassandra
in the cloud
Joey Lynch, Vinay Chella
Netflix’s Distributed Database Engineers

2. Who are we?
Vinay Chella
Distributed Systems Engineer
Focusing on Apache Cassandra and Data
Abstractions
Cloud Data Engineering
Netflix
Joey Lynch
Distributed Systems Engineer
Distributed system addict and data wrangler
Cloud Data Engineering
Netflix

3. Agenda Why use Cassandra?
Scale of Cassandra
Life of Cassandra Cluster
- Where does it start?
- Provisioning
- Keep it running
- Migration / Retiring
Murphy’s law applied

4. Why
Apache
Cassandra
— Millions of operations per sec
— Global data replication
— Failure isolation at rack level
— Chaos ready database
— Tunable consistency
— Log structured storage engine

5. — 10’s of thousands instances
— 100’s of global C* clusters
— >6 PB of data
— Millions of requests / second
— Replicating several GiB/sec data
across the globe
Scale

6. Story of Apache Cassandra and Netflix
Inception Provision Keep it running Migrations

7. Inception
Where does it all start?

8. Inception
Inception

9. Inception
Service philosophy
— Context not control
◆ Education
◆ Tooling
— SLOs are key
◆ Size, rate, latency,
availability
— Every party must be
responsible

10. Inception
Inception

11. Inception
Invest in DevEd

12. Inception
Inception

13. Better Tooling

14. Inception
Cost insights

15. Inception
Maintenance

16. Inception
Maintenance - Repair Insights

19. Maintenance - Backup Insights

22. Inception
Maintenance - Node Insights

24. Inception
SLOs are Key

26. Inception
Whom to page?

27. Inception
Good contracts
make good
partners!

28. Story of C*
Inception
Provision
Keep it running Migrations

29. Provision
Get up and running fast!

31. Provision
Prove it before serving it
Robust AMI build process Pre-flight check

32. Provision
AMI Build
process

33. Provision
Pre-flight check EC2 Instance lifecycle

34. Story of C*
Inception Provision
Keep it running
Migrations

35. Keep it
Running

36. Keep it Running
Imperative Control Plane Declarative Control Plane

37. Keep it Running
Declarative Control PlaneImperative Control Plane
Pros:
— Easy to implement
— Easy to understand
Cons:
— Fragile to failure
— Scaling issues
— Have to implement
parallelism
Pros:
— Lower total cost of
ownership
— Parallel out of the box
Cons
— Requires different
engineering mindset
— Hard to implement
— Have to stop
parallelism

38. Keep it Running
Declarative ToolsImperative Tools
jenkins.io/
— “Orchestrators”
◆ Jenkins
◆ Fabric
◆ Ansible
— Imperative
operation
ansible.com/fabfile.org/
— Distributed Databases
— Distributed operation
— Similar to Actor model
puppet.com
consul.io/

39. Declarative Control Plane
1. Obtain desires from
datastore
2. Obtain state of world
3. If state != desire, act!
Keep it Running

40. Keep it Running
Ring Health Imperative Solution
— O(#nodes) commands
— Single Network Path
— Hard to synthesize global
view

41. Keep it Running
Ring Health Declarative Solution
Stream Processing!

42. — Desire: Healthy <30m
— Observe: Full cluster
state
— Action: Remediate or
page
Keep it Running
Ring Health Declarative Solution

43. — O(#nodes) messages
— Degraded hardware is
hard to SSH into
◆ Failed ephemerals
◆ Failed network
Keep it Running
Hardware Health Imperative Solution

44. Keep it Running
Hardware Health Declarative Solution
— Desire: Degraded
hardware should die <10m
— Observe: Local disks,
creds, filesystems
— Action: Request
termination

45. Keep it Running
JVM Health Declarative Solution
— Desire: JVM
Throughput > 90%
— Observe: GC vs
Runtime
— Action: core dump +
terminate
https://github.com/Netflix-Skunkworks/jvmquake

46. Keep it Running
Backup Declarative Solution
— Desire: Full snapshot
every interval
— Observe:
◆ State of data
◆ State of S3
— Action:
◆ Upload diff

47. Keep it Running
Case Study: Cassandra Repair

48. Keep it Running
Case Study: Cassandra Repair

49. Keep it Running
At Scale Subrange is Required

50. Keep it Running
Repair Imperative Solution
— O(lots) commands
— Remote JMX = :(
— Single point of failure = :(
— 1 month job duration = D:

51. Keep it Running
Repair Declarative Solution
— Distributed Control Plane
— Makes progress when
database is available
— Every node follows repair
state machine
https://issues.apache.org/jira/browse/CASSANDRA-14346

52. Keep it Running
Repair Declarative Solution (#14346)

53. Keep it Running
Cassandra Ecosystem

54. Keep it Running
C* Instance ecosystem
PAGE
CDEPre-flight check
AWS IO detection
W
I
N
S
T
O
N
CDE Service
Alert Atlas Mantis
C*
C*
C*
Priam
Bolt
Cluster
Metadata
Remediation
C*

55. Story of C*
Inception Provision Keep it running
Migration

56. Migration

57. Migration
Migration Philosophy
— Immutable infrastructure
— Seamless and transparent migrations

58. Migration
Software Migration

59. Migration
Hardware Migration
Desires
S3
Instance
(i2.xl)
agent
C* Data
Incremental
backup of C*
Data
desired
instance?
i2.2xl
Instance
(i2.2xl)
agent
C* Data
Download
latest
SSTables
from S3
Copy differential
data from source
Desires
desired
instance? i2.2xl

61. Hardware Migration: The Numbers
Naive solution,
sequential transfer
Parallel S3 download
sequential fixup
Parallel download and
zone wide fixup

62. Migration
Data Migration

63. Cast Studies
Murphy’s
law

64. Cautionary Tale: Rarely run automation
— Auto remediation to clean up stale backups,
repair snapshots
◆ Triggers only above FATAL disk threshold
◆ Assumption mismatch

65. Cautionary Tale: Rarely run automation
— Auto remediation to clean up stale backups,
repair snapshots
◆ Triggers only above FATAL disk threshold
◆ Assumption mismatch
— Result:
◆ Automation ended up deleting real backup
data

66. Do you restore your backups constantly?

67. Take Away
Take Away
Invest in context sharing, tooling and
declarative(desire) infrastructure to run
Apache Cassandra at petabyte scale

68. Thank You.