This document discusses using Cassandra to store time-series metrics data. It describes how the schema was matched to storage by using a measurement column family with rows organized by metric ID and time. It also covers optimizing data expiration through techniques like TTL expiration, synchronized compactions, and leveraging immutable sstable modification times. Effective monitoring is emphasized as well, including dashboards to track the ring and using Cassandra log volumes to identify issues.

1. #CASSANDRA13
Time-Series Metrics with Cassandra
Mike Heffner

2. #CASSANDRA13
What we do.

3. #CASSANDRA13
October 2011
l Decision: All measurements in Cassandra
l Single EC2 Ring: 6 * m1.large
l Cassandra 0.8.x
l How does this work?

4. #CASSANDRA13
Today
l Multiple sharded rings
l ~250,000 writes / second
l EC2: m1.xlarge and m2.4xlarge
l Cassandra 1.1.x
l Read load: < 1%

5. #CASSANDRA13
Talk Highlights
l Matching Schema to Storage
l Optimally Expiring Data
l Monitor Everything

6. #CASSANDRA13
Matching Schema to Storage

7. #CASSANDRA13
What is a Measurement?
( Metric ID, Source )
(X, Y) => (Time stamp, Value)

8. #CASSANDRA13
Measurement CF
Example: Select measurements between times [T1, T2]:

9. #CASSANDRA13
Locating Rows
Let us calculate the maximum row size:
l 1 minute records
l 1 week TTL
l 7 days * 24 hours * 60 minutes => 10,080
l 3 Longs * 8 bytes * 10k => ~240KB (not bad)

10. #CASSANDRA13
Row Storage Over Time

11. #CASSANDRA13
Row Storage Over Time

12. #CASSANDRA13
Seek All The SStables

13. #CASSANDRA13
Examining CF SSTables
Metrics/metric_id_epochs_60 histograms
Offset SSTables
1 28821
2 58859
3 201198
4 178326
5 223016
6 154952
7 83289
8 21552
10 81104
1 2 3 4 5 6 7 8 10
nodetool cfhistograms Metrics metric_id_epochs_60

14. #CASSANDRA13
Splitting the Rows
mget(Rows: [12, EBase_30], [12, EBase_40], Columns: {31->45})
Retrieve Time Bases for Times 31->45 for metric ID 12:

15. #CASSANDRA13
Examining CF SSTables
Metrics/metric_id_epochs_60
Offset SSTables
1 28821
2 58859
3 201198
4 178326
5 223016
6 154952
7 83289
8 21552
10 81104
1 2 3 4 5 6 7 8 10
nodetool cfhistograms Metrics metric_id_epochs_60
Metrics/metric_id_epochs_60
Offset SSTables
1 3491820
2 5389762
3 4095760
4 1310741
5 9976 1 2 3 4 5 6 7 8 9 10
Before
After

16. #CASSANDRA13
/graph me

17. #CASSANDRA13
Optimally Expiring Data

18. #CASSANDRA13
TTL Expiration
l Churn of about 750GB / day
l 12 TB total
l 6% of data set
l gc_grace = 0
l STC

19. #CASSANDRA13
Synchronized Compactions

20. #CASSANDRA13

21. #CASSANDRA13
nodetool compact

22. #CASSANDRA13
* http://hight3ch.com/garbage-truck-crushing-a-car/

23. #CASSANDRA13
nodetool cleanup

24. #CASSANDRA13
Cleanup
l Not just for topology changes
l Tombstoned rows (not referenced)
l Rotated row keys decrease references
l Cons: Must process every sstable.

25. #CASSANDRA13
Immutable SStables

26. #CASSANDRA13
Leverage SStable Mod Time
l If now – mtime > TTL => all data is expired
l We can quickly eliminate entire sstables:
find -mtime +<TTL> -name *.db | xargs rm
l Fast and low overhead
l Cons: Rolling restart
26G 2013-05-17 09:44 Metrics-metrics_60-hf-7209-Data.db

27. #CASSANDRA13
nodetool setcompactionthreshold

28. #CASSANDRA13
Increasing minor compactions
l By default, STC requires a minimum of 4 ssts
l Leads to large non-compacted sstables
l Dropping to 2 can flatten the storage growth
nodetool setcompactionthreshold <ks> <cf> 2
l Cons: CPU/IO increase

29. #CASSANDRA13
Result

30. #CASSANDRA13
Effective Monitoring

31. #CASSANDRA13
Ring Dashboards

32. #CASSANDRA13
Disk Errors => Throw Away
l If you ever see this, replace!
end_request: I/O error, dev xvdb, sector 467940617
end_request: I/O error, dev xvdb, sector 467940617
l Mark node down, bootstrap new
l No metric for this?

33. #CASSANDRA13
Cassandra Log Volume
l Count log lines seen every 10 minutes
l Track over time
l Can identify:
- Unbalanced workloads
- Schema disagreements
- Phantom gossip nodes
- GC activity
l grep -v '.java' => exceptions

34. #CASSANDRA13
Q & A
Mike Heffner
/mheffner
/mheffner