Live traffic capture and replay in cassandra 4.0

Live traffic capture and replay
in Apache Cassandra 4.0
Vinay Chella

© DataStax, All Rights Reserved.ConfidentialConfidential© DataStax, All Rights Reserved.
Who has?
● Wondered what kinds of queries are hitting your
database?
● Wanted to duplicate queries to test clusters?
● Run tcpdump and run your machine out of disk
space?

Who Am I?
Vinay Chella
Apache Cassandra Committer
Cloud Data Architect
Cloud Database Engineering @ Netflix

Agenda
● History of Traffic capture
● Why Capture it
● Traffic capture in Cassandra - Today
● Traffic capture in Cassandra - Apache Cassandra 4.0
● Full Query Logging
○ Bin Log
● Demo
● BinLog applications in Cassandra

Database
tracers
Database native traffic
capture tools
Flood of
n/w tools
Many other web
server traffic capture
and replay tools
Early
90s
Late
90s
Early
2000s
Late
2000s
ethereal
Packet analyzer tool -
Wireshark
etherfind
Network packet capture
tool - tcpdump
History of traffic capture

Why capture traffic?
● Testing
● Debugging
● Shadowing

Why capture the traffic? - Testing
“Systems behave differently depending on environment and
traffic patterns. Since the behavior of utilization can change
at any time, sampling real traffic is the only way to reliably
capture the request path.”
- Principles of Chaos Engineering

Problem: Databases upgrades are hard
Reason: Databases does not have lot of test coverage
Problem: Trying to enumerate all the possible combinations of your
prod environment is quite challenging
Result: 100% test coverage on your test env is hard
Problem: Is my database ready for production traffic? How about load
testing it
Reality: Synthetic testing is hard and impossible to get it right
Why capture the traffic? - Testing

● Debugging tricky production bugs
● Figuring out performance issues
● Auditing which microservices are accessing the database
Why capture the traffic? - Debugging

● Shadowing, Mirroring, dark launch, cloning, roman riding
● Minimal production impact during upgrades
● Test with actual data
● Load test and verify without impacting production
Why capture the traffic? - Shadowing

How should the traffic
capture be done?

Performance
Debugging Testing
Shadowing
04 03
0201
4 Major themes
It should make
testing much easier
It should allow me
shadow my traffic
It should help in
debugging
It should not slow
down the live traffic

What do we have in
Cassandra today?

● Existing tools
○ tcpdump
○ cqltrace
○ wireshark
○ Other packet capture mechanisms
Traffic capture in Cassandra

● Custom binary protocol
○ Analyzing custom protocol is cumbersome
○ Duplicating the traffic at network level is a hard problem
“Tracing should not impact system performance”
- Brendan Gregg
Performance is critical, you should run these tools without worrying about
performance
Traffic capture in Cassandra - Why is it hard?

● Full Query Logging capabilities
● Ability to
○ Log every incoming request
○ Reuse/ re-play every incoming request
○ Reuse the logged trace at any time
○ Easily start/stop tracing at any time
And importantly without impacting the performance
So, what do we need from Cassandra

Who is really good at high performance with reliability?
Banks?
Traders?
Hospitals?
Insurance companies?
Of course, it is your Traders, High Frequency Traders…..

● Chronicle Queues
● Memory mapped file for IPC
● No GC - Off heap
● High speed and low cost logging
● Ring buffer with weighted queue
Everything is a ring buffer!!!!
● Would `ring buffer` be sufficient?
● Maybe?
○ Debugging, tracing, testing
● Maybe not?
○ Auditing and compliance
○ Guaranteed traffic cloning
BinLog
It all started with Ariel’s work on CASSANDRA-13983 - FQL (Full Query Logging)

Query Logging
● Based on BinLog
● Logs every incoming CQL request
● Highly performant
● Binary format output
● fqltool to make it readable
● Droppable jars for custom loggers

● Protocol version
● Query time
● Query
● Query values
● Type
● single, batch etc.,
What does it log

Theme #1: Debugging

Theme #1: Debugging
Client
Cassandra Node
fql_logs
Cassandra process

Theme #1: Debugging
$ ./nodetool enablefullquerylog --path /logs/cassandra/fql
--blocking: If the queue is full whether to block producers or drop samples. Default true.
--max-log-size: How many bytes of log data to store before dropping segments. Default 16 gigabytes.
--max-queue-weight: Maximum number of bytes of query data to queue to disk before blocking or dropping samples. Default 256
megabytes.
--path: Path to store the full query log at.
-pp, --print-port: Operate in 4.0 mode with hosts disambiguated by port number
--roll-cycle: How often to roll the log file (MINUTELY, HOURLY, DAILY). Default HOURLY.
$ ./nodetool disablefullquerylog

How to configure

Sample logs
Type: single
Protocol version: 4
Query time: 1536871241290
Query: USE "dev1"
Values:
Type: single
Protocol version: 4
Query time: 1536871301568
Query: INSERT INTO emp(key , column1 , value ) VALUES ( 't1',1,'tvalue');
Values:
Type: single
Protocol version: 4
Query time: 1536871306065
Query: SELECT * from emp ;
Values:

Performance
Debugging Testing
Shadowing
04 03
0201
4 Major themes
It should make
testing much
easier
It should allow me
to shadow traffic
It should help in
debugging
It should not slow
down the live traffic

Theme #2: Testing

Theme #2: Testing
cass_abc : Node1
Client fql_logs
Cassandra process
cass_xyz : Node6
fqltool replay
Cassandra process
fql_replay_logs
fqltool compare

Theme #2: Testing
$ ./tools/bin/fqltool replay --target <HOST>:<PORT>
/logs/cassandra/fql --store-queries /logs/cassandra/fql_replay/
$ ./tools/bin/fqltool compare --queries
/logs/cassandra/fql_replay/ /logs/cassandra/fql

Theme #3: Shadowing
Client
cass_abc : Node1
fql_logs
Cassandra process
cass_xyz : Node6
fqltool replay
Cassandra process

Theme #3: Shadowing
$ ./nodetool enablefullquerylog --path /logs/cassandra/fql
$ ./tools/bin/fqltool replay --target <HOST>:<PORT>
/logs/cassandra/fql

Theme #4: Performance

Audit Logging - How to configure.

$ ./nodetool enableauditlog
$ ./nodetool disableauditlog
StorageService.java:5420 - AuditLog is enabled with logger: [BinAuditLogger], included_keyspaces:
[movies, thumbs_ratings], excluded_keyspaces: [star_ratings], included_categories: [DDL, DML, DCL],
excluded_categories: [QUERY],included_users: [prod_user1, prod_user2], excluded_users: [admin_user,
ops_user]
Audit Logging - Ease of use with NodeTool

● Debugging
○ nodetool enablefullquerylog
● Testing
○ /cassandra/tools/bin/fqltool replay
○ /cassandra/tools/bin/fqltool compare
● Shadowing
○ nodetool enablefullquerylog
○ /cassandra/tools/bin/fqltool replay
● Compliance
○ nodetool enableauditlog (CASSANDRA-12151)
Wrapping it together

Take away
Cassandra 4.0 ships with
out of box traffic capture, replay, compare features
without compromising on performance

FAQ
● When will C* 4.0 be released?
○ Testing and benchmarking is in progress
○ Track the progress and plan @
https://cwiki.apache.org/confluence/display/CASSANDRA/4.0+Quality%3A+Components+and+Test+Plans
● How do I get my hands on these features?
○ Build trunk - https://github.com/apache/cassandra
○ Step by step @ https://blog.pythian.com/how-to-build-your-very-own-cassandra-4-0-release/
● What else is coming to Cassandra 4.0?
○ Come attend below talks tomorrow
■ Dinesh’s talk @ 10:45 AM - Two Years in the Making: What’s New with Apache Cassandra 4.0?
■ My talk @ 1:10 PM - Safer Restarts, Faster Streaming, and Better Repair, Just a Glimpse of Cassandra 4.0

Live traffic capture and replay in cassandra 4.0

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Live traffic capture and replay in cassandra 4.0

Similar a Live traffic capture and replay in cassandra 4.0 (20)

Más de Vinay Kumar Chella

Más de Vinay Kumar Chella (9)

Último

Último (20)

Live traffic capture and replay in cassandra 4.0