An internet television service migrated from a system supporting 5k users to one supporting 5 million users using Apache Cassandra in a microservices architecture. Key lessons learned included putting state on clients or in Cassandra, not other systems; testing different data models and workloads; following DataStax documentation; avoiding Thrift; ensuring proper DevOps practices; and running multi-node Cassandra clusters. A live migration from MySQL to Cassandra involved double writing to make writes idempotent. Cassandra allows greater scale than other databases by requiring data denormalization and duplication.

1. ©2013 DataStax Confidential. Do not distribute without consent.
@chbatey
Christopher Batey
Use case & Lessons learned: Internet
Television

2. @chbatey
Who am I?
• Built a a lot of systems with Apache
Cassandra at Sky
• Work on a testing library for Cassandra
• Help out Cassandra users
• Twitter: @chbatey

3. @chbatey
Agenda
• Migrating a system designed for 5k users to 5million
users
- Many different uses of Cassandra
- One example data model
• Deployment
• Lessons learned
• A live migration (mysql to Cassandra)

4. @chbatey
Internet Television
• Functionality
- Manage users - millions of them
- Manage devices - even more
- Entitlements - Who can watch what?
- Product catalogs
- Event logging for billing and analytics

5. @chbatey
Internet Television
• Non-functional requirements
- Don’t kill other systems
- Very spiky - Game of Thrones, Champions Leage
- Increasing user base - doubling per year
- Multi DC an absolute must

6. @chbatey
5k Users

7. @chbatey
100k+ Users

8. @chbatey
Champions League match?

9. @chbatey
Uh oh :(

10. @chbatey
A pile of cats

11. @chbatey
The solution
Micro(ish)services +

12. @chbatey
Strangulation

13. @chbatey
The new

14. @chbatey
Audit log
• Very heavy write - many events per user interaction with
the service

15. @chbatey
Modelling the traditional way
CREATE TABLE customer_events(
customer_id text,
staff_name text,
time timeuuid,
event_type text,
store_name text,
PRIMARY KEY (customer_id));
CREATE TABLE store(
name text,
location text,
store_type text,
PRIMARY KEY (store_name));
CREATE TABLE staff(
name text,
favourite_colour text,
job_title text,
PRIMARY KEY (name));

16. @chbatey
Your model should look like your queries

17. Modelling in Cassandra
CREATE TABLE customer_events(
customer_id text,
staff_id text,
time timeuuid,
store_type text,
event_type text,
tags map<text, text>,
PRIMARY KEY ((customer_id), time));
Partition Key
Clustering Column(s)

18. @chbatey
Lesson #1: State kills scalability
• There are only two places to put state in a scalable
systems:
- The client
- Cassandra

19. @chbatey
Lesson #2: Test & measure
• Only you know your work loads
- Try different schemas, record the performance
• Cassandra stress is your friend

20. @chbatey
Lesson #3: Use the DataStax docs

21. @chbatey
Lesson #4: Say no to thrift
http://exponential.io/blog/2015/01/08/cassandra-terminology/

22. @chbatey
Lesson #5: DevOps
• Know the hardware you are deploying Cassandra on
• No network storage
• AWS => Ephemeral storage

23. @chbatey
Lesson #6: Run multi node clusters

24. @chbatey
Live migration (mysql —> Cassandra)
• Double write - make the writes idempotent

25. @chbatey
Summary
• Cassandra allows you to run at a scale not possible with
other DBs
• The cost? Denormalising + duplicating

26. @chbatey
Questions?
• Think of any questions later? @chbatey