1. Key-Value-Stores:
The Key to Scaling?
Tim Lossen

2. Who?
• @tlossen
• backend developer
- Ruby, Rails, Sinatra ...
• passionate about technology

5. Problem

6. Challenge
• backend for facebook game

7. Challenge
• backend for facebook game
• expected load:
- 1 mio. daily active users
- 20 mio. total users
- 100 KB data per user

8. Challenge
• expected peak traffic:
- 10.000 concurrent users
- 200.000 requests / minute

9. Challenge
• expected peak traffic:
- 10.000 concurrent users
- 200.000 requests / minute
• write-heavy workload

10. Wanted
• scalable database
• with high throughput
- especially for writes

11. Options
• relational database
- with sharding

12. Options
• relational database
- with sharding
• nosql database
- key-value-store
- document db
- graph db

13. Options
• relational database
- with sharding
• nosql database
- key-value-store
- document db
- graph db

14. Options
• relational database
- with sharding
• nosql database
- key-value-store
- document db
- graph db

15. Options
• relational database
- with sharding
• nosql database
- key-value-store
- document db
- graph db

16. Shortlist
• Cassandra
• Redis
• Membase

17. Cassandra

18. Facts
• written in Java
- 55.000 lines of code
• Thrift API
- clients for Java, Ruby, Python ...

19. History
• originally developed by Facebook
- in production for “Inbox Search”
• later open-sourced
- top-level Apache project

20. Features
• high availability
- no single point of failure
• incremental scalability
• eventual consistency

21. Architecture
• Dynamo-like hash ring
- partitioning + replication
- all nodes are equal

22. Hash Ring

23. Architecture
• Dynamo-like hash ring
- partitioning + replication
- all nodes are equal
• Bigtable data model
- column families
- supercolumns

25. “Cassandra aims to run
on an in"astructure
of hundreds of nodes.”

26. Redis

27. Facts
• written in C
- 13.000 lines of code
• socket API
- redis-cli
- client libs for all major languages

28. Features
• high read & write throughput
- 50.000 to 100.000 ops / second

29. Features
• high read & write throughput
- 50.000 to 100.000 ops / second
• interesting data structures
- lists, hashes, (sorted) sets
- atomic operations

30. Features
• high read & write throughput
- 50.000 to 100.000 ops / second
• interesting data structures
- lists, hashes, (sorted) sets
- atomic operations
• strong consistency

31. Architecture
• in-memory database
- append-only log on disk
- virtual memory

32. Architecture
• in-memory database
- append-only log on disk
- virtual memory
• single instance
- master-slave replication
- clustering is on roadmap

33. “Memory is the new disk,
disk is the new tape.”
— Jim Gray

34. Membase

35. Facts
• written in C and Erlang
• API-compatible to Memcached
- same protocol
• client libs for all major languages

36. History
• developed by NorthScale & Zynga
- used in production (Farmville)
• released in June 2010
- Apache 2.0 License

37. Features
• “Memcached with persistence”
- extremely fast
- throughput scales linearly

38. Features
• “Memcached with persistence”
- extremely fast
- throughput scales linearly
• automatic data placement
- memory, ssd, disk

39. Features
• “Memcached with persistence”
- extremely fast
- throughput scales linearly
• automatic data placement
- memory, ssd, disk
• configurable replica count

40. Architecture
• cluster
- all nodes are alike
- one elected as “coordinator”

41. Architecture
• cluster
- all nodes are alike
- one elected as “coordinator”
• each node is master for part of key
space
- handles all reads & writes

42. Mapping Scheme

43. “simple, fast, elastic”

44. Solution

45. Which one
would you pick?

46. Decision
• Cassandra ?

47. Decision
• Cassandra ?
- too big, too complicated

48. Decision
• Cassandra ?
- too big, too complicated
• Membase ?

49. Decision
• Cassandra ?
- too big, too complicated
• Membase ?
- not yet available (then)

50. Decision
• Cassandra ?
- too big, too complicated
• Membase ?
- not yet available (then)
• Redis !

51. Motivation
• keep operations simple
• use as few machines as possible
- ideally, only one

52. Design
• two machines (+ load balancer)
- Redis master handles all reads /
writes
- Redis slave as hot standby

53. Design
• two machines (+ load balancer)
- Redis master handles all reads /
writes
- Redis slave as hot standby
- both machines used as app servers

54. Design
• two machines (+ load balancer)
- Redis master handles all reads /
writes
- Redis slave as hot standby
- both machines used as app servers
• dedicated hardware

55. Data model
• one Redis hash per user
- key: facebook id
• store data as serialized JSON
- booleans, strings, numbers,
timestamps ...

56. Advantages
• turns Redis into “document db”
- efficient to swap user data in / out
- atomic ops on parts
• easy to dump / restore user data

57. Capacity
• 4 GB memory for 20 mio. integer keys
- keys always stay in memory!

58. Capacity
• 4 GB memory for 20 mio. integer keys
- keys always stay in memory!
• 2 GB memory for 10.000 user hashes
- others can be swapped out

59. Capacity
• 4 GB memory for 20 mio. integer keys
- keys always stay in memory!
• 2 GB memory for 10.000 user hashes
- others can be swapped out
• 3.6 mio. ops / minute
- sufficient for 200.000 requests

60. Status
• game was launched in august
- currently still in beta

61. Status
• game was launched in august
- currently still in beta
• expect to reach 1 mio. daily active users
in Q1/2011

62. Status
• game was launched in august
- currently still in beta
• expect to reach 1 mio. daily active users
in Q1/2011
• will try to stick to 2 or 3 machines
- possibly bigger / faster ones

63. Conclusions
• use the right tool for the job

64. Conclusions
• use the right tool for the job
• keep it simple
- avoid sharding, if possible

65. Conclusions
• use the right tool for the job
• keep it simple
- avoid sharding, if possible
• don’t scale out too early
- but have a viable “plan b”

66. Conclusions
• use the right tool for the job
• keep it simple
- avoid sharding, if possible
• don’t scale out too early
- but have a viable “plan b”
• use dedicated hardware

67. Q&A

68. Links
• cassandra.apache.org
• redis.io
• membase.org
• tim.lossen.de