O'Reilly MySQL Conference 2011, Santa Clara

1. Redis to the Rescue?
O’Reilly MySQL Conference
2011-04-13

2. Who?
• Tim Lossen / @tlossen
• Berlin, Germany
• backend developer at wooga

4. Redis Intro
Case 1: Monster World
Case 2: Happy Hospital
Discussion

5. Redis Intro
Case 1: Monster World
Case 2: Happy Hospital
Discussion

6. What?
• key-value-store
• in-memory database
• “data structure server”

8. Data Types
• strings (integers)

9. Data Types
• strings (integers)
• lists
• hashes

10. Data Types
• strings (integers)
• lists
• hashes
• sets
• sorted sets

11. flickr.com/photos/atzu/2645776918

12. Performance
• same for reads / writes

13. Performance
• same for reads / writes
• 50 K ops/second
- regular notebook, EC2 instance

14. Performance
• same for reads / writes
• 50 K ops/second
- regular notebook, EC2 instance
• 200 K ops/second
- intel core i7 X980 (3.33 GHz)

15. Durability
• snapshots
• append-only log

16. Other Features
• master-slave replication
• virtual memory
• ...

17. Redis Intro
Case 1: Monster World
Case 2: Happy Hospital
Discussion

19. Daily Active Users
April May June July Aug Sept Oct 2010

20. Daily Active Users
April May June July Aug Sept Oct 2010

21. Challenge
• traffic growing rapidly

22. Challenge
• traffic growing rapidly
• bottleneck: write throughput
- EBS volumes pretty slow

23. Challenge
• traffic growing rapidly
• bottleneck: write throughput
- EBS volumes pretty slow
• MySQL already sharded
- 4 x 2 = 8 shards

24. Idea
• move write-itensive data to Redis

25. Idea
• move write-itensive data to Redis
• first candidate: inventory
- integer fields
- frequently changing

27. Solution
• inventory = Redis hash
- atomic increment / decrement !

28. Solution
• inventory = Redis hash
- atomic increment / decrement !
• on-demand migration of users
- with batch roll-up

29. Results
• quick win
- implemented in 2 weeks
- 10% less load on MySQL servers

30. Results
• quick win
- implemented in 2 weeks
- 10% less load on MySQL servers
• decision: move over more data

31. But ...
• “honeymoon soon over”

32. But ...
• “honeymoon soon over”
• growing memory usage
(fragmentation)
- servers need periodic “refresh”
- replication dance

33. Current Status
• hybrid setup
- 4 MySQL master-slave pairs
- 4 Redis master-slave pairs

34. Current Status
• hybrid setup
- 4 MySQL master-slave pairs
- 4 Redis master-slave pairs
• evaluating other alternatives
- Riak, Membase

35. Redis Intro
Case 1: Monster World
Case 2: Happy Hospital
Discussion

37. Challenge
• expected peak load:
- 16000 concurrent users
- 4000 requests/second
- mostly writes

38. “Memory is the new Disk,
Disk is the new Tape.”
⎯ Jim Gray

39. Idea
• use Redis as main database
- excellent (write) performance
- virtual memory for capacity

40. Idea
• use Redis as main database
- excellent (write) performance
- virtual memory for capacity
• no sharding = simple operations

41. Data Model
• user = single Redis hash
- each entity stored in hash field
(serialized to JSON)

42. Data Model
• user = single Redis hash
- each entity stored in hash field
(serialized to JSON)
• custom Ruby mapping layer
(“Remodel”)

44. {“level”: 4,
1220032045 u1
“xp”: 241}
u1_pets [“p7”, “p8”]
{“pet_type”:
p7
“Cat”}
{“pet_type”:
p8
“Dog”}
{“level”: 1,
1234599660 u1
“xp”: 22}
u1_pets [“p3”]
... ...

45. {“level”: 4,
1220032045 u1
“xp”: 241}
u1_pets [“p7”, “p8”]
{“pet_type”:
p7
“Cat”}
{“pet_type”:
p8
“Dog”}
{“level”: 1,
1234599660 u1
“xp”: 22}
u1_pets [“p3”]
... ...

46. {“level”: 4,
1220032045 u1
“xp”: 241}
u1_pets [“p7”, “p8”]
{“pet_type”:
p7
“Cat”}
{“pet_type”:
p8
“Dog”}
{“level”: 1,
1234599660 u1
“xp”: 22}
u1_pets [“p3”]
... ...

47. {“level”: 4,
1220032045 u1
“xp”: 241}
u1_pets [“p7”, “p8”]
{“pet_type”:
p7
“Cat”}
{“pet_type”:
p8
“Dog”}
{“level”: 1,
1234599660 u1
“xp”: 22}
u1_pets [“p3”]
... ...

48. {“level”: 4,
1220032045 u1
“xp”: 241}
u1_pets [“p7”, “p8”]
{“pet_type”:
p7
“Cat”}
{“pet_type”:
p8
“Dog”}
{“level”: 1,
1234599660 u1
“xp”: 22}
u1_pets [“p3”]
... ...

49. {“level”: 4,
1220032045 u1
“xp”: 241}
u1_pets [“p7”, “p8”]
{“pet_type”:
p7
“Cat”}
{“pet_type”:
p8
“Dog”}
{“level”: 1,
1234599660 u1
“xp”: 22}
u1_pets [“p3”]
... ...

50. Virtual Memory
• server: 24 GB RAM, 500 GB disk
- can only keep “hot data” in RAM

51. Virtual Memory
• server: 24 GB RAM, 500 GB disk
- can only keep “hot data” in RAM
• 380 GB swap file
- 50 mio. pages, 8 KB each

52. Dec 2010: Crisis
• memory usage growing fast

53. Dec 2010: Crisis
• memory usage growing fast
• cannot take snapshots any more
- cannot start new slaves

54. Dec 2010: Crisis
• memory usage growing fast
• cannot take snapshots any more
- cannot start new slaves
• random crashes

55. Analysis
• Redis virtual memory not
compatible with:
- persistence
- replication

56. Analysis
• Redis virtual memory not
compatible with:
- persistence
- replication
• need to implement our own!

57. Workaround
• “dumper” process
- tracks active users
- every 10 minutes, writes them
into YAML file

58. ruby redis disk

59. ruby redis disk

60. ruby redis disk

61. ruby redis disk

62. ruby redis disk

63. Workaround
• in case of Redis crash
- start with empty database
- restore users on demand from
YAML files

64. Real Solution
• Redis “diskstore”
- keeps all data on disk
- swaps data into memory as
needed

65. Real Solution
• Redis “diskstore”
- keeps all data on disk
- swaps data into memory as
needed
• under development (expected Q2)

67. Results
• average response time: 10 ms

68. Results
• average response time: 10 ms
• peak traffic:
- 1500 requests/second
- 15000 Redis ops/second

69. Current Status
• very happy with setup
- simple, robust, fast
- easy to operate
• still lots of spare capacity

70. Redis Intro
Case 1: Monster World
Case 2: Happy Hospital
Discussion

71. Advantages
• order-of-magnitude performance
improvement
- removes main bottleneck
- enables simple architecture

72. Disadvantages
• main challenge: durability
- diskstore very promising

73. Disadvantages
• main challenge: durability
- diskstore very promising
• no ad-hoc queries
- think hard about data model
- hybrid approach?

74. Conclusion
• Ruby + Redis = killer combo

75. Q&A

76. redis.io
github.com/tlossen/remodel
wooga.com/jobs