Given at RICON 2012

1. Getting started with
Riak in the Cloud

2. I work with I work with
Code! Data!
Michael Brodhead Ines Sombra
@mojinations @RandomMood

4. Agenda
Application
layer
Database
layer

5. A tale of 2 apps
• One brand
new (no code
written)
• One in production
running MySQL

9. Multiple
Regions

10. Multiple
Clouds

11. 11

12. The Libraries
riak-client
• simple Ruby
access to Riak's
HTTP &
protocol buffer
APIs

13. The Libraries
Ripple
• Higher level,
ORM interface
• ActiveModel
based buffer APIs

14. Simplicity

16. The Libraries
dm-riak
• Already using
DataMapper
• Hoping for trivial
switchover

17. The Libraries
Riak Shim
• database.yml
• Bucket naming
• Hides JSON
• 2i

18. Challenges
• Unit tests
• Data migration
• Deployment

19. Testing

20. Testing

21. Deployment

22. Use your app logic

23. How do we
get here?

24. “We need a
Riak cluster”
(@mojinations circa 3 months ago)
... but we have no
hardware

25. Let’s do it!

26. Why Riak Product?
• Linearly Scalable
• Fault-tolerant
• Ops friendly
• Product lacks
HA options

27. Riak @ Engine Yard
• We’ll use it first
• No custom Chef
• Provisioning,
configuration,
monitoring, &
backups

28. Your Choices
• Number of nodes
• Instance type
You are trying
to create a Riak
cluster. Do you
need any help?
• Data location
• Backend type

30. AWS Ops
The Riak on
AWS slide

31. Instances & I/O
• Riak’s I/O patterns
• Small blobs from many
places on disk
• EBS best at bulk reads &
writes

32. Instance types
• m1.large & m1.xlarge
• EBS-optimized NEW!
• Consider PIOPS NEW!
• High I/O Quadruple XL
(hi1.4xlarge) - SSDs! NEW!

33. Prepare
Yourself
More instances to compensate
for the performance variability
of virtualized resources

34. Cluster Sizes
• 5 nodes or more
Best performance
& growth
• Scales linearly
• Don’t be stingy

35. Ring Size
• Change default value!
• Cannot be reset
• Ring size: 256
8/64 vnodes per node
• Don’t go above 512

36. Provisioning
Riak Cluster
App MyRiakCluster: node-0
haproxy
HTTP
or MyRiakCluster: node-1
Protobuff
App
haproxy
MyRiakCluster: node-2
App
MyRiakCluster: node-3
haproxy
MyRiakCluster: node-4

37. Disable Swap
• Riak process pages should
not hit swap
• Swap usage
can result in
unresponsive server
• Let the kernel kill it

38. Mount & Scheduler
• mount with noatime flag
• Use deadline scheduler
on EBS
cat /sys/block/xvdj1/queue/scheduler
noop [deadline] cfq
echo deadline > /sys/block/xvdj1/
queue/scheduler

39. Linux Filesystems
• xfs, ext3
• ext4
• barrier = 0
• data = writeback
• ZFS not recommended

40. Backend choices
• Bitcask*
• help mitigate EBS
• LevelDB Can be
changed
• Memory and
optimized!

41. Kernel Optimizations
• /etc/sysctl.conf
vm.swappiness = 0
net.core.somaxconn= 4000
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 40000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_tw_reuse = 1
http://wiki.basho.com/Linux-Performance-Tuning.html

42. Monitoring
• stats or riak-admin status
• Collectd & Nagios
• Riaknostic
• Riak Control (maybe soon)

43. Backups
• Backup both the ring and
data directories
• Integrate with EY tools
• scp or rsync data directory
• EBS snapshots

44. Scaling Riak
• Vertical
riak-admin cluster replace <old> <new>
• Horizontal
• reduces node load
• batch is better

45. Benchmarks

46. Tests
• 5-node cluster / 5 hours
• 50 concurrent connections
• gets 10 /puts 5 /deletes 1
• Bitcask backend
• Protobuffs protocol

47. Medium - PIOPS 500

48. Medium - PIOPS 1K

49. Medium - Ephemeral
FAIL!

50. XL - EBS

51. XL -PIOPS 500

52. 4XL - EBS

53. 4XL - PIOPS 1k

54. What’s next?