In this talk, we will cover best practices in running RabbitMQ, things you should avoid doing, lesser known features, recent operations improvements and a bit of what's ahead. Given in October, 2015.

1. RabbitMQ Operations

2. About me

3. About me
• RabbitMQ staff engineer at Pivotal

4. About me
• RabbitMQ staff engineer at Pivotal
• @michaelklishin just about everywhere

6. About this talk

7. About this talk
• Brain dump from years of answering questions

8. About this talk
• Brain dump from years of answering questions
• Focusses on the most recent release (3.5.6)

10. Provisioning

11. Provisioning
• Be aware of mirrors: GitHub, Bintray, …

12. Provisioning
• Be aware of mirrors: GitHub, Bintray, …
• Looking into community-hosted mirrors

13. Provisioning
• Be aware of mirrors: GitHub, Bintray, …
• Looking into community-hosted mirrors
• Use packages + Chef/Puppet/…

15. OS resources

16. OS resources
• Modern Linux defaults are absolutely inadequate
for servers

17. ulimit -n default: 1024

19. Set ulimit -n and fs.file-max to 500K
and forget about it

20. TCP keepalive timeout:
from 11 minutes to over 2
hours by default

21. net.ipv4.tcp_keepalive_time = 6
net.ipv4.tcp_keepalive_intvl = 3
net.ipv4.tcp_keepalive_probes = 3

22. enable client heartbeats, e.g. with an
interval of 6-12 seconds

23. OS resources
• Modern Linux defaults are absolutely inadequate
for servers
• Tuning for throughput vs. high number of
concurrent connections

24. Throughput: larger TCP buffers

25. net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

26. rabbit.hipe_compile = true
(only on Erlang 17.x or 18.x)

27. Concurrent connections: smaller TCP
buffers, low tcp_fin_timeout,
tcp_tw_reuse = 1, …

28. rabbit.tcp_listen_options.sndbuf
rabbit.tcp_listen_options.recbuf
rabbit.tcp_listen_options.backlog

29. Reduce per connection
RAM use by 10x
rabbit.tcp_listen_options.sndbuf = 16384
rabbit.tcp_listen_options.recbuf = 16384

31. Reduce per connection
RAM use by 10x
Throughput drops by a comparable amount

32. net.ipv4.tcp_fin_timeout = 5

33. net.ipv4.tcp_tw_reuse = 1

34. Careful with tcp_tw_reuse
behind NAT*
* http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html

35. net.core.somaxconn = 4096

36. http://www.rabbitmq.com/networking.html

38. Disk space

39. Disk space
• Pay attention to what partition /var/lib ends up on

40. Disk space
• Pay attention to what partition /var/lib ends up on
• Transient messages can be paged to disk

41. Disk space
• Pay attention to what partition /var/lib ends up on
• Transient messages can be paged to disk
• RabbitMQ’s disk monitor isn’t supported on all
platforms

43. RAM usage

44. RAM usage
• rabbit.vm_memory_high_watermark

45. RAM usage
• rabbit.vm_memory_high_watermark
• rabbit.vm_memory_high_watermark_paging_ratio

46. rabbitmqctl status
rabbitmqctl report

47. RAM usage
• rabbit.vm_memory_high_watermark
• rabbit.vm_memory_high_watermark_paging_ratio
• Significant paging efficiency improvements in
3.5.5-3.5.6

48. RAM usage
• rabbit.vm_memory_high_watermark
• rabbit.vm_memory_high_watermark_paging_ratio
• Significant paging efficiency improvements in
3.5.5-3.5.6
• Disable rabbit.fhc_read_buffering (3.5.6+)

49. rabbitmqctl eval
‘file_handle_cache:clear_read_cache().’

50. recon

51. Ability to set VM RAM watermark as absolute
value is coming in 3.6

53. Stats collector falls behind

54. Stats collector falls behind
• Management DB stats collector can get
overwhelmed

55. Stats collector falls behind
• Management DB stats collector can get
overwhelmed
• Key symptom: disproportionally higher RAM use
on the node that hosts management DB

56. rabbitmqctl eval 'P = whereis(rabbit_mgmt_db), erlang:process_info(P).'

57. [{registered_name,rabbit_mgmt_db},
{current_function,{erlang,hibernate,3}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<5477.358.0>]},
{dictionary,[{'$ancestors',[<5477.358.0>,rabbit_mgmt_sup,rabbit_mgmt_sup_sup,
<5477.338.0>]},
{'$initial_call',{gen,init_it,7}}]},
{trap_exit,false},
{error_handler,error_handler},
{priority,high},
{group_leader,<5477.337.0>},
{total_heap_size,167},
{heap_size,167},
{stack_size,0},
{reductions,318},
{garbage_collection,[{min_bin_vheap_size,46422},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,0}]},
{suspending,[]}]

58. rabbit.collect_statistics_interval = 30000

59. rabbitmq_management.rates_mode = none

60. rabbitmqctl eval 'P = whereis(rabbit_mgmt_db), erlang:exit(P, please_crash).'

61. Parallel stats collector is coming in 3.7

63. Cluster formation

64. Cluster formation
• Node restart order dependency

65. Cluster formation
• Node restart order dependency
• github.com/rabbitmq/rabbitmq-clusterer

66. Cluster formation
• Node restart order dependency
• github.com/rabbitmq/rabbitmq-clusterer
• github.com/aweber/rabbitmq-autocluster

68. Backups

69. How do I back up?
• cp $RABBITMQ_MNESIA_DIR + tar

70. How do I back up?
• cp $RABBITMQ_MNESIA_DIR + tar
• Replicate everything off-site with exchange
federation + set message TTL via a policy

72. Hostname changes

73. rabbitmqctl rename_cluster_node [old name] [new name]

75. Network partition handling

76. Network partition handling
• When in doubt, use “autoheal”

77. Network partition handling
• When in doubt, use “autoheal”
• “Merge” is coming but has very real downsides,
too

79. Misc

80. Misc
• Don’t use default vhost and/or credentials

81. Misc
• Don’t use default vhost and/or credentials
• Don’t use 32-bit Erlang

82. Misc
• Don’t use default vhost and/or credentials
• Don’t use 32-bit Erlang
• Use reasonably up-to-date releases

83. Misc
• Don’t use default vhost and/or credentials
• Don’t use 32-bit Erlang
• Use reasonably up-to-date releases
• Participate in rabbitmq-users

84. Misc
• OCF resource template from Fuel (by Mirantis)

85. Misc
• OCF resource template from Fuel (by Mirantis)
• Use TLS

87. Coming in 3.6

88. Coming in 3.6
• In process file buffering disabled by default

89. Coming in 3.6
• In process file buffering disabled by default
• Queue master to node distribution strategies

90. Coming in 3.6
• In process file buffering disabled by default
• Queue master to node distribution strategies
• SHA-256 (or 512) for password hashing

91. Coming in 3.6
• In process file buffering disabled by default
• Queue master to node distribution strategies
• SHA-256 (or 512) for password hashing
• More responsive management UI with
pagination

92. Coming in 3.6
• In process file buffering disabled by default
• Queue master to node distribution strategies
• SHA-256 (or 512) for password hashing
• More responsive management UI with
pagination
• Streaming rabbitmqctl

93. Coming past 3.6

94. Coming past 3.6
• Pluggable cluster formation (à la ElasticSearch)

95. Coming past 3.6
• Pluggable cluster formation (à la ElasticSearch)
• On disk data recovery tools

96. Coming past 3.6
• Pluggable cluster formation (à la ElasticSearch)
• On disk data recovery tools
• Better CLI tools

97. Coming past 3.6
• Pluggable cluster formation (à la ElasticSearch)
• On disk data recovery tools
• Better CLI tools
• Easier off-site replication

98. Coming past 3.6
• Pluggable cluster formation (à la ElasticSearch)
• On disk data recovery tools
• Better CLI tools
• Easier off-site replication
• “Merge” partition handling strategy (no earlier
than 3.8)

100. Thank you

101. Thank you
• @michaelklishin
• github.com/michaelklishin
• rabbitmq-users
• Our team is hiring!