Here are the slides from Joseph Lynch's PuppetConf 2016 presentation called Automating Datastore Fleets with Puppet. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

1. Joey Lynch
jlynch@yelp.com
Data Shepherd
Automating Datastore Fleets with
Puppet

2. Yelp’s Mission
Connecting people with great
local businesses.

3. Yelp Stats
As of Q2 2016
92M 3272%108M

4. jlynch Stats
As of Q2 2016
Yelp 4+ years
Datastore
Whisperer
Distributed Systems Addict

5. Yelp Polyglot Stack
Presentation Logic Data

6. But Why?

7. Logic
Web Search Ads API Logging Svc 5 Platform
Data
MySQL Lucene Cass ... Kafka DB 5 Zookeeper
...
...

8. Logic
Web Search Ads API Logging Svc 5 Platform
Data
MySQL Lucene Cass ... Kafka DB 5 Zookeeper
...
...
It’s Complicated

9. MUST support
● Different datastores
● Different configurations
● Cloud and datacenter ready
● Ready for production
● Cope with Data Gravity
Yelp’s Datastore Problem

10. Yelp’s Datastore Solution

11. Different Datastores

12. Cluster Modules

13. Wrap the vendor
modules
Manage cluster
wide config

14. Different Configurations

15. Configure
● Functions help
● Hierarchical config
is even better

16. Configure - Hierarchy

17. Configure - Hierarchy

18. Configure - Hierarchy

19. Configure - Adding a Cluster

20. Configure - Adding a Cluster

21. Configure - Flexible

22. Configure - Flexible

23. Configure - Flexible

24. Functions
● No hard coding!
● Changes with infra
● Sometimes as
easy as
$::processorcount

25. Cloud and Datacenter Ready

26. Datacenter

27. Datacenter

28. Cloud

29. Cloud

30. ENC is very nice

31. ENC is very nice

32. Production Ready

33. Using Puppet to Manage Datastores
● About 100 clusters
● Over 500 datastore nodes
● Puppet runs take ~5 minutes
● Run Puppet every hour, CI/CD
○ Automation runs via separate scheduler

40. Monitor

41. Monitor Clusters

42. Monitor Clusters
https://engineeringblog.yelp.com/2016/06/monitoring-cassandra-at-scale.html

43. Monitor Clusters
https://engineeringblog.yelp.com/2016/06/monitoring-cassandra-at-scale.html
Totally fine
maybe
BAD

44. cron

45. KISS

47. DatabaseClient
configure_nerveconfigure_synapse
JSON files hacheck
ZookeeperSynapse Nerve
Healthchecks
ConfigurationConfiguration
Configuration
Healthchecks
Registration
Service Discovery

48. Service Discovery

49. Service Discovery
DatabaseClient
configure_nerveconfigure_synapse
JSON files hacheck
ZookeeperSynapse Nerve
Healthchecks
ConfigurationConfiguration
Configuration
Healthchecks
Registration

50. Service Discovery

51. Data Gravity

53. Autoscale!

54. Autoscale!
● Debian
package
● Cron

55. Backup!
● Another Debian
package
● Cron

56. Recap

57. Using Puppet to Manage Datastores

58. Using Puppet to Manage Datastores
cron
MonitorDiscoverAutomate
Puppet enables automation

59. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp

60. ● Server rack, unknown author
http://www.clipartkid.com/server-rack-cabinet-clip-art-at-clker-com-vector-clip-art-online-IPTtWc-clipart/
● Devops venn diagram, Kharnagy
https://commons.wikimedia.org/wiki/File:Devops.svg
● Steve Jobs, waldryno
https://pixabay.com/en/steve-jobs-technology-illustration-1249665/
Image Citations