This document discusses Noah, a system for orchestration and coordination of infrastructure components like configuration management tools, applications, servers, and services. Noah uses a hierarchical data model and API to represent these components as objects that can be manipulated and watched. It aims to address the need for orchestration beyond single nodes and provide synchronization between different aspects of infrastructure.
25. “ I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?”
26. “ I think..and I don't know if anyone would agree, that configuration management is a solved problem at this point, right?” WTF?
27. "The point I want to make..Configuration management is not a solved problem...and it's dangerous to make the mistake to think that the way we do things now is the best way to do them..." - Andrew Clay Shafer
28. “ what I was attempting to say ... is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are 'How do I think beyond the single node where this tool is running?'”
29. “ what I was attempting to say (epic fail, I might add) is that the current crop of configuration management tools have reached a usable point where they do enough (for now). What we’re seeing as questions now are ' How do I think beyond the single node where this tool is running? '”
30.
31. Provide mechanisms for coordination between applications, nodes, services, configuration management and other infrastructure aspects
32. Example Use Case 1) Capacity Reached. Tell Noah 2) Noah tells provisioning system 3) Capacity allocated. Tell Noah 4) Noah triggers Puppet on LB 5) Noah triggers Puppet on Nagios Noah Nagios Puppet (app) Load Balancer
34. "ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications."
In the beginning we did things manually. This was generally regarded as a very bad idea.
We took notes! (Yes this is on an actual server right now as we speak)
We wrote handbooks!
Then we started writing Shell scripts!
This script is actually broken.
Then we got smart!
In the general scheme of things, we've evolved. We treat our infrastructure as code. We've adopted behaviors and practices that developers have been doing for years. We're green, ship it!
At DevOpsDay Mt. View, a revelation was made.
The response was generally overwhelmingly supportive....
Ideas were “refined”
This is the heart of “orchestration”. This is the problem space that needs addressing right now.
Orchestraton means many things however. Luke mentioned “Command and Control” for instance. Impedence mismatch – mention Capistrano and boatload of “deploy” tools. Something is missing. Mechanisms – traditional orchestration
Just a quick example use case to demonstrate “orchestration”
Znodes are easily grokable. We get “paths”. Programatically friendly. Watches are cool. Downsides:
Have always needed something LIKE ZK. Haven't always had the “luxury” of language bindings. Haven't always needed the full feature set. Another unofficial goal is that it work in a “disconnected” world.
Going over the basics of interacting with Noah. All of this is on the wiki.
Host – node, server Service – http/https/ftp OSI L3 analogy. An open port. Host/Service inspired by Nagios Application – tomcat, apache, rails app. OSI L7 analogy Configuration – httpd.conf, server.xml, database.yml
Ephemerals – Arbitrary blob of information Tag – tags...not much to say here Links – similar to symlinks. Create a custom namespace Watches – Async pluggable callbacks
Explain the general idea behind programatic friendly paths. Operate with standard HTTP verbs – GET, PUT, DELETE
Explain the general idea behind programmatic friendly paths.
Common fields in all GETs
This is a GET to /hosts/
Remember that services must be bound to hosts.
Note the asterix here
JSON version is default in most cases: Accept: application/json to ensure this representation application/octet will ensure that you get a proper mime typed version.
Reminder of the paths
Note that services has a requirement on hostname in the path.
This is what you PUT to each path in previous slide Maybe have to jump back and forth.
Same paths as put. No payload required.
Explain a bit how ZK watches work Mention race condition.
Made up of endpoints and patterns. Persistence will eventually be tunable Supersets If you declare a watch for endpoint on: /hosts You can't declare a watch for endpoint on: /hosts/foo_host Subsets are the reverse
This creates a watch for anything under /applications. When something “changes” under /applications, the change will be send to the endpoint.
Again, watches have the same common fields
Here we're going to add a tag to that new application
Note that every example listed does not actually exist YET
Creator unrelated to the reciever is important. Allows proxy management You can write your own at the cost of fragility
James wrote these. Need to be updated to newest Noah API. Mention that this was supposed to be done before this presentation but there was some “confusion” about the schedule - Functions pull data from/put in Noah - noah_data fact pulls configurations from Noah Cover the mappings
Taking Noah out of the mix for a moment, what's a traditional workflow for managing applications with Puppet?
Problems I've seen Different repos Dev interaction What happens when we add a new setting? How many people run puppet in daemon mode or via cron as opposed to on demand? Rampination.
Adding Noah in the mix Looks a bit more complicated. Noah becomes your System of Record Talk about Volatile settings
This is where I disagree with Luke a little bit. There are valid use cases for “dynamic” configuration. JMX exists.
Explain CI
We currently do this for deploys at VA using a curl script in a bash step You can either trigger the deploy itself or wrap it around a puppet run. Both can be done by CI. Upshot is that you get more templated CI jobs.
This is similar to the original example use case.
The nagios event handler, again, can be implemented as a curl script. If there's a watch attached, you can trigger any number of actions. You can have endpoints in your applications that make it dynamically respond.
This is an immediate win
Ghetto job dependencies Easier than dealing with mysql command lines and escaping.
ZK and Doozer are a bit more complex. Require persistent connections. NoSQL and Nesoi are nice because they're http and json!
![.plan ,[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)