DevOps is the new rage among system administrators, applying agile software development techniques to infrastructure configuration management. In the center of the DevOps movement is the open-source Chef tool, implemented in Ruby atop CouchDB. Unsatisfied with the performance of the open-source and/or hosted Chef server and needing better integration with our Python web application, we set out to build a new implementation in Python atop MongoDB. This talk will give you an overview of Chef, reasons for doing a new implementation, and lots of code examples of how we made it all work together to get a chef server that screams. This talk is updated with the latest version of MongoPyChef, ported to run on Pyramid and open sourced at https://github.com/rick446/MongoPyChef

1. Rick Copeland @rick446
Arborian Consulting, LLC

2.  Infrastructure as Code
 Resources & Providers
 Cookbooks, Recipes, Clients, and Nodes

3. CouchDB Solr
SolrIndexer
Chef Server
RabbitMQ
RabbitMQ
RabbitMQ
(API)
HTTP REST API
Chef Server
knife chef-client
(Web UI)

4. Build, register, and authenticate the node
Synchronize cookbooks
Build resource collection (run the recipes in order)
Configure node (“converge”)
Run notification handlers

5. CouchDB Solr
Ruby
SolrIndexer
Chef Server
RabbitMQ
RabbitMQ
RabbitMQ
(API)
Ruby
Ruby
HTTP REST API
Chef Server
knife chef-client
(Web UI)

6.  Chef assumes a bootstrapped node exists
 Chef doesn’t keep release notes
 Code and infrastructure are versioned
differently
 Solution: Web app to manage deployments &
generate release notes

7. MonQ
MongoDB
SolrIndexer
Python
Chef Server
(API + web) Solr
Ruby
HTTP REST API
knife chef-client

8.  Reduce # of processes & technologies
 Don’t know Ruby well
 Keep private keys out of the system
 Integrate with existing authentication
 Performance

9. /clients /nodes
/data /environments
/roles /sandboxes
/cookbooks /search

10.  Mostly JSON  almost BSON
 References to Ruby files stored separately
{
"name": "allura-0.0.1”,…
"json_class": "Chef::CookbookVersion",
Ruby files stored on S3
"attributes": [
{
"name": "default.rb",
"url": "https://s3.amazonaws.com/opscode-platform-production-data/…
"checksum": "749a3a328d6c47a32d134b336183981f",
"path": "attributes/default.rb",
"specificity": "default”…

11. MongoDBValidator
role = collection(
'chef.role', doc_session,
Field('_id', S.ObjectId()),
Field('account_id', S.ObjectId(if_missing=None)),
Shorthand with
Field('name', str),
Python Types
Field('description', str),
Field('default_attributes', str),
Field('override_attributes', str),
Field('run_list', [ str ] ),
Field('env_run_lists', { str: [ str ]}),
Index('account_id', 'name', unique=True))
Embedded
Index Definitions
Documents

12. classRole(ModelBase): Models know
where they live
def__name__(self):
returnself.name
Models can be
def__json__(self): turned into dict (to
returndict( be JSONified)
chef_type='role',
json_class='Chef::Role',
…
default_attributes=loads(self.default_attributes),
Models can be
…)
updated from dict
defupdate(self, d):
self.name = d['name']
…
self.default_attributes = dumps(d['default_attributes'])
self.override_attributes = dumps(d['override_attributes'])
…

13. classRoleSchema(JSONModelSchema):
model_class=CM.role
chef_type='role’
json_class='Chef::Role’
exclude_fields=['_id', 'account_id']

14.  Find resource
 /foo/bar/bazRoot()[‘foo’][‘bar’][‘baz’]
 (Resource, method)  view
 @view_config(context=Resource, request_method=‘GET’, …)
 Authorization
 @view_config(…, permission=‘read’, …)
 Rendering
 @view_config(…, renderer=‘json’)

15. classResourceCollection(object):
__name__ = None Check auth
__model__ = None
key_property='name’
defallow_access(self, client, permission):
return permission == 'read'orclient.admin Traverse to find
sub-objects
def__getitem__(self, name):
obj = self.account.get_object(
self.__model__, **{
self.key_property: name })
ifobjisNone: raiseexc.HTTPNotFound()
obj.__parent__ = self
returnobj

16. @view_config(context=Roles, renderer='json', Resources can be
request_method='GET’, permission='read') located
deflist_roles(context, request):
returndict(
(n.name, request.resource_url(n))
fornincontext.find())
Convert and
@view_config(context=Roles, renderer='json', validate JSON
request_method='POST’,permission='create') input
defcreate_role(context, request):
n = context.new_object()
value = V.RoleSchema().to_python(
request.json, None)
n.update(value)
…

17.  Don’t trust the docs
 Don’t trust the docs
▪ Don’t trust the docs
 Use fat models
 Framework support for REST & JSON
 You’re gonna have to learn some Ruby anyway
 JSON != BSON

18.  Port from homegrown framework to Pyramid
 Better test coverage
 Search support (SOLR / ElasticSearch)
 More testing with real-world deployments
 Finalize integration with deployment
manager

19. Rick Copeland @rick446
Arborian Consulting, LLC

20. Rick Copeland @rick446
Arborian Consulting, LLC