Chef is an amazing tool but to really unlock its potential you need to look at how it integrates with the rest of your technology. This presentation is the story of how the NFL used Chef to transform its siloed infrastructure and practices into something more agile, automated, and reliable. This presentation will talk about the last 2 years of Chef at the NFL, including how we integrated it with our virtualization infrastructure, load balancers, storage, and application performance monitoring. We'll talk about some things that Chef taught us about infrastructure as code that we were able to apply to other areas, and things we learned to make our cookbooks easier to manage across groups.

1. @seanwalberg
BREAKING TECHNOLOGY
SILOS WITH CHEF
Sean Walberg <sean@ertw.com>
Infrastructure guy
National Football League

2. @seanwalberg
These are silos
(Every DevOps presentation needs a picture of a silo)

3. @seanwalberg
Yup

4. @seanwalberg
Tech view

5. @seanwalberg
People view

6. @seanwalberg
How do you change a
culture?

7. @seanwalberg
BLUF
Optimize for conversations – automate
the bad ones away
Make a menu – Do one thing, and do it
well.
Don’t neglect your team

8. @seanwalberg
Start Development
Go faster!
STOP!

9. @seanwalberg
OFF SEASON, 2014

10. @seanwalberg

11. @seanwalberg
LET’S FIX SOME PROBLEMS
• Environmental drift
• Configuration files
• Developer access to servers

12. @seanwalberg
INTER/INTRA ENVIRONMENT DRIFT

13. @seanwalberg
AUTOMATE ALL THE THINGS!

14. @seanwalberg
NFL NOW ENVIRONMENT
• About a dozen services and ~100 servers by the
end in production, ~200 in total
• Cookbook per app
• Environment settings in Chef environment

15. @seanwalberg
CONFIGURATION FILES
http://www.mommysbusy.com/wp-content/uploads/2013/10/Messy-Desk.jpg

16. @seanwalberg
CAN WE FIX IT? YES WE CAN!
Developer modifies config on dev server
Infrastructure team diffs and templates
Other environment settings are added to cookbook or environment
Deploy to staging/preview/production

17. @seanwalberg
http://6iee.com/372028.html
DEVELOPER ACCESS TO SERVERS

18. @seanwalberg
• Development only
• Except when we’re in a crunch
• And then we turn off Chef
• Then I spend hours cleaning it up
DEVELOPER ACCESS TO SERVERS

19. @seanwalberg
KNIFE-VSPHERE
• Clone + Bootstrap VM
• Take snapshots
• Resize/add disk
• Change settings
• Run commands

20. @seanwalberg
DURING THE SEASON

21. @seanwalberg
NODE, ADD THYSELF TO A POOL!
f5_pool ”pool_name" do
host node['fqdn']
ip node['ipaddress']
port app_port
not_if { node['apps']['skip_load_balancer’]}
end

22. @seanwalberg
START ROLLING OUT CHEF
• role[base] – everything new
• LDAP
• Access control
• Base packages
• role[minimal] – just get Chef on it
• Base packages
• SSH keys

23. @seanwalberg
RETROSPECTIVE
• Chef + knife-vsphere + f5 worked great
• Still many manual steps though
• Why do developers still need access to servers?
• Can we manage config files asynchronously?
• No standardization in frameworks.

24. @seanwalberg
OFF SEASON, 2015
Microservices, yay!

25. @seanwalberg
I REALLY WANT TO FIX THIS STUFF
• Developers on servers
• Standardization of frameworks
• Config files

26. @seanwalberg
LET’S HAVE SOME CONVERSATIONS
• “Is the config on the box what it should be?”
• “I need to see the logs!”
• “This framework looks cool, maybe I’ll try it.”
• “I don’t trust what you’re telling me.”
• “I didn’t know that makes your life harder.”

27. @seanwalberg
THEY’RE ALL RELATED!

28. @seanwalberg
WE’RE A RESTAURANT AND WE
HAVE ONE DISH

29. @seanwalberg
LE MENU
• Everything is built around the name of the app
• Start with a consistent build pipeline
• Start projects from a template
• Log with slf4j, we’ll config graylog for you
• Deployable fat JAR (for Java stuff)
• Instrumentation is added on the server
• One chef recipe

30. @seanwalberg
CONSISTENT NAMING
Tags: [app:sso]
Runlist: recipe[nfl-apps]
Server: locenvssozz
Service: sso
Logs: /var/log/sso/
Deployable: /opt/nfl/sso/sso.jar
Graylog: tag logs with app:sso
AppD Tier: sso
Repo: sso

31. @seanwalberg
LOGGING
template "/opt/nfl/…/logback.xml” do
source 'logback.xml.erb'
owner 'root'
mode '755'
variables(
app_service: app,
instance_id: instance_id,
environment: node.chef_environment
)
end

32. @seanwalberg
ADD SOME CONTEXT TO LOGS
<staticAdditionalField>_app:<%= @app_service %></staticAdditionalField>
<staticAdditionalField>_env:<%= @environment %></staticAdditionalField>
<staticAdditionalField>_instance:<%= @instance_id %></staticAdditionalField>

33. @seanwalberg
EASY CHANGING OF LOGGING
<!-- Shortcut to debug -->
<% node['apps']['debug_packages'].each do |package| %>
<logger name="<%= package %>" level="debug" />
<% end %>
<!-- Fine tuning -->
<% if node['apps'].key? 'loglevel’
node['apps']['loglevel'].each do |package, level| %>
<logger name="<%= package %>" level="<%= level %>" />
<% end; end %>
<!-- Default logging based on tag -->
<logger name="com.nfl.dm.<%= @app_service %>" level="info" />

34. @seanwalberg
APPDYNAMICS (APM)
Install agents
Configure agents based on attributes
No knowledge of app
Include_recipe “nfl-appdynamics”
Adjust startup scripts
Handle custom AppD config
Legacy stuff
recipe[nfl-appdynamics::agent]
Fix your own startup scripts and custom config

35. @seanwalberg
LET’S FIX CONFIGURATION
• Chef drops a consul agent on each server
• Joins it to the cluster for that environment
• Sets startup scripts for the app to tell the app where to
find consul, and the configs within the KV store
• Starter template provides a module that reads Consul on
startup and configs Spring Boot
• Config is in a repo that anyone can use
I would be lying if I said we got this
right on the first try!

36. @seanwalberg
SAME PATTERN AS BEFORE
Install consul
Join to #{node.chef_environment}
No knowledge of app
Include_recipe “nfl-consul”
Adjust startup scripts
Populate service discovery values
Legacy stuff
Recipe[nfl-consul]
Fix your own startup scripts

37. @seanwalberg
USING CONFIG
• Developers commit to Consulation (YAML)
• Peer review + linting/smoketest in Phabricator
• Secrets in Vault
• Auto deploy to environments on merge
The Rules
1. There is no other config but Consulation
2. If it changes meaning, change the key name
3. If you don’t know what it is, ignore it

38. @seanwalberg
ATTRIBUTES ARE A GOOD
INTERFACE
default['apps']['os_memory'] = '1024’
if jvm_memory is not overridden
system_memory = memory_from_ohai
jvm_memory = system_memory – os_memory
end
# Wrapper cookbook can tweak either jvm memory or os_memory

39. @seanwalberg
OR USE A ROLE
$ knife role show os1536_memory_adjustment
chef_type role
name: os1536_memory_adjustment
override_attributes:
apps:
os_memory: 1536

40. @seanwalberg
DO YOU SERVE GLUTEN FREE?
Yes, you can order off the menu.
But we need to talk, and it’ll take longer.
How many conventions are we changing?

41. @seanwalberg
RETROSPECTIVE
• Config evolved, but worked great
• Standardized apps meant no need for dev box
• Devs on servers virtually eliminated
• Time for a new service is ~30min
• Deploy anytime!
• Conversations became higher value

42. @seanwalberg
WHAT ELSE CAN WE AUTOMATE?
• DNS
• AppD configuration
• Builds and deploys
• Fastly (CDN) configuration

43. @seanwalberg
The code needs to be the
source of truth
Configuration Transformation
API driven deploy

44. @seanwalberg
This becomes VCL
This is a functional
test
CHEF TAUGHT US THAT CONFIGURATION
DOESN’T HAVE TO SUCK

45. @seanwalberg
This is so
much better
than that

46. @seanwalberg
DON’T AUTOMATE THE
CONVERSATIONS AWAY

47. @seanwalberg
NOW, LET’S GET BETTER AT
USING CHEF
Again, conversations will help us.

48. @seanwalberg
CONTINUOUS INTEGRATION
“Does it converge?”

49. @seanwalberg
PEER REVIEW

50. @seanwalberg
AUTO LINTING

51. @seanwalberg
ESTABLISH A WORKFLOW

52. @seanwalberg
ALSO REMEMBER
You’re coding for the most
inexperienced member of your team
AKA the “the new person”

53. @seanwalberg
LET’S WRAP
• Think about conversations
• Chef is more than just configs on servers
• Reduce complexity with a menu
• Apply the “Chef Way” elsewhere
• Keep improving!

54. @seanwalberg
THANKS!
Sean Walberg <sean@ertw.com>
@seanwalberg