SlideShare una empresa de Scribd logo
1 de 87
Descargar para leer sin conexión
Operations Level Up
Mandi Walls	

LOPSA-East IT 2014	

May 3, 2014
whoami
• Mandi Walls	

• @lnxchk	

• Technical Practice Manager at Chef
2
What is this madness
Operating complex systems is hard enough.	

!
We should be intentional about making it better
when we can.
3
Why Do I Care?
4
http://static6.businessinsider.com/image/4a64e07c4b5437860086a0c9/uh-oh-cond-nast-hires-mckinsey.jpg
Future of Operations
http://www.flickr.com/photos/x-ray_delta_one/5871906878/5
Evolution of a Practice
• Craft Stage	

• Commercial Stage	

• Engineering Stage
6 http://www.flickr.com/photos/thaisfraga182/5285413020/sizes/z/in/photostream/
Craft Stage
• Hand crafted artisanal organic free range
bespoke systems	

• Lots of personal heroics	

• Land of the BOFHs
7
Commercial Stage
• Folklore written down	

• Standard procedures emerge	

• Training begins to occur
8
Engineering Stage
• Application of scientific principles	

• Measurement	

• Experimentation towards greater efficiency
9
New Workflows
• Visibility and planning	

• Version control and code review	

• Testing, testing, and more testing	

• Metrics collection and interpretation
Basically, borrow
some stuff from Dev
10 http://websites-development.com/sites/default/files/git_branch_strategy.png
New Goals
• Transparency - are we working on the right things	

• Reliability - can we keep it running	

• Resiliency - can we rebuild it? Do we have the technology?	

• Correctness - are we sure it’s doing what we want it to do
11
Building Trust
More than keeping the lights on.
12
Ops Identity Crisis
http://www.flickr.com/photos/muffett68/7214428636/
I don’t write code, I’m a sysadmin
I have to spend all my time fixing dumb things
This takes too much time.
These tools are too
hard to learn.
I will write my own
thing.
I’m faster if I don’t have to talk to
anyone about what’s going on.
13
So, some things to work on
• Some tools for mitigating risk	

• Some processes and tips for making the right thing the easy thing	

• Increase efficiency, learn some stuff, reevaluate your own work	

• Don’t be afraid of borrowing from other disciplines
14
15 http://www.packriveryaks.com/
Opportunity Cost
The value of the things you could be
doing while you were shaving that yak
16
Employability
17 http://www.flickr.com/photos/sourmash/74666764/
RiskVectors
• What Ops thinks of as risk	

• New code, releases, tasks	

• Other sources of risk	

• Old products and workflows	

• Unrepeatable processes	

• Personal heroics
http://www.flickr.com/photos/baresone/4473290629/sizes/z/in/photostream/18
Assessment of Risk
• Is your process:	

• well documented	

• repeatable	

• reliable	

• easy to do right?
http://www.flickr.com/photos/lemusgro/5494317161/sizes/z/in/photostream/19
EASY TO DO RIGHT
Seriously. I’m not kidding.
20
New Tools
• Git and hooks for Ops	

• Packaging your stuff	

• Borrowing sanity checks from other places	

• Basic testing without doing a lot of zomgcoding	

• Configuration Management
21
Tool 1 - Working with git
http://mattbanks.me/wp-content/uploads/2012/07/Git-Logo.png22
Why git
• Distributed version control 	

• Everyone gets a copy	

• Hub/spoke model for sharing	

• Simple set up	

• Easy to run a local git server	

• Other offerings, like github, are pretty
awesome too
23
.git/config
[core]!
! repositoryformatversion = 0!
! filemode = true!
! bare = false!
! logallrefupdates = true!
[remote origin]!
! fetch = +refs/heads/*:refs/remotes/origin/*!
! url = ssh://localhost/srv/git/bindfiles.git!
[branch master]!
! remote = origin!
! merge = refs/heads/master!
Remote origin!
24
$ vi db.192
Example workflow: zonefiles
25
Add “wat.local” with final octet 24
Add a new host
26
$ git status
git status
# On branch master!
# Changed but not updated:!
# (use git add file... to update what will be committed)!
# (use git checkout -- file... to discard changes in working directory)!
#!
#!modified: db.192!
#!
no changes added to commit (use git add and/or git commit -a)
27
git tells you what it wants
# Changed but not updated:	

# (use git add file... to update what will be committed)	

# (use git checkout -- file... to discard changes in working
directory)	

#	

#	

 modified: db.192
28
$ git add db.192
$ git status
git add
# On branch master!
# Changes to be committed:!
# (use git reset HEAD file... to unstage)!
#!
#! modified: db.192!
#
29
git commit
• git add stages your changes
locally	

• git commit will write them to
your local git repository	

• add your comment either
inline with “-m” or git commit
will open a buffer for you
30
git commit -m “this commit is awesome”
$ git commit
git commit
31
[master 22371ab] Added wat.local to reverse file!
1 files changed, 4 insertions(+), 0 deletions(-)
$ git status
# On branch master!
# Your branch is ahead of 'origin/master' by 1 commit.!
#!
nothing to commit (working directory clean)
32
Making Good Comments
• At least explain what you did, Lucy	

• If there is a ticket somewhere, add that in the comment	

• If you made multiple changes, call them all out
33
$ git push
git push
Counting objects: 5, done.!
Compressing objects: 100% (3/3), done.!
Writing objects: 100% (3/3), 335 bytes, done.!
Total 3 (delta 1), reused 0 (delta 0)!
To ssh://localhost/srv/git/bindfiles.git!
06fa560..22371ab master - master
34
git push
• git push sends your changes to the central git
server	

• git pull brings everyone else’s changes into
your local repo	

• Don’t hoard changes; push and pull often
35
Ok?
• What did we forget to do?
36
Update the Serial!
• Lots of administration tasks have tribal knowledge you need	

• Zonefiles have a Serial that needs to be incremented when you make a
change	

• They are potentially outage-causing or hair-pulling problems that can
be avoided	

• Let’s let git remember to do that for us
37
commit hooks
• You can put hooks into your git repos	

• Little tasks that happen at various steps in the process	

• We can add a pre-commit hook to our bindfiles repo	

• So you don’t have to remember! Saves time later! Helps junior staff!
38
$ cp /srv/myrepo/pre-commit .git/hooks
$ cat .git/hooks/pre-commit
pre-commit
#!/bin/bash!
num=`git diff master db.192 | grep ^+ | wc | awk '{print $1}'`!
if [ $num -gt 1 ] ; then!
serial=`git diff master db.192 | grep -i serial`!
if [ $? -ne 0 ] ; then !
echo You made a change to the zone file but didn't update the Serial value!
exit 1;!
fi!
fi
39
pre-commit
• Rather messy, off-the-cuff example	

• git diff master db.192!
• Looks for changes between what’s in the current master on your
local repo	

• If the db.192 file has changed but the value for Serial is the same, it
prints and error and exits with a non-zero return code	

• git stops processing the commit, saving you headaches later
40
Generalized pre-commit
41
What else to hook?
• Services with config checkers	

• make a change to the config, run the checker in a hook 	

• nagios, named, apache, etc come with check tools	

• Other syntax checking	

• ruby, json, config management tools
Make it	

EASY	

to do 	

RIGHT
42
Other git things
• git clone	

• gives new team members a working copy of the repo	

• git branch	

• Lets you and other team members work on non-prod stuff	

• Use sparingly; less complexity means more reliable results
43
Tool 2: fpm
• How do you get files, apps, stuff deployed on your hosts?	

• scp -r?	

• tarballs?	

• build everything on every host, you gentoo fans?	

• crash cart? (omg)
44
Packaging
• Reap the benefits of what’s built in to your package manager	

• Versioning	

• Dependencies	

• Metadata	

• Build-once, install-many	

• File transfer built right into stuff like yum and apt repos!
45
Package.All.The.Things.
46 http://cdn.meme.li/instances/300x300/38833426.jpg
Make it easy: fpm
• Creating packages from scratch is tedious	

• There’s some esoteric stuff in the package managers	

• You really only need a few things
47
fpm
• fpm,“f’ing package managers”!	

• Jordan Sissell	

• Creates multiple kinds of packages from various resources	

• https://github.com/jordansissel/fpm
48
$ fpm -h
fpm
Intro:!
This is fpm version 0.4.37!
If you think something is wrong, it's probably a bug! :)!
Please file these here: https://github.com/jordansissel/fpm/issues!
You can find support on irc (#fpm on freenode irc) or via email with!
fpm-users@googlegroups.com!
Usage:!
fpm [OPTIONS] [ARGS] ...!
!
Parameters:!
[ARGS] ... Inputs to the source package type. For the 'dir' type, this is the files and
directories you want to include in the package. For others, like 'gem', it specifies the packages to download and use
as the gem input!
!
Options:!
-t OUTPUT_TYPE the type of package you want to create (deb, rpm, solaris, etc)!
-s INPUT_TYPE the package type to use as input (gem, rpm, python, etc)!
-C CHDIR Change directory to here before searching for files
omg it just keeps going.... 49
Create a package
• rpm-ify our zonefiles	

• They’re in our git repo right now	

• Live in /var/named for reals	

• If we package them, we get versioning and other data
50
fpm settings
• -s dir : we’re working with raw files rather than a gem, rpm, etc	

• -t rpm : create an rpm package	

• -v 1.0 : first version!	

• --prefix=/var/named : where the files will be installed	

• -n “zonefiles” : name of the package	

• --after-install /srv/velocity/restart_named.sh : run this after installing	

• db* : the files to be packaged
$ fpm -s dir -t rpm -v 1.0 --prefix=/var/named 
-n zonefiles --after-install 
/srv/myfiles/restart_named.sh db*
51
$ fpm -s dir -t rpm -v 1.0 --prefix=/var/named 
-n zonefiles --after-install 
/srv/velocity/restart_named.sh db*
Created rpm {:path=zonefiles-1.0-1.x86_64.rpm}
$ rpm -qpl zonefiles-1.0-1.x86_64.rpm
/var/named/db.192!
/var/named/db.local
52
• Nice!	

• Now we can install it
53
$ sudo rpm -ihv zonefiles-1.0-1.x86_64.rpm
Results!
Preparing... ########################################### [100%]!
1:zonefiles ########################################### [100%]!
Stopping named: .[ OK ]!
Starting named: [ OK ]
$ dig @localhost -x 192.168.1.22
;; QUESTION SECTION:!
;22.1.168.192.in-addr.arpa.!IN! PTR!
;; ANSWER SECTION:!
22.1.168.192.in-addr.arpa. 604800 IN! PTR!wat.local.
54
Put the bits together
• Your zonefiles are in a git repo	

• The repo has syntax and error
checking pre-commit hooks	

• The repo can also have
packaging and deploy post-
commit hooks	

• Smooth the process, make the
right way the easiest way
http://www.flickr.com/photos/62904109@N00/2636859006/sizes/z/in/photostream/55
Tool 3:Testing
• Lots of work in the dev space	

• TDD, BDD, test, test	

• Write tests first, prove they fail, write
code to make them pass	

• More risk reduction	

• Looks scary
http://www.flickr.com/photos/nobleup/3995733415/sizes/z/in/photostream/56
basic tests
• So, you’re running DNS	

• What else is do you have?	

• Monitoring server!
http://www.flickr.com/photos/richardmoross/490988453/sizes/z/in/photostream/57
What can we borrow?
• Nagios plugins!	

• Extensive set of checks for all sorts of services	

• Usable from the command line
58
$ ls /usr/lib64/nagios/plugins
nagios plugins
check_breeze check_game check_mrtgtraf check_overcr check_swap!
check_by_ssh check_hpjd check_mysql check_pgsql check_tcp!
check_clamd check_http check_mysql_query check_ping check_time!
check_cluster check_icmp check_nagios check_pop check_udp!
check_dhcp check_ide_smart check_nntp check_procs check_ups!
check_dig check_imap check_nntps check_real check_users!
check_disk check_ircd check_nrpe check_rpc check_wave!
check_disk_smb check_jabber check_nt check_sensors eventhandlers!
check_dns check_ldap check_ntp check_simap negate!
check_dummy check_ldaps check_ntp_peer check_smtp urlize!
check_file_age check_load check_ntp.pl check_snmp utils.pm!
check_flexlm check_log check_ntp_time check_spop utils.sh!
check_fping check_mailq check_nwstat check_ssh!
check_ftp check_mrtg check_oracle check_ssmtp!
!
Hey! A DNS Checker!
59
check_dns
• We can use these plugins to test out what we’re doing 	

• Don’t require any additional frameworks or scary things	

• Many of them work just fine over the network, too
60
$ /usr/lib64/nagios/plugins/check_dns -h
Check DNS
check_dns v1.4.16 (nagios-plugins 1.4.16)!
Copyright (c) 1999 Ethan Galstad nagios@nagios.org!
Copyright (c) 2000-2008 Nagios Plugin Development Team!
! nagiosplug-devel@lists.sourceforge.net!
!
This plugin uses the nslookup program to obtain the IP address for the given host/domain query.!
An optional DNS server to use may be specified.!
If no DNS server is specified, the default server(s) specified in /etc/resolv.conf will be used.!
!
Usage:!
check_dns -H host [-s server] [-a expected-address] [-A] [-t timeout] [-w warn] [-c crit]!
!
61
$ check_dns -H box.local -s 127.0.0.1 -a 192.168.1.21
When check_dns is ok
DNS OK: 0.004 seconds response time. box.local
returns 192.168.1.21|time=0.004142s;;;0.000000
$ echo $?
0
62
$ check_dns -H box.local -s 127.0.0.1 -a 192.168.1.22
check_dns errors
DNS CRITICAL - expected '192.168.1.22' but got '192.168.1.21'!
$ echo $?
2
63
cool
• Now we have a way to test our changes	

• Behaves in a predictable way	

• Now let’s add one more component: a test harness
64
Tool 4: bats
• Bash Automated Testing System	

• Like all good tools, bats is impossible to google	

• https://github.com/sstephenson/bats
omg these are adorable.	

http://www.flickr.com/photos/37539972@N06/3980094382/sizes/z/in/photostream/
65http://www.etsy.com/shop/theitsybitsyspider
What the bats
• Allows you to test that UNIX programs do what you expect	

• Write stuff in bash to test other system commands	

• Easy to get hold of return codes, output	

• Let’s see an example: checking the nagios configs
66
$ bats /srv/myfiles/nagios.bats
Using bats
1..1!
ok 1 nagios is all good
Ran one test
All good!
67
$ cat /srv/myfiles/nagios.bats
#!/usr/bin/env bats!
@test nagios is all good {!
result=$(sudo service nagios checkconfig)!
[ $? -eq 0 ]!
}
Run a system command!
Check the return code!
Also grabs output, but 	

we don’t need that here
68
We can do this!
http://www.flickr.com/photos/usnationalarchives/3678696585/
69
Tool 5: Config Management
70
• Oh you’re totally going to automate yourself out of a job...
Why?
71
0
1
2
3
4
5
6
Work To 	

Be Done
Work
Doable By
N Ops
Work That Won’t
Get Done
Features of Config Management
• Repeatability - configure services the same way, every time	

• Reliability - ensure that the services are always configured correctly	

• Documentation - record of what actions were taken on the system	

• Idempotent - only take action if necessary
72
A Chef Recipe
package “named” do!
action :install!
end!
service “named” do!
action [:start, :enable]!
end!
package “zonefiles” do!
action :install!
notifies :restart, “service[named]”!
end!
73
CM Tools
• Record your configuration into version control 	

• Build hosts in your datacenter, in the cloud, build reals, build virtuals	

• Support heterogeneous environments	

• Install packages, write configurations, manage services, users, groups,
files, registry settings, etc
74
Windows?
• Learn. PowerShell.	

• Then get into DSC	

• DSC support is coming for CM tools, and will be a powerful way to
manage Windows environments
75
New Workflows
76
make a
change
in the
cm files
check
into git
git
hooks
check
for
errors
run a
few
tests
deploy
to
hosts
make a
change
in the
app
code
check
into git
git
hooks
check
for
errors
run a
few
tests
build a
package
add to
artifact
repo
Our Goals:Transparency
• Are we working on something that adds value?
77
Our Goals: Reliability
• Does our new process keep things running?
78
Our Goals: Resiliency
• Does our new process make it easy to rebuild, recover, scale?
79
Our Goals: Correctness
• Does our new process ensure that the work we’re doing is correct?
80
Building from here
81 http://www.flickr.com/photos/kalmyket/691478431/sizes/z/in/photostream/
Cheaper Resources
• Do more real-world testing	

• Local virtuals - vagrant, cloud providers	

• Linux containers - docker	

• Make Dev and QA really look like prod
82
Build Server
• Jenkins,Travis,Team City, etc	

• Build and test configs and app code together	

• Never forget a step in your new workflow!
83 http://www.flickr.com/photos/hubmedia/2141860216/sizes/z/in/photostream/
MakeYour Job Better
• When your job is better, so is your life	

• Fewer emergencies, less opaqueness of systems and processes
encourages collaboration and shared duty	

• Be intentional about the things that we do and our goals	

• Know that what you do day to day is improving
84
Takeaways
• Reliable, repeatable processes	

• Make stuff easy to do right	

• Reduce risk of mistakes, misunderstandings	

• Reduce the need for personal heroics	

• Be intentional about the work we do and
focus on being valuable
http://www.flickr.com/photos/ginnerobot/2877212845/sizes/z/in/photostream/85
Thanks!
• Thanks for your kind attention	

• Please keep the conversation going with your teams
86
We’re Hiring!
http://getchef.com/careers/
87

Más contenido relacionado

La actualidad más candente

Getting Git Right
Getting Git RightGetting Git Right
Getting Git RightSven Peters
 
VCS for Teamwork - GIT Workshop
VCS for Teamwork - GIT WorkshopVCS for Teamwork - GIT Workshop
VCS for Teamwork - GIT WorkshopAnis Ahmad
 
Code reviews vs Pull requests
Code reviews vs Pull requestsCode reviews vs Pull requests
Code reviews vs Pull requestsTim Pettersen
 
Starting with Git & GitHub
Starting with Git & GitHubStarting with Git & GitHub
Starting with Git & GitHubNicolás Tourné
 
Intro to git and git hub
Intro to git and git hubIntro to git and git hub
Intro to git and git hubVenkat Malladi
 
Packaging is the Worst Way to Distribute Software, Except for Everything Else
Packaging is the Worst Way to Distribute Software, Except for Everything ElsePackaging is the Worst Way to Distribute Software, Except for Everything Else
Packaging is the Worst Way to Distribute Software, Except for Everything Elsemckern
 
Git the Docs: A fun, hands-on introduction to version control
Git the Docs: A fun, hands-on introduction to version controlGit the Docs: A fun, hands-on introduction to version control
Git the Docs: A fun, hands-on introduction to version controlBecky Todd
 
Automating Your Workflow with Gulp.js - php[world] 2016
Automating Your Workflow with Gulp.js - php[world] 2016Automating Your Workflow with Gulp.js - php[world] 2016
Automating Your Workflow with Gulp.js - php[world] 2016Colin O'Dell
 
Princeton jug git_github
Princeton jug git_githubPrinceton jug git_github
Princeton jug git_githubYakov Fain
 
Git and github fundamental
Git and github fundamentalGit and github fundamental
Git and github fundamentalRajesh Kumar
 

La actualidad más candente (20)

Getting Git Right
Getting Git RightGetting Git Right
Getting Git Right
 
Hello, Git!
Hello, Git!Hello, Git!
Hello, Git!
 
Introduction to Git
Introduction to GitIntroduction to Git
Introduction to Git
 
VCS for Teamwork - GIT Workshop
VCS for Teamwork - GIT WorkshopVCS for Teamwork - GIT Workshop
VCS for Teamwork - GIT Workshop
 
Code reviews vs Pull requests
Code reviews vs Pull requestsCode reviews vs Pull requests
Code reviews vs Pull requests
 
Introduction to Git (part 1)
Introduction to Git (part 1)Introduction to Git (part 1)
Introduction to Git (part 1)
 
Starting with Git & GitHub
Starting with Git & GitHubStarting with Git & GitHub
Starting with Git & GitHub
 
git and github
git and githubgit and github
git and github
 
Logstash and friends
Logstash and friendsLogstash and friends
Logstash and friends
 
Git & Github for beginners
Git & Github for beginnersGit & Github for beginners
Git & Github for beginners
 
Introduction to Git and GitHub
Introduction to Git and GitHubIntroduction to Git and GitHub
Introduction to Git and GitHub
 
Intro to git and git hub
Intro to git and git hubIntro to git and git hub
Intro to git and git hub
 
Packaging is the Worst Way to Distribute Software, Except for Everything Else
Packaging is the Worst Way to Distribute Software, Except for Everything ElsePackaging is the Worst Way to Distribute Software, Except for Everything Else
Packaging is the Worst Way to Distribute Software, Except for Everything Else
 
An API Your Parents Would Be Proud Of
An API Your Parents Would Be Proud OfAn API Your Parents Would Be Proud Of
An API Your Parents Would Be Proud Of
 
Gitlikeapro 2019
Gitlikeapro 2019Gitlikeapro 2019
Gitlikeapro 2019
 
Git and Github workshop
Git and Github workshopGit and Github workshop
Git and Github workshop
 
Git the Docs: A fun, hands-on introduction to version control
Git the Docs: A fun, hands-on introduction to version controlGit the Docs: A fun, hands-on introduction to version control
Git the Docs: A fun, hands-on introduction to version control
 
Automating Your Workflow with Gulp.js - php[world] 2016
Automating Your Workflow with Gulp.js - php[world] 2016Automating Your Workflow with Gulp.js - php[world] 2016
Automating Your Workflow with Gulp.js - php[world] 2016
 
Princeton jug git_github
Princeton jug git_githubPrinceton jug git_github
Princeton jug git_github
 
Git and github fundamental
Git and github fundamentalGit and github fundamental
Git and github fundamental
 

Similar a Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.

Open Source Tools for Leveling Up Operations FOSSET 2014
Open Source Tools for Leveling Up Operations FOSSET 2014Open Source Tools for Leveling Up Operations FOSSET 2014
Open Source Tools for Leveling Up Operations FOSSET 2014Mandi Walls
 
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflowTomas Doran
 
The Basics of Open Source Collaboration With Git and GitHub
The Basics of Open Source Collaboration With Git and GitHubThe Basics of Open Source Collaboration With Git and GitHub
The Basics of Open Source Collaboration With Git and GitHubBigBlueHat
 
Git installation and configuration
Git installation and configurationGit installation and configuration
Git installation and configurationKishor Kumar
 
Que nos espera a los ALM Dudes para el 2013?
Que nos espera a los ALM Dudes para el 2013?Que nos espera a los ALM Dudes para el 2013?
Que nos espera a los ALM Dudes para el 2013?Bruno Capuano
 
Git One Day Training Notes
Git One Day Training NotesGit One Day Training Notes
Git One Day Training Notesglen_a_smith
 
Practical Git - NYC Code Camp
Practical Git - NYC Code CampPractical Git - NYC Code Camp
Practical Git - NYC Code CampChristopher Gomez
 
Git for folk who like GUIs
Git for folk who like GUIsGit for folk who like GUIs
Git for folk who like GUIsTim Osborn
 
Untangling fall2017 week2_try2
Untangling fall2017 week2_try2Untangling fall2017 week2_try2
Untangling fall2017 week2_try2Derek Jacoby
 
Untangling fall2017 week2
Untangling fall2017 week2Untangling fall2017 week2
Untangling fall2017 week2Derek Jacoby
 
OSDC 2013 | Introduction into Chef by Andy Hawkins
OSDC 2013 | Introduction into Chef by Andy HawkinsOSDC 2013 | Introduction into Chef by Andy Hawkins
OSDC 2013 | Introduction into Chef by Andy HawkinsNETWAYS
 
Continuous Integration, the minimum viable product
Continuous Integration, the minimum viable productContinuous Integration, the minimum viable product
Continuous Integration, the minimum viable productJulian Simpson
 
Git 101: Git and GitHub for Beginners
Git 101: Git and GitHub for Beginners Git 101: Git and GitHub for Beginners
Git 101: Git and GitHub for Beginners HubSpot
 
Package manages and Puppet - PuppetConf 2015
Package manages and Puppet - PuppetConf 2015Package manages and Puppet - PuppetConf 2015
Package manages and Puppet - PuppetConf 2015ice799
 
'Intro to Infrastructure as Code' - DevOps Belfast
'Intro to Infrastructure as Code' - DevOps Belfast'Intro to Infrastructure as Code' - DevOps Belfast
'Intro to Infrastructure as Code' - DevOps BelfastJohn Fitzpatrick
 
Reproducible research: practice
Reproducible research: practiceReproducible research: practice
Reproducible research: practiceC. Tobin Magle
 

Similar a Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014. (20)

Open Source Tools for Leveling Up Operations FOSSET 2014
Open Source Tools for Leveling Up Operations FOSSET 2014Open Source Tools for Leveling Up Operations FOSSET 2014
Open Source Tools for Leveling Up Operations FOSSET 2014
 
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflow
 
Switching to Git
Switching to GitSwitching to Git
Switching to Git
 
The Basics of Open Source Collaboration With Git and GitHub
The Basics of Open Source Collaboration With Git and GitHubThe Basics of Open Source Collaboration With Git and GitHub
The Basics of Open Source Collaboration With Git and GitHub
 
Git installation and configuration
Git installation and configurationGit installation and configuration
Git installation and configuration
 
Que nos espera a los ALM Dudes para el 2013?
Que nos espera a los ALM Dudes para el 2013?Que nos espera a los ALM Dudes para el 2013?
Que nos espera a los ALM Dudes para el 2013?
 
Git One Day Training Notes
Git One Day Training NotesGit One Day Training Notes
Git One Day Training Notes
 
Hacking on WildFly 9
Hacking on WildFly 9Hacking on WildFly 9
Hacking on WildFly 9
 
Practical Git - NYC Code Camp
Practical Git - NYC Code CampPractical Git - NYC Code Camp
Practical Git - NYC Code Camp
 
Git for folk who like GUIs
Git for folk who like GUIsGit for folk who like GUIs
Git for folk who like GUIs
 
Untangling fall2017 week2_try2
Untangling fall2017 week2_try2Untangling fall2017 week2_try2
Untangling fall2017 week2_try2
 
Untangling fall2017 week2
Untangling fall2017 week2Untangling fall2017 week2
Untangling fall2017 week2
 
Git tips and tricks
Git   tips and tricksGit   tips and tricks
Git tips and tricks
 
OSDC 2013 | Introduction into Chef by Andy Hawkins
OSDC 2013 | Introduction into Chef by Andy HawkinsOSDC 2013 | Introduction into Chef by Andy Hawkins
OSDC 2013 | Introduction into Chef by Andy Hawkins
 
Continuous Integration, the minimum viable product
Continuous Integration, the minimum viable productContinuous Integration, the minimum viable product
Continuous Integration, the minimum viable product
 
Git 101: Git and GitHub for Beginners
Git 101: Git and GitHub for Beginners Git 101: Git and GitHub for Beginners
Git 101: Git and GitHub for Beginners
 
Package manages and Puppet - PuppetConf 2015
Package manages and Puppet - PuppetConf 2015Package manages and Puppet - PuppetConf 2015
Package manages and Puppet - PuppetConf 2015
 
'Intro to Infrastructure as Code' - DevOps Belfast
'Intro to Infrastructure as Code' - DevOps Belfast'Intro to Infrastructure as Code' - DevOps Belfast
'Intro to Infrastructure as Code' - DevOps Belfast
 
Reproducible research: practice
Reproducible research: practiceReproducible research: practice
Reproducible research: practice
 

Más de Mandi Walls

DOD Raleigh Gamedays with Chaos Engineering.pdf
DOD Raleigh Gamedays with Chaos Engineering.pdfDOD Raleigh Gamedays with Chaos Engineering.pdf
DOD Raleigh Gamedays with Chaos Engineering.pdfMandi Walls
 
Addo reducing trauma in organizations with SLOs and chaos engineering
Addo  reducing trauma in organizations with SLOs and chaos engineeringAddo  reducing trauma in organizations with SLOs and chaos engineering
Addo reducing trauma in organizations with SLOs and chaos engineeringMandi Walls
 
Full Service Ownership
Full Service OwnershipFull Service Ownership
Full Service OwnershipMandi Walls
 
PagerDuty: Best Practices for On Call Teams
PagerDuty: Best Practices for On Call TeamsPagerDuty: Best Practices for On Call Teams
PagerDuty: Best Practices for On Call TeamsMandi Walls
 
InSpec at DevOps ATL Meetup January 22, 2020
InSpec at DevOps ATL Meetup January 22, 2020InSpec at DevOps ATL Meetup January 22, 2020
InSpec at DevOps ATL Meetup January 22, 2020Mandi Walls
 
Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019Mandi Walls
 
Using Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure SecurityUsing Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure SecurityMandi Walls
 
Adding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17xAdding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17xMandi Walls
 
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Mandi Walls
 
BuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec WorkshopBuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec WorkshopMandi Walls
 
InSpec Workshop at Velocity London 2018
InSpec Workshop at Velocity London 2018InSpec Workshop at Velocity London 2018
InSpec Workshop at Velocity London 2018Mandi Walls
 
DevOpsDays InSpec Workshop
DevOpsDays InSpec WorkshopDevOpsDays InSpec Workshop
DevOpsDays InSpec WorkshopMandi Walls
 
Adding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpecAdding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpecMandi Walls
 
InSpec - June 2018 at Open28.be
InSpec - June 2018 at Open28.beInSpec - June 2018 at Open28.be
InSpec - June 2018 at Open28.beMandi Walls
 
habitat at docker bud
habitat at docker budhabitat at docker bud
habitat at docker budMandi Walls
 
Ingite Slides for InSpec
Ingite Slides for InSpecIngite Slides for InSpec
Ingite Slides for InSpecMandi Walls
 
Habitat at LinuxLab IT
Habitat at LinuxLab ITHabitat at LinuxLab IT
Habitat at LinuxLab ITMandi Walls
 
InSpec Workshop DevSecCon 2017
InSpec Workshop DevSecCon 2017InSpec Workshop DevSecCon 2017
InSpec Workshop DevSecCon 2017Mandi Walls
 
Habitat Workshop at Velocity London 2017
Habitat Workshop at Velocity London 2017Habitat Workshop at Velocity London 2017
Habitat Workshop at Velocity London 2017Mandi Walls
 
InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017Mandi Walls
 

Más de Mandi Walls (20)

DOD Raleigh Gamedays with Chaos Engineering.pdf
DOD Raleigh Gamedays with Chaos Engineering.pdfDOD Raleigh Gamedays with Chaos Engineering.pdf
DOD Raleigh Gamedays with Chaos Engineering.pdf
 
Addo reducing trauma in organizations with SLOs and chaos engineering
Addo  reducing trauma in organizations with SLOs and chaos engineeringAddo  reducing trauma in organizations with SLOs and chaos engineering
Addo reducing trauma in organizations with SLOs and chaos engineering
 
Full Service Ownership
Full Service OwnershipFull Service Ownership
Full Service Ownership
 
PagerDuty: Best Practices for On Call Teams
PagerDuty: Best Practices for On Call TeamsPagerDuty: Best Practices for On Call Teams
PagerDuty: Best Practices for On Call Teams
 
InSpec at DevOps ATL Meetup January 22, 2020
InSpec at DevOps ATL Meetup January 22, 2020InSpec at DevOps ATL Meetup January 22, 2020
InSpec at DevOps ATL Meetup January 22, 2020
 
Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019Prescriptive Security with InSpec - All Things Open 2019
Prescriptive Security with InSpec - All Things Open 2019
 
Using Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure SecurityUsing Chef InSpec for Infrastructure Security
Using Chef InSpec for Infrastructure Security
 
Adding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17xAdding Security to Your Workflow With InSpec - SCaLE17x
Adding Security to Your Workflow With InSpec - SCaLE17x
 
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
 
BuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec WorkshopBuildStuff.LT 2018 InSpec Workshop
BuildStuff.LT 2018 InSpec Workshop
 
InSpec Workshop at Velocity London 2018
InSpec Workshop at Velocity London 2018InSpec Workshop at Velocity London 2018
InSpec Workshop at Velocity London 2018
 
DevOpsDays InSpec Workshop
DevOpsDays InSpec WorkshopDevOpsDays InSpec Workshop
DevOpsDays InSpec Workshop
 
Adding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpecAdding Security and Compliance to Your Workflow with InSpec
Adding Security and Compliance to Your Workflow with InSpec
 
InSpec - June 2018 at Open28.be
InSpec - June 2018 at Open28.beInSpec - June 2018 at Open28.be
InSpec - June 2018 at Open28.be
 
habitat at docker bud
habitat at docker budhabitat at docker bud
habitat at docker bud
 
Ingite Slides for InSpec
Ingite Slides for InSpecIngite Slides for InSpec
Ingite Slides for InSpec
 
Habitat at LinuxLab IT
Habitat at LinuxLab ITHabitat at LinuxLab IT
Habitat at LinuxLab IT
 
InSpec Workshop DevSecCon 2017
InSpec Workshop DevSecCon 2017InSpec Workshop DevSecCon 2017
InSpec Workshop DevSecCon 2017
 
Habitat Workshop at Velocity London 2017
Habitat Workshop at Velocity London 2017Habitat Workshop at Velocity London 2017
Habitat Workshop at Velocity London 2017
 
InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017InSpec Workflow for DevOpsDays Riga 2017
InSpec Workflow for DevOpsDays Riga 2017
 

Último

Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 

Último (20)

Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 

Updated non-lab version of Level Up. Delivered at LOPSA-East, May 3, 2014.

  • 1. Operations Level Up Mandi Walls LOPSA-East IT 2014 May 3, 2014
  • 2. whoami • Mandi Walls • @lnxchk • Technical Practice Manager at Chef 2
  • 3. What is this madness Operating complex systems is hard enough. ! We should be intentional about making it better when we can. 3
  • 4. Why Do I Care? 4 http://static6.businessinsider.com/image/4a64e07c4b5437860086a0c9/uh-oh-cond-nast-hires-mckinsey.jpg
  • 6. Evolution of a Practice • Craft Stage • Commercial Stage • Engineering Stage 6 http://www.flickr.com/photos/thaisfraga182/5285413020/sizes/z/in/photostream/
  • 7. Craft Stage • Hand crafted artisanal organic free range bespoke systems • Lots of personal heroics • Land of the BOFHs 7
  • 8. Commercial Stage • Folklore written down • Standard procedures emerge • Training begins to occur 8
  • 9. Engineering Stage • Application of scientific principles • Measurement • Experimentation towards greater efficiency 9
  • 10. New Workflows • Visibility and planning • Version control and code review • Testing, testing, and more testing • Metrics collection and interpretation Basically, borrow some stuff from Dev 10 http://websites-development.com/sites/default/files/git_branch_strategy.png
  • 11. New Goals • Transparency - are we working on the right things • Reliability - can we keep it running • Resiliency - can we rebuild it? Do we have the technology? • Correctness - are we sure it’s doing what we want it to do 11 Building Trust
  • 12. More than keeping the lights on. 12
  • 13. Ops Identity Crisis http://www.flickr.com/photos/muffett68/7214428636/ I don’t write code, I’m a sysadmin I have to spend all my time fixing dumb things This takes too much time. These tools are too hard to learn. I will write my own thing. I’m faster if I don’t have to talk to anyone about what’s going on. 13
  • 14. So, some things to work on • Some tools for mitigating risk • Some processes and tips for making the right thing the easy thing • Increase efficiency, learn some stuff, reevaluate your own work • Don’t be afraid of borrowing from other disciplines 14
  • 16. Opportunity Cost The value of the things you could be doing while you were shaving that yak 16
  • 18. RiskVectors • What Ops thinks of as risk • New code, releases, tasks • Other sources of risk • Old products and workflows • Unrepeatable processes • Personal heroics http://www.flickr.com/photos/baresone/4473290629/sizes/z/in/photostream/18
  • 19. Assessment of Risk • Is your process: • well documented • repeatable • reliable • easy to do right? http://www.flickr.com/photos/lemusgro/5494317161/sizes/z/in/photostream/19
  • 20. EASY TO DO RIGHT Seriously. I’m not kidding. 20
  • 21. New Tools • Git and hooks for Ops • Packaging your stuff • Borrowing sanity checks from other places • Basic testing without doing a lot of zomgcoding • Configuration Management 21
  • 22. Tool 1 - Working with git http://mattbanks.me/wp-content/uploads/2012/07/Git-Logo.png22
  • 23. Why git • Distributed version control • Everyone gets a copy • Hub/spoke model for sharing • Simple set up • Easy to run a local git server • Other offerings, like github, are pretty awesome too 23
  • 24. .git/config [core]! ! repositoryformatversion = 0! ! filemode = true! ! bare = false! ! logallrefupdates = true! [remote origin]! ! fetch = +refs/heads/*:refs/remotes/origin/*! ! url = ssh://localhost/srv/git/bindfiles.git! [branch master]! ! remote = origin! ! merge = refs/heads/master! Remote origin! 24
  • 25. $ vi db.192 Example workflow: zonefiles 25
  • 26. Add “wat.local” with final octet 24 Add a new host 26
  • 27. $ git status git status # On branch master! # Changed but not updated:! # (use git add file... to update what will be committed)! # (use git checkout -- file... to discard changes in working directory)! #! #!modified: db.192! #! no changes added to commit (use git add and/or git commit -a) 27
  • 28. git tells you what it wants # Changed but not updated: # (use git add file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: db.192 28
  • 29. $ git add db.192 $ git status git add # On branch master! # Changes to be committed:! # (use git reset HEAD file... to unstage)! #! #! modified: db.192! # 29
  • 30. git commit • git add stages your changes locally • git commit will write them to your local git repository • add your comment either inline with “-m” or git commit will open a buffer for you 30 git commit -m “this commit is awesome”
  • 31. $ git commit git commit 31
  • 32. [master 22371ab] Added wat.local to reverse file! 1 files changed, 4 insertions(+), 0 deletions(-) $ git status # On branch master! # Your branch is ahead of 'origin/master' by 1 commit.! #! nothing to commit (working directory clean) 32
  • 33. Making Good Comments • At least explain what you did, Lucy • If there is a ticket somewhere, add that in the comment • If you made multiple changes, call them all out 33
  • 34. $ git push git push Counting objects: 5, done.! Compressing objects: 100% (3/3), done.! Writing objects: 100% (3/3), 335 bytes, done.! Total 3 (delta 1), reused 0 (delta 0)! To ssh://localhost/srv/git/bindfiles.git! 06fa560..22371ab master - master 34
  • 35. git push • git push sends your changes to the central git server • git pull brings everyone else’s changes into your local repo • Don’t hoard changes; push and pull often 35
  • 36. Ok? • What did we forget to do? 36
  • 37. Update the Serial! • Lots of administration tasks have tribal knowledge you need • Zonefiles have a Serial that needs to be incremented when you make a change • They are potentially outage-causing or hair-pulling problems that can be avoided • Let’s let git remember to do that for us 37
  • 38. commit hooks • You can put hooks into your git repos • Little tasks that happen at various steps in the process • We can add a pre-commit hook to our bindfiles repo • So you don’t have to remember! Saves time later! Helps junior staff! 38
  • 39. $ cp /srv/myrepo/pre-commit .git/hooks $ cat .git/hooks/pre-commit pre-commit #!/bin/bash! num=`git diff master db.192 | grep ^+ | wc | awk '{print $1}'`! if [ $num -gt 1 ] ; then! serial=`git diff master db.192 | grep -i serial`! if [ $? -ne 0 ] ; then ! echo You made a change to the zone file but didn't update the Serial value! exit 1;! fi! fi 39
  • 40. pre-commit • Rather messy, off-the-cuff example • git diff master db.192! • Looks for changes between what’s in the current master on your local repo • If the db.192 file has changed but the value for Serial is the same, it prints and error and exits with a non-zero return code • git stops processing the commit, saving you headaches later 40
  • 42. What else to hook? • Services with config checkers • make a change to the config, run the checker in a hook • nagios, named, apache, etc come with check tools • Other syntax checking • ruby, json, config management tools Make it EASY to do RIGHT 42
  • 43. Other git things • git clone • gives new team members a working copy of the repo • git branch • Lets you and other team members work on non-prod stuff • Use sparingly; less complexity means more reliable results 43
  • 44. Tool 2: fpm • How do you get files, apps, stuff deployed on your hosts? • scp -r? • tarballs? • build everything on every host, you gentoo fans? • crash cart? (omg) 44
  • 45. Packaging • Reap the benefits of what’s built in to your package manager • Versioning • Dependencies • Metadata • Build-once, install-many • File transfer built right into stuff like yum and apt repos! 45
  • 47. Make it easy: fpm • Creating packages from scratch is tedious • There’s some esoteric stuff in the package managers • You really only need a few things 47
  • 48. fpm • fpm,“f’ing package managers”! • Jordan Sissell • Creates multiple kinds of packages from various resources • https://github.com/jordansissel/fpm 48
  • 49. $ fpm -h fpm Intro:! This is fpm version 0.4.37! If you think something is wrong, it's probably a bug! :)! Please file these here: https://github.com/jordansissel/fpm/issues! You can find support on irc (#fpm on freenode irc) or via email with! fpm-users@googlegroups.com! Usage:! fpm [OPTIONS] [ARGS] ...! ! Parameters:! [ARGS] ... Inputs to the source package type. For the 'dir' type, this is the files and directories you want to include in the package. For others, like 'gem', it specifies the packages to download and use as the gem input! ! Options:! -t OUTPUT_TYPE the type of package you want to create (deb, rpm, solaris, etc)! -s INPUT_TYPE the package type to use as input (gem, rpm, python, etc)! -C CHDIR Change directory to here before searching for files omg it just keeps going.... 49
  • 50. Create a package • rpm-ify our zonefiles • They’re in our git repo right now • Live in /var/named for reals • If we package them, we get versioning and other data 50
  • 51. fpm settings • -s dir : we’re working with raw files rather than a gem, rpm, etc • -t rpm : create an rpm package • -v 1.0 : first version! • --prefix=/var/named : where the files will be installed • -n “zonefiles” : name of the package • --after-install /srv/velocity/restart_named.sh : run this after installing • db* : the files to be packaged $ fpm -s dir -t rpm -v 1.0 --prefix=/var/named -n zonefiles --after-install /srv/myfiles/restart_named.sh db* 51
  • 52. $ fpm -s dir -t rpm -v 1.0 --prefix=/var/named -n zonefiles --after-install /srv/velocity/restart_named.sh db* Created rpm {:path=zonefiles-1.0-1.x86_64.rpm} $ rpm -qpl zonefiles-1.0-1.x86_64.rpm /var/named/db.192! /var/named/db.local 52
  • 53. • Nice! • Now we can install it 53
  • 54. $ sudo rpm -ihv zonefiles-1.0-1.x86_64.rpm Results! Preparing... ########################################### [100%]! 1:zonefiles ########################################### [100%]! Stopping named: .[ OK ]! Starting named: [ OK ] $ dig @localhost -x 192.168.1.22 ;; QUESTION SECTION:! ;22.1.168.192.in-addr.arpa.!IN! PTR! ;; ANSWER SECTION:! 22.1.168.192.in-addr.arpa. 604800 IN! PTR!wat.local. 54
  • 55. Put the bits together • Your zonefiles are in a git repo • The repo has syntax and error checking pre-commit hooks • The repo can also have packaging and deploy post- commit hooks • Smooth the process, make the right way the easiest way http://www.flickr.com/photos/62904109@N00/2636859006/sizes/z/in/photostream/55
  • 56. Tool 3:Testing • Lots of work in the dev space • TDD, BDD, test, test • Write tests first, prove they fail, write code to make them pass • More risk reduction • Looks scary http://www.flickr.com/photos/nobleup/3995733415/sizes/z/in/photostream/56
  • 57. basic tests • So, you’re running DNS • What else is do you have? • Monitoring server! http://www.flickr.com/photos/richardmoross/490988453/sizes/z/in/photostream/57
  • 58. What can we borrow? • Nagios plugins! • Extensive set of checks for all sorts of services • Usable from the command line 58
  • 59. $ ls /usr/lib64/nagios/plugins nagios plugins check_breeze check_game check_mrtgtraf check_overcr check_swap! check_by_ssh check_hpjd check_mysql check_pgsql check_tcp! check_clamd check_http check_mysql_query check_ping check_time! check_cluster check_icmp check_nagios check_pop check_udp! check_dhcp check_ide_smart check_nntp check_procs check_ups! check_dig check_imap check_nntps check_real check_users! check_disk check_ircd check_nrpe check_rpc check_wave! check_disk_smb check_jabber check_nt check_sensors eventhandlers! check_dns check_ldap check_ntp check_simap negate! check_dummy check_ldaps check_ntp_peer check_smtp urlize! check_file_age check_load check_ntp.pl check_snmp utils.pm! check_flexlm check_log check_ntp_time check_spop utils.sh! check_fping check_mailq check_nwstat check_ssh! check_ftp check_mrtg check_oracle check_ssmtp! ! Hey! A DNS Checker! 59
  • 60. check_dns • We can use these plugins to test out what we’re doing • Don’t require any additional frameworks or scary things • Many of them work just fine over the network, too 60
  • 61. $ /usr/lib64/nagios/plugins/check_dns -h Check DNS check_dns v1.4.16 (nagios-plugins 1.4.16)! Copyright (c) 1999 Ethan Galstad nagios@nagios.org! Copyright (c) 2000-2008 Nagios Plugin Development Team! ! nagiosplug-devel@lists.sourceforge.net! ! This plugin uses the nslookup program to obtain the IP address for the given host/domain query.! An optional DNS server to use may be specified.! If no DNS server is specified, the default server(s) specified in /etc/resolv.conf will be used.! ! Usage:! check_dns -H host [-s server] [-a expected-address] [-A] [-t timeout] [-w warn] [-c crit]! ! 61
  • 62. $ check_dns -H box.local -s 127.0.0.1 -a 192.168.1.21 When check_dns is ok DNS OK: 0.004 seconds response time. box.local returns 192.168.1.21|time=0.004142s;;;0.000000 $ echo $? 0 62
  • 63. $ check_dns -H box.local -s 127.0.0.1 -a 192.168.1.22 check_dns errors DNS CRITICAL - expected '192.168.1.22' but got '192.168.1.21'! $ echo $? 2 63
  • 64. cool • Now we have a way to test our changes • Behaves in a predictable way • Now let’s add one more component: a test harness 64
  • 65. Tool 4: bats • Bash Automated Testing System • Like all good tools, bats is impossible to google • https://github.com/sstephenson/bats omg these are adorable. http://www.flickr.com/photos/37539972@N06/3980094382/sizes/z/in/photostream/ 65http://www.etsy.com/shop/theitsybitsyspider
  • 66. What the bats • Allows you to test that UNIX programs do what you expect • Write stuff in bash to test other system commands • Easy to get hold of return codes, output • Let’s see an example: checking the nagios configs 66
  • 67. $ bats /srv/myfiles/nagios.bats Using bats 1..1! ok 1 nagios is all good Ran one test All good! 67
  • 68. $ cat /srv/myfiles/nagios.bats #!/usr/bin/env bats! @test nagios is all good {! result=$(sudo service nagios checkconfig)! [ $? -eq 0 ]! } Run a system command! Check the return code! Also grabs output, but we don’t need that here 68
  • 69. We can do this! http://www.flickr.com/photos/usnationalarchives/3678696585/ 69
  • 70. Tool 5: Config Management 70 • Oh you’re totally going to automate yourself out of a job...
  • 71. Why? 71 0 1 2 3 4 5 6 Work To Be Done Work Doable By N Ops Work That Won’t Get Done
  • 72. Features of Config Management • Repeatability - configure services the same way, every time • Reliability - ensure that the services are always configured correctly • Documentation - record of what actions were taken on the system • Idempotent - only take action if necessary 72
  • 73. A Chef Recipe package “named” do! action :install! end! service “named” do! action [:start, :enable]! end! package “zonefiles” do! action :install! notifies :restart, “service[named]”! end! 73
  • 74. CM Tools • Record your configuration into version control • Build hosts in your datacenter, in the cloud, build reals, build virtuals • Support heterogeneous environments • Install packages, write configurations, manage services, users, groups, files, registry settings, etc 74
  • 75. Windows? • Learn. PowerShell. • Then get into DSC • DSC support is coming for CM tools, and will be a powerful way to manage Windows environments 75
  • 76. New Workflows 76 make a change in the cm files check into git git hooks check for errors run a few tests deploy to hosts make a change in the app code check into git git hooks check for errors run a few tests build a package add to artifact repo
  • 77. Our Goals:Transparency • Are we working on something that adds value? 77
  • 78. Our Goals: Reliability • Does our new process keep things running? 78
  • 79. Our Goals: Resiliency • Does our new process make it easy to rebuild, recover, scale? 79
  • 80. Our Goals: Correctness • Does our new process ensure that the work we’re doing is correct? 80
  • 81. Building from here 81 http://www.flickr.com/photos/kalmyket/691478431/sizes/z/in/photostream/
  • 82. Cheaper Resources • Do more real-world testing • Local virtuals - vagrant, cloud providers • Linux containers - docker • Make Dev and QA really look like prod 82
  • 83. Build Server • Jenkins,Travis,Team City, etc • Build and test configs and app code together • Never forget a step in your new workflow! 83 http://www.flickr.com/photos/hubmedia/2141860216/sizes/z/in/photostream/
  • 84. MakeYour Job Better • When your job is better, so is your life • Fewer emergencies, less opaqueness of systems and processes encourages collaboration and shared duty • Be intentional about the things that we do and our goals • Know that what you do day to day is improving 84
  • 85. Takeaways • Reliable, repeatable processes • Make stuff easy to do right • Reduce risk of mistakes, misunderstandings • Reduce the need for personal heroics • Be intentional about the work we do and focus on being valuable http://www.flickr.com/photos/ginnerobot/2877212845/sizes/z/in/photostream/85
  • 86. Thanks! • Thanks for your kind attention • Please keep the conversation going with your teams 86