3. @joshcorman
@RealGeneKim
Session
ID:
Session
Classifica0on:
Josh Corman, Gene Kim
VERY ROUGH 1ST Draft
Security is Dead.
Long Live Rugged DevOps:
IT at Ludicrous Speed…
CLD-106
Intermediate
11. @joshcorman
@RealGeneKim
• The
The Cavalry isn’t coming… It falls to usı
Problem
Statement
Our
society
is
adop0ng
connected
technology
faster
than
we
are
able
to
secure
it.
Mission
Statement
To
ensure
connected
technologies
with
the
poten0al
to
impact
public
safety
and
human
life
are
worthy
of
our
trust.
Collec9ng
exis0ng
research,
researchers,
and
resources
Connec9ng
researchers
with
each
other,
industry,
media,
policy,
and
legal
Collabora9ng
across
a
broad
range
of
backgrounds,
interests,
and
skillsets
Catalyzing
posi0ve
ac0on
sooner
than
it
would
have
happened
on
its
own
Why
Trust,
public
safety,
human
life
How
Educa0on,
outreach,
research
Who
Infosec
research
community
Who
Global,
grass
roots
ini0a0ve
What
Long-‐term
vision
for
cyber
safety
Medical
Automo0ve
Connected
Home
Public
Infrastructure
I Am The Cavalryı
12. @joshcorman
@RealGeneKim
The Rugged Manifesto
I am rugged... and more importantly, my code is rugged.
I recognize that software has become a foundation of our
modern world.
I recognize the awesome responsibility that comes with
this foundational role.
I recognize that my code will be used in ways I cannot
anticipate, in ways it was not designed, and for longer
than it was ever intended.
I recognize that my code will be attacked by talented and
persistent adversaries who threaten our physical,
economic, and national security.
I recognize these things - and I choose to be rugged.
I am rugged because I refuse to be a source of
vulnerability or weakness.
I am rugged because I assure my code will support its
mission.
I am rugged because my code can face these challenges
and persist in spite of them.
I am rugged, not because it is easy, but because it is
necessary... and I am up for the challenge.
13. @joshcorman
@RealGeneKim
The Rugged Manifesto
I am rugged... and more importantly, my code is rugged.
I recognize that software has become a foundation of our
modern world.
I recognize the awesome responsibility that comes with
this foundational role.
I recognize that my code will be used in ways I cannot
anticipate, in ways it was not designed, and for longer
than it was ever intended.
I recognize that my code will be attacked by talented and
persistent adversaries who threaten our physical,
economic, and national security.
I recognize these things - and I choose to be rugged.
I am rugged because I refuse to be a source of
vulnerability or weakness.
I am rugged because I assure my code will support its
mission.
I am rugged because my code can face these challenges
and persist in spite of them.
I am rugged, not because it is easy, but because it is
necessary... and I am up for the challenge.
14. @joshcorman
@RealGeneKim
Our Goals
§ Play Mad Chemists
§ The Best & Brightest of DevOps
§ The Best & Brightest of Security
§ Cause High Value / High Connection
§ Merge our Tribes for Mutual Awesomeness
§ Catalyze New Patterns and Solutions
23. @RealGeneKim
10 deploys per day
Dev & ops cooperation at Flickr
John Allspaw & Paul Hammond
Velocity 2009
Source: John Allspaw (@allspaw) and Paul Hammond (@ph)
25. Little bit weird
Sits closer to the boss
Thinks too hard
Pulls levers & turns knobs
Easily excited
Yells a lot in emergencies
Source: John Allspaw (@allspaw) and Paul Hammond (@ph)
33. @RealGeneKim
High Performers Are More Agile
30x 8,000x
more frequent
deployments
faster lead times
than their peers
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
34. @RealGeneKim
High Performers Are More Reliable
2x 12x
the change
success rate
faster mean time
to recover (MTTR)
Source: Puppet Labs 2013 State Of DevOps: http://puppetlabs.com/2013-state-of-devops-infographic
35. @RealGeneKim
High Performers Win In The Marketplace
2x 50%more likely to
exceed profitability,
market share &
productivity goals
higher market
capitalization growth
over 3 years*
Source: Puppet Labs 2014 State Of DevOps
37. @RealGeneKim
“As a lifelong Ops practitioner, I know
we need DevOps to make our work
humane.
In the past, I’ve worked every holiday, on
my birthday, my spouse’s birthday, and
even on the day my son was born.”
Nathan Shimek
Engineering Manager, New Context
@nathan_shimek
39. @RealGeneKim
The First Way: Outcomes
§ Creating single repository for code and environments
§ All Ops artifacts in version control
§ Determinism in the release process
§ Consistent Dev, Test and Production environments, all properly
built before deployment begins
§ Developers checking in code daily, being productive
§ Automated regression testing
§ Features being deployed daily without catastrophic failures
§ Decreased lead time
§ Faster cycle time and release cadence
40. @RealGeneKim
The Second Way: Outcomes
§ Peer review of code and environment changes
§ Disciplined automated testing enabling many simultaneous
small, agile teams to work productively
§ Proactive monitoring of the production environment
§ Defects and security issues getting fixed faster than ever
§ High trust culture
§ All groups communicating and coordinating better
§ Everybody is getting more work done
44. @joshcorman
@RealGeneKim
§ we’ve seen what true integration of infosec into
the daily work of Dev and Ops; and it is good
§ key learnings of the DevOps Enterprise 2015
§ Ed Bellis example: Capital One: DevOpsSec
§ examples of practices: preventive, detective/
corrective
49. @joshcorman
@RealGeneKim
DevOps Enterprise: Lessons Learned
§ On Oct 21-23, we held the DevOps Enterprise
Summit, a conference for horses, by horses
§ Speakers included leaders from:
§ Macy’s, Disney, GE Capital, Blackboard, Telstra, US
Department of Homeland Security, CSG, Raytheon,
Ticketmaster, Union Bank of California
50. @joshcorman
@RealGeneKim
Observations
§ They were using the same technical practices
and getting the same sort of metrics as the
unicorns
§ Target: 10+ deploys per day, < 10 incidents per month
§ Capital One: 100s of deploys per day, lead time of
minutes
§ Macy’s: 1,500 manual tests every 10 days, now 100Ks
automated tests run daily
§ Nationwide Insurance: Retirement Plans app (COBOL
on mainframe)
51. @joshcorman
@RealGeneKim
Observations
§ The transformation stories are among the most
courageous I’ve ever heard –
§ Often the transformation leader was putting themselves
in personal jeopardy
§ Why? Absolute clarity and conviction that it was the
right thing for the organization
*
56. @joshcorman
@RealGeneKim
Risk I/O DevOps By the Numbers
Small & Frequent Commits
• Average between 75 & 125
commits commits to Master/week
• Simplicity is your friend
57. @joshcorman
@RealGeneKim
Risk I/O DevOps By the Numbers
Small & Frequent Commits
• Average between 75 & 125
commits commits to Master/week
• Simplicity is your friend
Security Automation at Risk I/O
Chef All the Things!
Test All the Things! (including security)
Static + Dynamic Throughout
Continuous Integration via CircleCI
Open-Sourced Cookbooks
ModSecurity
(airbag)
Nessus (air bag ctrl) Nmap
(brakes)
SSH
iptables
(shoulder belt)
encrypted volumes Duo 2FA openVPN
ChatOps = Slack + graphite + logstash + sensu + pagerduty
58. @joshcorman
@RealGeneKim
Risk I/O DevOps By the Numbers
Small & Frequent Commits
• Average between 75 & 125
commits commits to Master/week
• Simplicity is your friend
Security Automation at Risk I/O
Chef All the Things!
Test All the Things! (including security)
Static + Dynamic Throughout
Continuous Integration via CircleCI
Open-Sourced Cookbooks
ModSecurity
(airbag)
Nessus (air bag ctrl) Nmap
(brakes)
SSH
iptables
(shoulder belt)
encrypted volumes Duo 2FA openVPN
ChatOps = Slack + graphite + logstash + sensu + pagerduty
DevOps as a Compliance Enabler
Automation as Evidence & Doc
Cookbooks
Leveraging the ELK Stack
Elasticsearch
Logstash
Kibana
Github + Code Climate + Risk I/O
Compliance Automation Extra Credit: https://
telekomlabs.github.io/
@Eellis
59. @RealGeneKim
The
DevOps
Audit
Defense
Toolkit
h]p://bit.ly/DevOpsAudit
James
DeLuccia
IV
Jeff
Gallimore
Gene
Kim
Byron
Miller
60. @RealGeneKim
Breaking The Bottlenecks In The Flow
§ Environment creation
§ Code deployment
§ Test setup and run (mention @rohansingh)
§ Overly tight architecture
§ Development
§ Product management
65. @joshcorman
@RealGeneKim
§ outline concrete tangible things that can be done together to fulfill it
§ Accelerating to transition from here to there
§ Deming -> SW Supply Chain Rigor
§ Better/Fewer suppliers.
§ Better Supply
§ Traceability/Visibility throughout for Prompt/Agile recall
§ “Congressional Bill” - now or never (Jim Routh)
§ Expanding the DevOps Enterprise community
§ we can have mutual benefit through DevOps and software supply chains
§ legislation
67. @joshcorman
@RealGeneKim67
4/20/15
Product Vulnerability Disclosures Following the HeartBleed Announcement (Circle Size Indicates CVSS Severity Score)
F5
New OpenSSL Disclosures (Both CVSS Level 10)
Here
IBM
Cisco
IBM
McAfee
Initial 'HeartBleed' OpenSSL Disclosure (CVSS Level 5 (underscored))
NumberofProductsIncludedinAnnouncement
0
10
20
30
40
50
60
70
80
90
100
110
120
Days Since HeartBeed Announcement
0 10 20 30 40 50 60 70 80 90 100 110 120
X
Axis:
Time
(Days)
following
ini0al
HeartBleed
disclosure
and
patch
availability
Y
Axis:
Number
of
products
included
in
the
vendor
vulnerability
disclosure
Z
Axis
(circle
size):
Exposure
as
measured
by
the
CVE
CVSS
score
COMMERCIAL
RESPONSES
TO
OPENSSL
69. @joshcorman
@RealGeneKim
True Costs & Least Cost Avoiders
ACME
Enterprise
Bank
Retail
Manufacturing
BioPharma
Educa0on
High
Tech
Enterprise
Bank
Retail
Manufacturing
BioPharma
Educa0on
High
Tech
Enterprise
Bank
Retail
Manufacturing
BioPharma
Educa0on
High
Tech
74. @joshcorman
@RealGeneKim
ON
TIME.
Faster
builds.
Fewer
interrup9ons.
More
innova9on.
ON
BUDGET.
More
efficient.
More
profitable.
More
compe99ve.
ACCEPTABLE
QUALITY/RISK.
Easier
compliance.
Higher
quality.
Built-‐in
audit
protec9on.
76. @joshcorman
@RealGeneKim
ON
TIME.
Faster
builds.
Fewer
interrup9ons.
More
innova9on.
ON
BUDGET.
More
efficient.
More
profitable.
More
compe99ve.
ACCEPTABLE
QUALITY/RISK.
Easier
compliance.
Higher
quality.
Built-‐in
audit
protec9on.
Agile
/
CI
78. @joshcorman
@RealGeneKim
ON
TIME.
Faster
builds.
Fewer
interrup9ons.
More
innova9on.
ON
BUDGET.
More
efficient.
More
profitable.
More
compe99ve.
ACCEPTABLE
QUALITY/RISK.
Easier
compliance.
Higher
quality.
Built-‐in
audit
protec9on.
DevOps
/
CD
Agile
/
CI
80. @joshcorman
@RealGeneKim
ON
TIME.
Faster
builds.
Fewer
interrup9ons.
More
innova9on.
ON
BUDGET.
More
efficient.
More
profitable.
More
compe99ve.
ACCEPTABLE
QUALITY/RISK.
Easier
compliance.
Higher
quality.
Built-‐in
audit
protec9on.
SW
Supply
Chain
DevOps
/
CD
Agile
/
CI
82. @joshcorman
@RealGeneKim
Toyota
Advantage
Toyota
Prius
Chevy
Volt
Unit
Cost
61%
$24,200
$39,900
Units
Sold
13x
23,294
1,788
In-‐House
Produc0on
50%
27%
54%
Plant
Suppliers
16%
(10x
per)
125
800
Firm-‐Wide
Suppliers
4%
224
5,500
Comparing the Prius and the Volt
84. @joshcorman
@RealGeneKim
H.R. 5793 “Cyber Supply Chain Management and Transparency Act of 2014”
§ Elegant Procurement Trio
1) Ingredients:
§ Anything sold to $PROCURING_ENTITY must provide a Bill of
Materials of 3rd Party and Open Source Components (along with
their Versions)
2) Hygiene & Avoidable Risk:
§ …and cannot use known vulnerable components for which a
less vulnerable component is available (without a written and
compelling justification accepted by $PROCURING_ENTITY)
3) Remediation:
§ …and must be patchable/updateable – as new vulnerabilities will
inevitably be revealed
86. @joshcorman
@RealGeneKim
Want More Learn More?
To receive the following:
§ A copy of this presentation
§ The 140 page excerpt of The Phoenix Project
§ Videos and slides from DevOps Enterprise 2014
§ Information on DevOps Enterprise 2015
§ Link to the DevOps Audit Defense Toolkit
§ Announcement of The Phoenix Project audiobook
§ See early drafts of our upcoming DevOps Cookbook
Just pick up your phone, and send an email:
To: realgenekim@SendYourSlides.com
Subject: devops
realgenekim@SendYourSlides.com
devops