DevOps Roadtrip NYC

JASON HAND |
DevOps Evangelist
• Over 17 years of experience as a
developer, system administrator,
and support specialist
• Fully emerged into the world of
agile development and the
DevOps movement with Colorado
tech startups
#DevOpsRoadTrip
@jasonhand

Agenda
12:00 - 1:00 p.m. - Lunch
1:00 – 1:15 p.m. – Opening Remarks – Jason Hand
1:15 – 2:05 p.m. – Jason Yee & Aaron Aldrich
2:05 – 2:20 p.m. - Break
2:20 – 3:25 p.m. – Break Out Sessions
3:25 – 3:35 p.m. - Break
3:35 – 4:30 p.m. – Dave Zwieback
4:45 – 6:00 p.m. – Happy Hour

A little about VictorOps…
Incident Management platform combining
real-time data, rich context, automation, and
group collaboration enabling teams to handle
incidents as they occur as well as continuously
improve people, processes, and tools over
time.
#DevOpsRoadTrip

Words are how we think – stories are how we link.
- Christina Baldwin
Oral narrative is and for a long time has been the
chief basis of culture itself.
- John D. Niles
Stories from the road

Jason Yee
Datadog
Technical Writer/Evangelist
Jason is a community builder,
developer, traveler and chef. He
currently works as a technical writer &
evangelist at Datadog. When he’s not
working you can often find him on a
plane trying to accumulate more miles
or in a restaurant cooking.

MONITORING 101JASON YEE, DATADOG @GITBISECT

@GITBISECT
EVANGELIST / DOCUMENTARIAN
DEVOPSDAYS PDX / WEB
TRAVEL HACKER
about me:
@DATADOGHQ
SAAS BASED MONITORING
TRILLION DATA POINTS PER DAY
WE’RE HIRING! BIT.LY/DATADOG-JOBS
about Datadog:

DAMON EDWARDS & JOHN WILLIS - DEVOPSDAY LOS ANGELES
WHAT IS DEVOPS? ▸ Culture
▸ Automation
▸ Metrics
▸ Sharing
TW: @gitbisect @datadoghq

METRICS
CULTURE, AUTOMATION & SHARING ARE GREAT, BUT WHAT ABOUT

Follow
@honest_update
on Twitter

COLLECTING DATA IS CHEAP;
NOT HAVING IT WHEN YOU
NEED IT CAN BE EXPENSIVE
SO INSTRUMENT ALL THE THINGS!

4 QUALITIES OF GOOD METRICS
NOT ALL METRICS ARE EQUAL

1. MUST BE WELL UNDERSTOOD

2. SUFFICIENT GRANULARITY

METRICS 101
HOW GRANULAR?
▸ AWS: “You can send custom metrics to
Amazon CloudWatch as frequently as you like,
but statistics will only be available at one
minute granularity.”
▸ MS Azure: 1m up to 24h; 1h up to 7d; 1d up
to 30d
▸ Google Stackdriver: 1m

3. TAGGED & FILTERABLE

Query Based Monitoring
“What’s the average throughput of
application:nginx per version ?”
“Alert me when one of my pod from replication
controller:foo is not behaving like the others?”
“Show me rate of HTTP 500 responses from nginx”
“… across all data centers”
“… running my app version 2….”

4. LONG-LIVED

METRICS 101
HOW LONG?
▸ AWS: “Metrics data are available for 2 weeks.
For longer term (>2 weeks) ofﬂine storage,
please retrieve your metrics using our
GetMetricStatistics API…”
▸ MS Azure: Minute: 48h; Hour: 30d; Day: 90d
▸ Google Stackdriver: “Time series data points
are kept for 6 weeks and then deleted.”

When to alert?

Cryptic Alerts
W
H
A
T
?

EVERY ALERT MUST BE ACTIONABLE

RECURSE UNTIL YOU FIND THE TECHNICAL CAUSES

A PARTICLE AT REST
OR IN MOTION
CONTINUES IN THAT
STATE UNLESS
COMPELLED BY
FORCES TO CHANGE IT.
Newton’s First Law of
Motion

METRICS
ALERTING & RESOLVING ISSUES REQUIRES THE RIGHT

Aaron Aldrich
Cage Data
Director of Support Services
Aaron Aldrich is the Director of Support Services
and DevOps Evangelist at Cage Data. Following
a career that’s been user-facing and
operationally focussed has given him an
empathetic view of technology’s role in our lives.
Aaron is a founding organizer of the DevOps CT
meetup. His writing can occasionally be found on
cagedata.com/blog if it’s greater than 140
characters or on twitter @crayzeigh if fewer.

Managing Fires
leadership’s role in incident management
@CrayZeigh

Aaron Aldrich
@CrayZeigh
DevOps Consultant
Cage Data, Inc.
cagedata.com/devops/

use your tools for little fires
@CrayZeigh

escalation
processes
@CrayZeigh

crisis requires a team and
leadership
@CrayZeigh

bestow
leadership
before
crisis
@CrayZeigh

crisis leaders should be authorized to make
important business decisions
@CrayZeigh

delegate authority as far as
possible
@CrayZeigh

Troubleshooting
101
@CrayZeigh

“Never let a serious crisis go to waste. What I mean
by that it’s an opportunity to do things you think
you could not do before.”
-Rahm Emanuel
@CrayZeigh

take it incrementally
@CrayZeigh

what will make it work?
@CrayZeigh

what will make it whole?
@CrayZeigh

“Brevity is the
soul of wit.
Specificity is
the soul of
narrative
[getting work
done]”
@CrayZeigh

make work
visible to all
@CrayZeigh

leaders handle human
problems
@CrayZeigh

More on the plate,
less on the mind
@CrayZeigh

make sure team
members are available
to make substitutions
@CrayZeigh

tell people to stop
working and get
rest
@CrayZeigh

public communication
@CrayZeigh

communicate
quickly
@CrayZeigh

update
your
customers
often
@CrayZeigh

empower your team,
remove barriers
@CrayZeigh

Breakout Sessions
◻ Burnout – Aaron Aldrich
◻ Why + How ChatOps? – Jason Hand
◻ Learning Reviews – Dave Zwieback
◻ Monitoring – Jason Yee
#DevOpsRoadTrip

Dave Zwieback
x.ai
VP of Engineering and Data Science
Dave Zwieback has been working with complex,
mission-critical I.T. services and teams for two
decades. His career spans small high-tech
startups, non-profits, and behemoth engineering,
financial services, and pharmaceutical firms. He
is currently the VP of Engineering and Data
Science at x.ai. Dave is the author of “Beyond
Blame: Learning From Failure and Success”
(http://bit.ly/beyondblame). You can follow follow
Dave @mindweather.

What we really want is to..
Continuously
Improve

Automatio
n
Awarenes
s
Collaboratio
n
Documentatio User Empathy
Learnin
g
Key Themes

The systems we engineer, maintain, and improve are
Complicated
.. or ..
Known
unknowns

The systems we engineer, maintain, and improve are
ComplexUnknown
unknowns

Failure in a complex
system is inevitable which
is EXACTLY why learning
from it is so critical.

Failure not seen as opportunity to learn
Source: “Six Trends Shape DevOps Adoption, Q1 2015”, Forrester report

Awareness
http://blog.vmware.co

© 2015 Forrester Research, Inc. Reproduction Prohibited 19
Single Source Of Truth Lacking In Many
Orgs – 95% only most of the time or less
Source: April 15, 2015 “Six Trends That Will Shape DevOps Adoption”, Forrester report

Collaboration
http://neolivemarketing.com/wp-content/uploads/2015/09/Collaboration.jpg

Teams siloed throughout life cycle

User Empathy
https://open.buffer.com/wp-content/uploads/2015/12/empathy3.jpg

Automation
http://thelifedesignproject.com/wp-content/uploads/2009/09/373881476_217d24ef6d.jpg

Delays In Notifications Leads To
Customers Finding the Problem First

Reduce MTTR
State of DevOps Report (2015)
– by Puppet Labs

TimeToRepair(TTR)
Continuous Improvement Efforts
Reactive (0 – 4)
(chaotic)
Tactical (5 – 9)
(obvious)
Integrated (10 -14)
(complicated)
Strategic (15 – 18)
(complex)
✓ No automation
✓ No operational stack
awareness
✓ Poor collaboration between
teams (Dev & Ops)
✓ Documentation not available
✓ No standardized
communication
✓ High focus on consistent
continuous learning
✓ Uses a NOC
✓ Some monitoring & alerting
instrumentation
✓ Collaboration in crisis
✓ "Mission critical" processes are
available
✓ Understood crisis
communication protocols
✓ Remediation data available to
IT Operations
✓ Team rotations, paging
policies, role hunting
✓ Continuous improvement of
key health indicators
✓ Technical collaboration across
all incidents
✓ Docs up to date and easily
accessible
✓ Consistent real-time
communication practices
✓ Automated docs and remediation
✓ Actionable Alerts with full context
✓ High collaboration among all teams
✓ Documentation part of remediation
✓ Targeted, proactive crisis comms
✓ High focus on continuous learning
Incident
Management
Maturity

2016 State Of On-call Survey:
jhand.co/SOOC2016

Happy Hour
Co-Sponsored by
#DevOpsRoadTrip

DENVER - SEATTLE - SAN FRANCISCO - MINNEAPOLIS - NEW YORK CITY

DevOps Roadtrip NYC

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a DevOps Roadtrip NYC

Similar a DevOps Roadtrip NYC (20)

Último

Último (20)

DevOps Roadtrip NYC