In the highly regulated environment of automotive, software quality can be difficult but it doesn't need to be. ESG partners with software teams to improve their team's performance through developer operations. From culture to tool integrations, ESG takes a holistic approach to help teams measurably improve their software development lifecycle and the quality of its output.

1. Software Quality for the Future
of Connected Vehicles and IoT
Devon Bleibtrey
Director of Technology - ESG
Twitter: @devonbleibtrey
Email: devon.bleibtrey@esg-usa.com

2. Bit About Me
● Director of Technology @ ESG
● Founder of NextRelease.io
● 15+ Years Experience in Developer Operations
● 10 Years Experience in Automotive Industry
● DevOps Rollouts @ Fortune 50s & High Tech Startups
● Bachelors of Computer Science - Central Michigan University

3. Infotainment systems have become the
#1 source of customer complaints
about new vehicles.
- Consumer Reports Auto Reliability Survey
“Infotainment Systems | Car Connectivity - Consumer Reports.” Infotainment Systems | Car Connectivity - Consumer Reports,
www.consumerreports.org/cro/magazine/2014/02/high-tech-automotive-headaches/index.htm.

4. Auto software is becoming increasingly complex.
Itu. (2017, August 28). Is software taking over the auto industry? Retrieved from https://news.itu.int/software-taking-over-auto-industry/

5. Software is taking over the Auto Industry
Itu. (2017, August 28). Is software taking over the auto industry? Retrieved from https://news.itu.int/software-taking-over-auto-industry/

6. The majority of deployments are still manual.
("DevOps by the Numbers: DevOps, Self-Service Infrastructure, and Deployment Automation - DZone DevOps" 2017)
71%
Manual & Auto
23%
Manual Full Automation
6%

7. MANUAL DEPLOYMENT: NIGHTMARE SCENARIO
A software update is
made and then
manually flashed onto
a set of vehicles.

8. MANUAL DEPLOYMENT: NIGHTMARE SCENARIO
Quality assurance and
validation teams
learn of a new
serious issue.

9. MANUAL DEPLOYMENT: NIGHTMARE SCENARIO
Developers receive logs
from QA teams,
scrambling to apply a
quick fix.

10. MANUAL DEPLOYMENT: NIGHTMARE SCENARIO
Compounding issues
develop while “fixing”
on the fly.

11. OVER THE AIR UPDATES & CONTAINERS
More software in the car &
over the air updates requires
fundamental cultural &
infrastructure changes to
ensure we meet the demand
without drowning in bugs &
damaging quality.

12. WHERE ARE WE AT TODAY IN BEING READY
What’s our starting point and where do we need
to get to, so we can enable this change?

13. HOW DOES AUTOMOTIVE MEASURE UP IN SOFTWARE
Average Automotive
Software Team
For the primary application or service you work on, what is your lead time for
changes (i.e. how long does it take to go from code commit to code successfully
running in production)?
For the primary application or service you work on, how often does your
organization deploy code?
2018 State of DevOps Report. (n.d.). Retrieved from
https://puppet.com/resources/whitepaper/2018-state-of-devops-report

14. HOW DOES AUTOMOTIVE MEASURE UP IN SOFTWARE
2018 State of DevOps Report. (n.d.). Retrieved from
https://puppet.com/resources/whitepaper/2018-state-of-devops-report
Average Automotive
Software Team
For the primary application or service you work on, how long
does it generally take to restore service when a service incident
occurs (e.g. unplanned outage, service impairment)
For the primary application or service you work on, what percentage of changes
results either in degraded service or subsequently requires remediation (e.g., leads
to service impairment, service outage, requires a hotfix, rollback, fix forward, patch)?

15. THE ROAD TO HIGH PERFORMANCE SOFTWARE TEAMS
2018 State of DevOps Report. (n.d.). Retrieved from https://puppet.com/resources/whitepaper/2018-state-of-devops-report
2,604x faster recovery times from issues
7x lower change failure rates
22% less time on unplanned work or rework
46x more frequent software deployments
2,555x faster software iteration times
Elite software teams see:

16. THAT’S GREAT, NOW HOW DO I GET THERE
How do you achieve high performance output
to enhance my software quality?

17. TOOLS, TOOLS, & BUZZ WORDS
“We just need to be more Agile and eliminate Waterfall”
“All we need, is to do work in 2-week
Sprints”
“If we only had a better ticket manager.
We should bring in JIRA”
“Everyone is using GitHub, if we house
our code there it’ll fix our problems”
“We really need traceability across the
V, let’s get IBM Rational”
“If we had Unit Tests & Static Code
Analysis it’d solve our software quality
issues”
“If we could only Deploy Daily it’d let
us fix things faster, we should
implement CI/CD”

18. SHIFT CULTURE TO PERFORMANCE-ORIENTED
When researchers at Google studied over 180 engineering teams, they
found that the most important factor in predicting a high- performing
team is psychological safety, or feeling safe taking risks around your
team. This was followed by dependability, structure and clarity of work,
meaning, and impact.
- State of DevOps 2018

19. DON’T MISTAKE METHODS FOR PRINCIPLES
Tools & Methods can be nitrous to a well tuned engine.
But missing principles and adding tools without
objectives and key results can easily become cinder
blocks around your waist.
They can mask fundamental problems and prolong the
amount of time before they are resolved.

20. I caution against assuming that tools are the
key to a successful transition to a DevOps
culture. As Sun Tzu wrote: "Tactics without
strategy is the noise before defeat".
“
Mike D. Kail, CTO at Everest.org

21. BACK TO THE BASICS: DEVOPS IS CULTURAL, NOT JUST A WORKFLOW STRATEGY
1. What are your business objectives & compliance requirements?
2. How do you measure success of those objectives?
3. Which measurements of success are most impacted by your
developer operations?
4. How is your culture oriented to facilitate achieving that success?
5. What change can you make, without a new tool, that will make a 10x
impact on your software quality?

22. EVERY COMPANY HAS UNIQUE CHALLENGES
Every company has unique challenges & is at a
different point in their DevOps journey, but here
are a few ideas to make a 10x impact

23. OBJECTIVES & KEY RESULTS
● Define clear objectives and how you
measure success towards them
● Be transparent about what the objectives
and key results are to your entire team
● Empower teams and individuals to define
how they can help the company achieve
those goals
● Limit focus to 3-5 objectives and iterate
● Add tools to measure, automate, and
augment your processes as necessary
WORKPATH: OKRs Objectives and Key Results

24. INTROSPECTION & PROCESS ITERATION - ELIMINATE STAGNATION
● No process is ever perfect
● Coordinate cadenced team and company
wide process retrospectives
● Identify what is working & what is not
● Experiment with new processes where things
are not working and measure their
effectiveness
● Add tools to measure, automate, and
augment your processes as necessary

25. SHIFT TO GENERATIVE CULTURE
● Work to break down barriers and
encourage cross functional teams
● Share risks across your teams
● Innovate on how you can empower
experimentation & learning while
maintaining compliance
○ Sandbox Teams
○ Early Development Prototype Teams
○ Beta User Group Deployments
○ Focus Group Deployments
● Helpful Guides on Getting Started:
○ Harvard Business Review
○ Strategy&
○ The LEAN Startup
● Add tools to measure, automate, and
augment your processes as necessary
“A Typology of Organisational Cultures” BMJ Quality & Safety 2004 by Professor R Westrum from
the Department of Sociology at Eastern Michigan University.

26. TECHNOLOGY: THE ACCELERANT
● Prerequisite: Continuous Introspection, Learning, and Improvement
● Identify objectives and key results then iterate on process, learn what works
for your company / team, and add tooling as necessary to meet objectives
● Do NOT try to do the entire pipeline at once
○ Higher learning curve
○ More resistance to change
○ Higher probability to reverting to past habits
○ Extremely difficult to identify what changes are having what impact
● Focus on the human element and positive habit forming

27. It’s OK to start at your own speed.
Process innovation doesn’t happen overnight.
The Puppet 2018 State of DevOps Report
reveals a set of core practices based on
sharing to start focusing on right now.
“2018 State of DevOps Report.” Puppet, puppet.com/resources/whitepaper/state-of-devops-report.

28. Thank you for your time today!
If you’d like a free developer operations audit and
recommendation report, with tailored tooling suggestions,
send me a request.
Contact Us:
devon.bleibtrey@esg-usa.com
THANK YOU!