An old presentation from 2012 that I delivered at QCON. Somewhat relevant in today's world of Continuous Delivery.

1. Dev
OPS
Stephen Burton,Tech Evangelist
@BurtonSays
What can DevOps Learn from F1?

2. • App Support
• Developer
• Systems Engineer
• Pre-Sales
• Product Management
• Tech Evangelist
About Me

3. 2011 Formula 1 World Champion
SebastianVettel
Who is this?

4. • 20 Races on the World Calendar
• 12Teams, 24 Cars & Drivers
• Cost is ~$7.5 Million per car
• Spend $75 - $300 Million per year
• 0-100-0 in < 5 seconds
Formula 1

5. • F1 car has 80,000 components,1KM wiring &100+ sensors
• 99.9% assembled correctly still means 80 POF’s
• Monaco GP - Manholes are welded down
• Brakes on F1 Car is like hitting a brick wall at 200mph (5G)
• Downforce a baby elephant sitting on drivers head
Interesting F1 Facts

6. “DevOps is about being agile,
and going from
A-Ha to Cha Ching
as quickly as possible”
John Willis
DevOps Evangelist
Why F1 and DevOps?

7. Innovation. Agility. Change.

8. Simple?

10. Hidden Complexity

11. Cutting Edge Technology

12. Cool Innovation

13. Simple?

14. Active
Directory
Tibco BW
3rd Party
Web Service
Oracle
Cassandra
Tomcat
Tomcat
JBoss ASP.NET
3rd Party
Web Service
SQL
Server
3rd Party
Java App
JBoss
JDBC
SOAP
ADO.NET
HTTP
SOAP
SOAP
LDAP
ADO.NET
THRIFT
JMS THRIFT
JMS
HTTP
Hidden Complexity

15. REALLY
Really.

16. Really. Really.

17. Seriously.

18. What you can’t see...

19. Cool Integration

20. SUCCESS IN FORMULA 1
is Winning Races
Success.

21. Ideas & Agility Wins Races

22. Dev
Developers want
Agility & Change
OPS
Operations want
Availability & Stability
WHAT IS SUCCESS FOR DEVOPS ?

23. COMPARINGTEAMS
Formula 1
•Driver
•Sponsors
•Architects
•Designers
•Engineers
•Testing
•Operations
•Support
Applications
•End User
•Business
•Architects
•Designers
•Developers
•Testing
•Operations
•Support

24. MANAGING CHANGE
ISN’T EASY
Design > Build >Test > Deploy > Support
Slowdowns
Outages
Mis-configuration
Bottlenecks
Bugs
End User
Complaints
Business
Requirements
Project
Plans
More Business
Requirements

25. CHANGE
Doesn’t Happen OvernightThough

26. Change or Your Competition Will

27. • Cars Evolve
• Up to 30 new parts per race
F1 Teams Collaborate
•Engineering
• Aero, Engine,Transmission, ....
•Operations
• Mechanics,Telemetry, Pit Crew

28. Change is Constant

29. Develop Test SupportDeployDesign
Operations work hand in hand with Engineering
Race Weekend
Formula 1 Lifecycle

30. MEASURING SUCCESS
• Downforce
• Performance
• Reliability
• Race Wins
Telemetry &
Monitoring
Driver Results
Monitoring is critical to Managing Change

31. End User Experience is Critical

32. Monitoring is also Critical

33. Measure Impact

34. The NOC

35. MONITOR & MANAGE
IMPACT OF CHANGE
Fast Slow Fail

36. Tools are Important

37. SO, DOYOU KNOW
HOW FAST AND RELIABLE
YOUR CODE IS
IN PRODUCTION?
<1% of Developers actually do

38. CODE CAN BE
FUNCTIONALLY PERFECT
Text
But how Fast and Reliable will it be?

39. Concurrency DataVolume Resource
3 Things
Impact Application Performance

40. DEV
DataVolume ResourceConcurrency
Very Fast
QA/TEST
DataVolume ResourceConcurrency
Fast
PRODUCTION
DataVolume ResourceConcurrency
?
THE PROBLEM

41. F1TESTING
• 3 Pre-seasonTests (Integration & Soak testing)
• 1 Mid-seasonTest (For Ferrari to catchup)
• WindTunnel & Simulator (Unit & Functional & LoadTesting)
• Race Weekends (ProductionTesting)

42. HANDS UP IFYOU DO
PERFORMANCE TESTING
ON EVERY RELEASE
(Its tough with Agile cycles)

43. SUCCESS
ISN’T HANDING OVER A
RELEASETO OPS
DEPLOY
ME

44. GIFTS CAN HAVE SURPRISES
NOBODY LIKES SURPRISES

45. DEV SUCCESS
ISN’TTHE APP LASTING
1 HOUR IN PRODUCTION

46. • Ops has a 48 hour warranty period with Dev
• If the App experiences consistent slowdowns or outages
• AllTickets get re-assigned to Development
A DevOps Warranty?

47. REAL-LIFE SURPRISESWhen Things go Wrong

48. BLUE - Concurrency (Trx/min)
GREEN - Avg ResponseTime
RED - Resource (CPU %)
Dealing with Spikes

49. BLUE - Concurrency (Trx/min)
GREEN - Avg ResponseTime
Not Dealing with Spikes

50. T
Po
Po
T
Production Ground to a halt for 2 hours
And again the next day
Who owns JVM Configuration? Ops or Dev?
Not Learning from Failure

51. Whose responsible for this? Ops?
Bad Implementation

52. Whose responsible for this? Ops?
Missing Configuration

53. Whose responsible for this? Ops?
Inefficient Code

54. Root Cause: Cache wasn’t thread safe
46,463 Checkouts in the Day
2,492 were impacted
Avg. $100 per Checkout
Cost of Deadlock: ~ $249,000
Code Deadlock

55. IMAGINE ENGINEERS
WITH NO RACEVISIBILITY
Develop Test SupportDeployDesign
Car would never win races or finish
Race Weekend

56. Before
After

57. Before
After

58. • # Agile Releases or % Availability is pointless
• End User Experience?
• BusinessTransactionThroughput?
• MTBF and Business Impact?
DevOps Need Common Goals

59. • Feedback Loop from Ops to Dev
• Share Goals,Tools & Metrics
• Continuous Improvement
• Learn from Failure
• Win and LooseTogether
Lessons Learned
•Don’t write slow SQL Queries
•Remember to use caching
•Don’t rely on ORM
•Assume everything will fail
•Logging everything is not a good idea
•Remember to do performance testing
DevOps - Collaborate & Learn

60. • Define Success
• Embrace Change, don’t fear it
• Monitor, Measure and Manage Impact
• Learn from Failure
• Have Fun and remember
Top Tips for DevOps

61. DevOps is about Team Work

62. +
Visit the AppDynamics Booth
FreeT-shirt
Free Application Monitoring
in Production
www.appdynamics.com

63. We’re Hiring!