In just three years, Square has achieved ubiquitous recognition for mobile card processing grossing over $10B a year in credit card transactions. At the heart of Square's technology are many financial systems which must operate safely, correctly, and sustain rapid growth. During this tech talk, Pascal will describe the concept of an immune system, go over best practices, share lessons leaned and provide a detailed layering of best practices. This talk will cover non-controversial topics such ad TDD, but from new angles. We'll also cover emerging practices like continuous deployment, and softer areas such as engineering management practices geared towards safety. You'll come out of this session with a fresh perspective on how to build software.

1. The Hard and Soft Skills
Required to Avoid Outages
@pascallouis

6. Code Bless Ship Maintain Profit! git rm

7. Tactics
• Entity bound ids (e.g. Id<T>)
• Textual ids MWDN-YP89-OLVL-USER
• Testable configurations
• etc.
Code Bless Ship Maintain Profit! git rm

8. TDD
• Not controversial (anymore)
• Living code documentation
• Enables collaboration
• Technique to encode invariants
Code Bless Ship Maintain Profit! git rm

9. Gold Tests
• Tests which can be changed by a (small)
subset of engineering
• Enforced via policy or technology
Code Bless Ship Maintain Profit! git rm

10. Expressive Tests
• “Change your language and you change
your thoughts” — Karl Albrecht
• Can be implementation agnostic
Code Bless Ship Maintain Profit! git rm

11. ...
Given feed PaymentEventFeedListener receives:
"""
{
"payment_id": "EPT-300",
"isTivoReplay": false,
"merchant": {
"token": "m-1"
},
...
}
"""
Then expect table balance_changing_events order by id:
| event_type | status | process_attempts |
| HOLD | UNPROCESSED | 1 |
| CAPTURE | UNPROCESSED | 0 |
When then the time is 2012-01-06 17:10:00
And balance changing event queue processes items
Then expect table balance_changing_events order by id:
| event_type | status | process_attempts |
| HOLD | UNPROCESSED | 2 |
| CAPTURE | PROCESSED | 1 |
Code Bless Ship Maintain Profit! git rm

12. Automated
Manual
Quality
or or
Time
Code Bless Ship Maintain Profit! git rm

13. Code Analysis
• In theory: static vs dynamic
• In practice: pre vs post-production
Code Bless Ship Maintain Profit! git rm

14. Pre Analysis
• Type Checking
• Testing, CI
• Linters
• Forbidden Call Analysis
Code Bless Ship Maintain Profit! git rm

15. Post Analysis
• Logging
• Metrics
• Invariant Checking
Code Bless Ship Maintain Profit! git rm

16. Alerting & Reporting
Sign
Res
pon
al
Precise Imprecise
se
Immediate Alert Oups!
Deferred Report Report
Code Bless Ship Maintain Profit! git rm

17. Metrics vs Checks
OK
?
WARNING
1
0
200ms
0ms
Code Bless Ship Maintain Profit! git rm

18. Code Bless Ship Maintain Profit! git rm

19. Post-Mortem
• When Anytime there are issues!
• Why Learn and avoid mistakes of the past
• How Blameless
Code Bless Ship Maintain Profit! git rm

20. Post-Mortem
• Go through the timeline
• The Good, The Bad and the Ugly
• Action Items
Code Bless Ship Maintain Profit! git rm

21. Root Cause Analysis
Code Bless Ship Maintain Profit! git rm

22. Code Bless Ship Maintain Profit! git rm

23. Proportional Investing
• When you lose N hours to maintenance, you
spend an equivalent N hours on improving
things.
Code Bless Ship Maintain Profit! git rm

24. Wrap Up
• Safety unleashes creativity
• Many touch points, many ways to do it —
find what’s right for you
• It’s not all about technology, most is about
the culture and the people