Talk by Amit Garde, VP Engineering at Swiggy on the topic "An innocent's guide to ignoring best practices" at AWS Community Day, Bangalore 2018

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
BENGALURU

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
An Innocent’s Guide To
Ignoring Best
Practices...
AMIT GARDE, VP Engineering, Swiggy | 6 October 2018

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Preliminaries
• What this talk is *not* going to be…
• Prescriptive
• A laundry list of things to do or to not do
• What this talk is *trying* to be
• A perspective on how “Best Practice” emerges, why a grain of salt is useful, and
why we could usefully borrow ideas from other fields of human endeavour
• What this talk *might* end up being
• Whimsical
• Discursive
• …In which case consider it an attempt at AWS stand-up 

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Computer Science is a “Constructed Field”
• Brought into existence from “thought stuff”
• We don’t deal with the universe in the same way as say Physics or
Biology
• No Universal Laws, no Physical Constants
• Can give us a false sense of how much control we can assert

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Practitioner Mindset and the evolution of Best Practice
• Clouds – And AWS in particular - are Complex
• Operating complex cloud based systems is hard
• Deal with what’s breaking or what’s going to break next
• Very focused on the here-and-now
• Not a lot of time for contemplative meditation :-/
• “Don’t just sit there, do something”…Need something actionable
• We try to deal with complexity by trying to superimpose simplicity
• Hence “Best Practices” (Or tenets, or standard operating procedures or
rules of thumb or…)

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Perspective
“Good artists borrow, great artists steal…”
• We can do worse than learn from other fields
• Surprising connections and insights can come from seemingly unrelated
places
• …which can inform and improve the state of the practitioner’s art

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Straw Poll

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Strands

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. The Anthropic Principle
Weak Anthropic Principle
We must be prepared to take account of the fact that our location in the
Universe is necessarily privileged to the extent of being compatible with
our existence as observers.
That we exist implies that the Universe allows for our existence.

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Anthropic Principle
BUT
The evidence for our existence means the Universe allows our existence,
but it doesn't mean the Universe must have unfolded exactly this way. It
doesn't mean our existence is mandatory. And it doesn't mean the
Universe must have given rise to us exactly as we are. In other words, you
cannot say “the Universe must be the way it is because we’re here.”

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2. A War Story (Literally)
World War II. The USAF wanted to figure out where armour should be
added to fighter planes (You can’t just add armour everywhere as it makes
the fighters less manoueverable and consume more fuel and so limit
range). Data suggested that fighters that returned to base after a mission
had the most bullest holes on the rest of the plane, then the fuel system,
then the fuselage, then the engines. The air force asked how additional
armour should be distributed across where the bullet damage was the
worst.
They asked Abraham Wald, a member of the Statistical Research Group…

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Survivorship Bias
From the Wikipedia Entry for Survivorship Bias:
The logical error of concentrating on the people or things that made it past
some selection process and overlooking those that did not, typically
because of their lack of visibility.

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3. Evolution
Struggle for existence/Survival of the fittest: The weak get weeded out, the
best survive and propagate…
It’s more nuanced than that:
• Every organism that exists today is an evolutionary winner
• But need not be tomorrow’s
• Evolution does not produce optimality in any absolute sense

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4. The Dunning-Kruger Effect
• The scope of people’s ignorance is often invisible to them
• The least competent people can be the most confident about their
abilities
• ”Their deficits leave them with a double burden—not only does their
incomplete and misguided knowledge lead them to make mistakes but
those exact same deficits also prevent them from recognizing when
they are making mistakes and other people choosing more wisely.”
(Dunning, 2011)

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
5. Sensitive Dependence On Initial Conditions
A.k.a (more glamorously) Chaos Theory or the Butterfly Effect.
“Can a butterfly flapping its wings in Brazil cause a tornado in Texas?”

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Complex systems can be strongly nonlinear
Which is exactly what amplifies the effect of a divergence in initial
conditions…
If a best practice has been distilled from some other system’s observed
operating experience…
…How much did that system diverge from yours and how sensitive is yours
to initial conditions?

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
6. Entropy and the Second Law of Thermodynamics
Entropy: S = k ln(# of equiprobably microstates)
More colloquially: The lack of order or predictability
The Second Law of Thermodynamics (hand waving):
The entropy of a closed system never decreases.
“This law also seems plausible for software systems; as a system is
modified, its disorder, or entropy, tends to increase. This is known
as software entropy.”

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
7. Complex System Failures
A very (for me, personally) influential paper:
How Complex Systems Fail, Richard Cook
If the only thing you take away from this talk is a reference to this paper,
you’re still ahead of the game !

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Complex System Failures
18 Observations distilled from an analysis of failures in complex systems
• #3 Catastrophe requires multiple failures
• #5 Complex systems run in degraded mode
• #7 Post accident attribution to a ‘root cause’ is fundamentally wrong
• #8 Hindsight biases post-accident assessments of human performance
• #10 All practitioner actions are gambles

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Weaving The Strands

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
So remind me how this is about AWS/Best Practices/42…?
As system operators we practice our craft in a context
A context that is strongly influenced or skewed by the experience of others
We tend to extrapolate from others’ claimed experience and the prestige of
their name
And often adopt a “wisdom of crowds” best practice without a critical
analysis of why it makes sense in our specific context

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Citations

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dunning-Kruger
Unskilled and unaware of It: How difficulties in recognizing one’s own
incompetence lead to inflated self-assessments
Kruger J, Dunning D
Journal of Personal and Social Psychology, V77(6) 1999 pp1121-1134
The Dunning-Kruger effect: On being ignorant of one’s own
ignorance
David Dunning
Advances in Experimental Social Psychology, V44 2011 pp247-296

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dunning-Kruger
Unskilled and unaware of It: How difficulties in recognizing one’s own
incompetence lead to inflated self-assessments
Kruger J, Dunning D
Journal of Personal and Social Psychology, V77(6) 1999 pp1121-1134
The Dunning-Kruger effect: On being ignorant of one’s own
ignorance
David Dunning
Advances in Experimental Social Psychology, V44 2011 pp247-296

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Complex System Failure
How Complex Systems Fail
Richard I. Cook, Cognitive Systems Laboratory, University of Chicago
http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems
%20Fail.pdf