An internal training talk that Michelle D'Netto and I periodically give for Customer Support representatives at Etsy. Introduces advanced Software Quality concepts such as the halting problem, the impossibility of complete testing and the extreme difficulty of discovering all of the significant bugs in one's own software. Winds up by encouraging anyone responsible for online customer experience, to envision themselves as a participant observer embedded in the rapidly-evolving culture of the Web.

1. Software Entomology
or, Where Do Bugs Come From?
Noah Sussman and Michelle D’Netto
etsy

2. There will always be bugs in production

3. Most Bugs Are Non-Obvious

4. Debugging is hard

5. I can remember the exact
instant when I realized that a
large part of my life... was
going to be spent in finding
mistakes in my own programs
~Maurice Wilkes, 1949

6. Admiral Grace Hopper liked
to tell a story in which a
technician solved a glitch
in the Harvard Mark II
mainframe by pulling an
actual insect out from
between the contacts of
one of its relays
~ The Jargon File

8. Complete testing is impossible.

9. Even the simplest Web
site has more edge cases
than you could test, even
if you ran one test per
second until the heat
death of the universe.

10. assure that the probability of failure
due to hibernating bugs
is sufficiently low
to be acceptable
~ Boris Beizer, 1983

11. What constitutes a “sufficiently low”
number of problems?

12. Declaring a
program
“bug free,”
is an
entirely
subjective
excercise

13. Engineers suck at finding
bugs in their own code

14. Engineers + designers are
too close to the software

15. Computer programs cannot detect
bugs in other computer programs

16. Debugging is twice as
hard as writing the code
in the first place.
~ Brian Kernighan

18. Getting frustrated by software is
not the same as reporting a bug

19. Non-source-aware users
tend to report only surface
symptoms; they take their
environment for granted...
~ ESR, “How Many Eyeballs Tame Complexity”, 1997

20. Users never read manuals
Users would save time in the long
term by learning more about the
system. But that's not how people
behave in the real world
~Jakob Nielsen

21. On two occasions
I have been asked...
if you put into the
machine wrong
figures, will the right
answers come Babbage, 1864
~Charles
out?

23. When you report a bug
you have done a hard thing

24. A bug report is a story about a problem and
a good story has a beginning, middle and end

25. Description of problem
Steps to reproduce
Expected resolution

26. Provide context, and lots of it

27. Be Quantitative
Engineers respond well to hard evidence

28. Graphmonitorit didn’t happen
At Etsy we
or over 100,000 real-time metrics

29. A good editor is
meticulously attentive to detail,
a stickler for technical correctness,
aware of the work as a whole and
sensitive to the expectations of the audience

30. You are doing science
Keep a journal
Take screenshots (use Skitch)
Log IRC, Skype and other chats
Make multiple redundant backups (use DropBox)
Save everything— you won’t know you need it until you need it

31. How many times has this happened?
How many users were affected?
How much time has been lost?
What was the overall cost?

32. Users

33. The Support
Team spend
the most time
engaging and
empathizing
with users

34. You are a participant observer

35. When you talk to your users, and
you write down what you learn,
and you share what you wrote
with others, you’re doing science

36. digital computer: less than 70 years old,
personal computer: less than 40,
the Web: less than 20,
mobile Web: less than 15

37. The Web is going to keep on growing and changing

38. 80% of us aren’t
even on the Web yet

39. Most users of a Web site are doing something that is entirely new to them
And you are teaching them how to do it