The document summarizes the findings of a qualitative study comparing performance bugs to non-performance bugs in the Chrome and Firefox browsers. The study analyzed hundreds of bug reports to identify differences across four dimensions: impact on the user, context of the bug, discussion around fixing the bug, and the actual bug fix. The study found that performance bugs generally have a greater negative impact, are more complex to fix, require more discussion and collaboration, and are more likely to block software releases.

1. A Qualitative Study on
Performance Bugs
Shahed Zaman Bram%Adams
Ahmed E. Hassan MCIS,&École&Polytechnique&de&Montréal
SAIL, Queen’s University

2. Bugs, Bugs, Bugs!
Costly Affect%
Reputa4on
2

3. Bugs, Bugs, Bugs!
482%bugs/week
2

4. Most%research%treats%all%bugs%equally!%
Does%this%make%sense?

5. Bugs, Bugs, Bugs!
problem&in&efficiency/speed&at&which&
Performance%Bug =
a&system&performs&at&runDEme
4

6. systemEkilling%
performance%problems
hHp://www.pcmag.com/arEcle2/0,2817,2325772,00.asp 5

7. Security versus Performance Bugs:
A Case Study on
Shahed&Zaman,&Bram&Adams,&Ahmed&E.&Hassan
SoUware&Analysis&and&Intelligence&Lab&(SAIL),&Queen’s&University
6

8. MSR%2011%Quan4ta4ve%Study
Time People Bug%Fix
&&&&&Performance&bugs&take& Performance&bugs&are&fixed& Performance&bug&fixes&
more%4me&to&fix by&more&experienced& are&more&complex
developers
7

9. Why?
8

10. A Qualitative Study on
Performance Bugs
Shahed Zaman Bram%Adams
Ahmed E. Hassan MCIS,&École&Polytechnique&de&Montréal
SAIL, Queen’s University

11. Study Setup
Chrome ®
295,198%bugs 44,997%bugs
7,603%performance%bugs 510%performance%bugs
100%Bugs 100%Bugs
+ +
100%Bugs 100%Bugs
= =
200%Bugs 200%Bugs
Performance%Bugs NonEperformance%bugs

12. bug%reports
Our%heuris4cs%have:
100&±&10%%precision
83&±&10%%recall
‘slow’,%‘perf’,%
‘hang’
Our%heuris4cs%have:
90&±&10%%precision
97&±&10%%recall 11

13. analyze
12

14. bug%report
+%a]achments
comments
13

15. update&taxonomy
analyze
think
14

16. update&taxonomy
analyze
think
Iterative
Process 14

17. context&of&the&bug
4 Dimensions
(20 subdimensions)
bug&fix&validaEon
impact&on&the&user
bug&fix

18. context&of&the&bug
4 Dimensions
(20 subdimensions)
bug&fix&validaEon
impact&on&the&user
bug&fix

19. sta4s4cally%
Findings significant%
difference
is&regression
Performance
NonDPerf.
blocks&release
WFM&after&a&long&Time
People&Threaten&to&switch
0 5 10 15 20 25 %

20. WorksForMe
No Traceability
17

21. Findings
is&regression
Performance
NonDPerf.
blocks&release
WFM&after&a&long&Time
scares&users&away
0 5 10 15 20 25 %

22. context&of&the&bug
4 Dimensions
(20 subdimensions)
bug&fix&validaEon
impact&on&the&user
bug&fix

23. Performance
Findings NonDPerf.
has&measurements
has&test&cases
has&stack&traces
Replication&Problems
Reported&by&Member
Duplicate&Bugs
0 10 20 30 40 50 60 %

24. automa4c%performance%
regression%tes4ng
21

25. automa4c%performance%
performance&
regression%tes4ng
regression&
idenEfied&
automaEcally
21

26. solicits%comparison%to%
other%browsers
22

27. Performance
Findings NonDPerf.
has&measurements
has&test&cases
has&stack&traces
has&replication&problems
Reported&by&Member
Duplicate&Bugs
0 10 20 30 40 50 60 %

28. Replication Problems
<more
<empty> empty>
incomplete&report plugins/addDons
24

29. Hard to Track Bugs of
External Applications
25

30. Performance
Findings NonDPerf.
has&measurement&data
has&test&cases
has&stack&traces
has&replication&problems
is&reported&by&developer
is&a&duplicate
0 10 20 30 40 50 60 %

31. context&of&the&bug
4 Dimensions
(20 subdimensions)
bug&fix&validaEon
impact&on&the&user
bug&fix

32. Findings Performance
NonDPerf.
has&discussion
Depends&on&Other&Bugs
Blocks&Other&Bugs
0 10 20 30 40 50 %

33. Performance Bugs
Require Collaboration
understand&the&problem
reach&consensus
idenEfy&bug&fix
me&too
29

34. Findings Performance
NonDPerf.
has&discussion
depends&on&other&bug
blocks&other&bug
0 10 20 30 40 50 %

35. Rebasing Performance
adjust&baseline
31

36. context&of&the&bug
4 Dimensions
(20 subdimensions)
bug&fix&validaEon
impact&on&the&user
bug&fix

37. NO
& diff
er
en
pe ce
rfo &b e
rm tw
an ee
ce n &p
&an atc
d &n h &d
on i sc
Dp e us
rfo si o
rm ns
an &an
ce d &r
&b u ev
gs iew
s&o
f
33

38. is&regression + +
blocks&release + +
WFM&aUer&a&long&Eme + +
impact&on&the&user scares&users&away + +
has&measurements + +
has&test&cases + +
contains&stacktrace + +
has&replicaEon&problems + +
context&of&the&bug
is&reported&by&a&developer ? +
is&a&duplicate E E
has&discussion + +
depends&on&other&bug + +
bug&fix
blocks&other&bug + E

39. is&regression + +
blocks&release + +
WFM&aUer&a&long&Eme + +
impact&on&the&user scares&users&away + +
has&measurements + +
has&test&cases + +
contains&stacktrace + +
has&replicaEon&problems + +
context&of&the&bug
is&reported&by&a&developer ? +
is&a&duplicate E E
has&discussion + +
depends&on&other&bug + +
bug&fix
blocks&other&bug + E

40. MSR%2011%Quan4ta4ve%Study%(reprise)
Performance&bugs&are&fixed&
&&&&&Performance&bugs&take& Performance&bug&fixes&
by&more&experienced&
more%4me&to&fix are&more&complex
developers
replication problems more release
blocking
more dependencies
m ore discussion users leaving
WFM after a long time 36

41. Threats to Validity
‘slow’,%‘perf’,%
‘hang’
human&bias
heurisEcs&for&
only&4&
only&2&subject performance&bug&
dimensions
systems idenEficaEon
&&&20&subD
dimensions 37

42. Study Setup
Chrome ®
no traceability
295,198%bugs context&of&the&bug
44,997%bugs
7,603%performance%bugs 510%performance%bugs
4 Dimensions
(20 subdimensions)
bug&fix&validaEon
impact&on&the&user
100%Bugs 100%Bugs
+ +
100%Bugs 100%Bugs
= =
200%Bugs 200%Bugs
Performance%Bugs NonEperformance%bugs
bug&fix
is&regression + + MSR%2011%Quan4ta4ve%Study%(reprise)
blocks&release + +
WFM&aUer&a&long&Eme + + Performance&bugs&are&fixed&
&&&&&Performance&bugs&take& Performance&bug&fixes&
impact&on&the&user by&more&experienced&
scares&users&away + + more%4me&to&fix are&more&complex
developers
has&measurements + +
has&test&cases + +
contains&stacktrace
no reproducability + +
discussion!
has&replicaEon&problems + +
? + replication problems more release
is&reported&by&a&developer
context&of&the&bug
is&a&duplicate E E blocking
more dependencies
has&discussion + +
depends&on&other&bug + + more discussion users leaving
bug&fix
blocks&other&bug + E WFM after a long time 37