The document describes an empirical study on the impact of two antipatterns - Blob and Spaghetti Code - on program comprehension. It presents three experiments where subjects performed comprehension tasks on code with and without the antipatterns. The experiments measured subjects' performance in terms of effort, time taken, and percentage of correct answers. The results were analyzed to test hypotheses about whether the antipatterns negatively or positively impacted comprehension. The goal was to provide quantitative evidence on the relationship between antipatterns and program comprehension.

1. Antipatterns and
Comprehension
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
An Empirical Study of the Impact of Two
Related Work Antipatterns on Program Comprehension
Experimental
Study
Study Results
Marwen Abbes, Foutse Khomh,
Discussions
Yann-Ga¨l Gu´h´neuc, and Giuliano Antoniol
e e e
Conclusion and
Future Work
Questions ´
DGIGL, Ecole Polytechnique de Montr´al, Qu´bec, Canada
e e
References
E-mails: {marwen.abbes, yann-gael.gueheneuc,
giuliano.antoniol}@polymtl.ca, foutse.khomh@queens.ca
CSMR 2011
February 3, 2011
Pattern Trace Identification, Detection, and Enhancement in Java
SOftware Cost-effective Change and Evolution Research Lab

2. Antipatterns and
Comprehension Outline
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction Introduction
Related Work
Experimental Related Work
Study
Study Results
Discussions
Experimental Study
Conclusion and
Future Work
Study Results
Questions
References
Discussions
Conclusion and Future Work
Questions
2 / 37

3. Antipatterns and
Comprehension Introduction
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study OO Paradigm
Study Results There is many benefits in using OO for analysis, design,
Discussions coding.
Conclusion and
Future Work These benefits include better understandability, easier
Questions modifications, greater productivity, and simpler reuse.
References
These benefits rely mainly on intuition instead of
observations or experimental results.
3 / 37

4. Antipatterns and
Comprehension Introduction
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study Empirical Studies
Study Results There exist few empirical studies confirming the benefits
Discussions of OO.
Conclusion and
Future Work It is only recently that researchers started to study the
Questions “OO claims” empirically.
References
Obtained results show that OO does not always bring
the expected benefits.
4 / 37

5. Antipatterns and
Comprehension Introduction
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction Empirical Studies
Related Work
The literature suggests many explanations for the
Experimental
Study apparent lack of benefits of OO:
Study Results OO notions (objects, polymorphism, inheritance,
Discussions patterns, and so on) are not easy to learn and use
Conclusion and adequately.
Future Work
The use of OO impacts negatively the understandability
Questions of systems.
References Antipatterns deteriorate the quality of OO systems and
them harder to maintain.
Therefore, the benefits of OO require further empirical
studies, in particular to show its effectiveness and
efficacy or lack thereof.
5 / 37

6. Antipatterns and
Comprehension Introduction
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction Goal
Related Work
Antipatterns are conjectured in the literature to
Experimental
Study decrease the quality of OO systems.
Study Results Yet, despite the many studies on antipatterns, few
Discussions
studies empirically investigated the impact of
Conclusion and
Future Work antipatterns on program comprehension.
Questions We want to gather quantitative evidence on the relation
References between antipatterns and program comprehension (or
lack thereof).
We perform a quantitative analysis of the impact of two
antipatterns on developers’ performance during
comprehension and maintenance tasks.
6 / 37

7. Antipatterns and
Comprehension Related Work
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Antipatterns and Quality
Study
Webster’s first book on quality-assurance problems [1].
Study Results
Discussions
Brown’s 40 antipatterns [2].
Conclusion and
Antipatterns are bad practices when solving design
Future Work problems.
Questions Antipatterns are conjectured in the literature to
References decrease the quality of systems.
Fowler’s 22 code smells and refactorings [3] to improve
the design of existing code.
7 / 37

8. Antipatterns and
Comprehension Related Work
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study Antipatterns and Detection
Study Results Many approaches exist for specifying and detecting
Discussions
code smells and antipatterns, e.g., [4, 5, 6].
Conclusion and
Future Work We detect antipatterns using DEX, which stems from
Questions our method DECOR (recall of 100% and an average
References precision greater than 60%) to locate and remove them
in the systems that we study.
8 / 37

9. Antipatterns and
Comprehension Related Work
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Antipatterns and Software Evolution and Maintenance
Introduction
Related Work Several works have studied the impact of antipatterns
Experimental on evolution phenomena.
Study
Vaucher et al. [7] studied in the versions of two
Study Results
different systems the evolution of Blobs:
Discussions
They tracked the occurrences of the Blob.
Conclusion and
Future Work They observed that Blobs are, in few cases, created by
design as the best solution to a particular problem.
Questions
References
Olbrich et al. [8] studied occurrences of the Blob and
Shotgun Surgery:
They tracked the occurrences of these antipattersn.
They observed that they are more change-prone than
other classes.
These works suggest that antipatterns impact program
evolution and maintenance.
9 / 37

10. Antipatterns and
Comprehension Related Work
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Antipatterns and Understandability
Introduction
Several works have studied the impact of antipatterns
Related Work
on evolution phenomena.
Experimental
Study Deligiannis et al. [9] conducted a controlled experiment
Study Results on the impact of Blob on the maintainability of two
Discussions systems.
Conclusion and They concluded that the maintainability OO systems
Future Work depends on good practices and design.
Questions They did not assess the impact of Blob on the
References developers’ ease of understanding.
Du Bois et al. [10] studied the impact on
understandability of the decomposition of a Blob.
They found that students had more difficulties
understanding the original Blob than other
decompositions.
Their study did not reveal any objective notion of
“optimal” understandability.
10 / 37

11. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
In a Nutshell
Related Work
Experimental
We perform 3 experiments to assess subjects’
Study comprehension of source code in the presence of two
Study Results antipatterns:
Discussions Experiment 1 includes one and only one Blob.
Conclusion and Experiment 2 includes one and only one Spaghetti Code.
Future Work
Experiment 3 includes both antipatterns.
Questions
References
In each experiment, we assign two systems to each
subject: one containing one occurrence of one (or both)
antipattern(s) and one without any occurrence.
We then measure and compare the subjects’
performances for both systems.
11 / 37

12. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol Hypotheses
Introduction H0Blob : There is no statistically significant difference
Related Work between the subjects’ average performance when
Experimental executing comprehension tasks on the source code of
Study
systems containing one occurrence of the antipattern
Study Results
Discussions
Blob and their average performance with source code
Conclusion and
without any antipattern.
Future Work
If we reject the previous null hypothesis, then:
Questions
Either, E1Blob : the subjects’ average performance is
References
better when executing comprehension tasks on systems
containing no occurrence of the Blob;
Or, E2Blob : the subjects’ average performance is better
when executing comprehension tasks on systems
containing one occurrence of the Blob;
We have two similar null hypotheses H0SpaghettiCode and
H0Blob+SpaghettiCode and sets of alternative hypotheses for
the other two experiments.
12 / 37

13. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Objects
Introduction
We choose three systems for each experiment, all
Related Work
developed in Java.
Experimental
Study
Numbers of Release
Study Results
Experiments Systems Classes SLOCs dates
Discussions YAMM 0.9.1 64 11,272 1999
Conclusion and 1 JVerFileSystem 167 38,480 2008
Future Work AURA 95 10,629 2008
Questions GanttProject 2.0.6 527 68,545 2008
2 and 3 JFreeChart 1.0.13 989 302,844 2009
References
Xerces 2.7.0 740 233,331 2008
Table: Object Systems
We perform each experiment on 3 systems, because a
single system could be intrinsically easier/more complex
to understand.
13 / 37

14. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Objects
Related Work We use DECOR to ensure that each system has at least
Experimental one occurrence of antipattern.
Study
Study Results
We validate the detected occurrences manually.
Discussions For each system, we select randomly a subset of classes
Conclusion and responsible for managing a specific task to limit the size
Future Work
Questions
of the source code given to the subjects.
References For example, in JFreeChart, we choose the source code
of the classes responsible for editing and displaying the
properties of a plot.
We refactor each subset of each system to remove all
other occurrences of (other) antipatterns to reduce
possible bias.
14 / 37

15. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study
Independent and Dependent Variables
Study Results
Discussions
The independent variable is the presence of the
Conclusion and
occurrence(s) of the specific antipattern(s).
Future Work
Our dependent variables measure the subjects’
Questions
performance, in terms of effort, time spent, and
References
percentage of correct answers.
15 / 37

16. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Dependent Variable: Effort
Related Work
Experimental
Study
Study Results
Discussions
Conclusion and
Future Work
Questions
References
Figure: TLX
16 / 37

17. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study Dependent Variable: Time
Study Results
Discussions
Conclusion and
Future Work
Questions
References
Figure: Timer.
17 / 37

18. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study
Study Results Dependent Variable: Correct Answers
Discussions We compute the percentage of correct answers for each
Conclusion and
Future Work
question by dividing the number of correct elements
Questions
found by the subject by the total number of correct
References elements that they should have found.
18 / 37

19. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction Subjects
Related Work
Each experiment was performed by 24 anonymous
Experimental
Study subjects, S1 to S24.
Study Results Some subjects were enrolled in the M.Sc. and Ph.D.
Discussions
programs in Computer and Software engineering at
Conclusion and
Future Work
´
Ecole Polytechnique de Montr´al or in Computer
e
Questions Science in Universit´ de Montr´al.
e e
References Others were professionals working for software
companies in the Montr´al area, recruited through the
e
authors’ industrial contacts.
All subjects were volunteers and could withdraw at any
time, for any reason.
19 / 37

20. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Questions
Introduction
Related Work
Category 1: Finding a focus points:
Experimental
Question 1: Where is the code involved in the
Study implementation of this behavior?
Study Results Question 2: Which type represents this domain
Discussions concept or this UI element or action?
Conclusion and
Future Work
Category 2: Expanding a focus points:
Questions Question 1: Where is this method called or this type
References
referenced?
Question 2: What data can we access from this object?
Category 3: Understanding a code subset:
Question 1: How are these types or objects related?
Question 2: What is the behavior that these types
provide together and how is it distributed over these
types?
20 / 37

21. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Design
Related Work Our design is a 2×3 factorial design:
Experimental We have three different systems, each with two
Study
possibilities.
Study Results
For each combination, we prepare a set of
Discussions
comprehension questions, making up treatments.
Conclusion and
Future Work We have six different groups of subjects, each one
Questions affected to each one treatment.
References
With Antipattern(s) Without Antipattern(s)
System 1 S3 , S7 , S9 , S11 , S12 , S18 , S21 , S24 S1 , S5 , S8 , S10 , S15 , S16 , S20 , S22
System 2 S1 , S2 , S6 , S14 , S15 , S17 , S20 , S22 S4 , S7 , S9 , S11 , S13 , S18 , S19 , S23
System 3 S4 , S5 , S8 , S10 , S13 , S16 , S19 , S23 S2 , S3 , S6 , S12 , S14 , S17 , S21 , S24
Table: Experimental Study
21 / 37

22. Antipatterns and
Comprehension Experimental Study
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction Analysis Method
Related Work
We use the (non-parametric) Mann-Whitney test to
Experimental
Study compare sets of dependent variables and assess whether
Study Results their difference is statistically significant.
Discussions
We also test the hypotheses with the (parametric)
Conclusion and
Future Work Student’s t-test.
Questions We use the Cohen d effect size, which indicates the
References magnitude of the effect of a treatment on the
dependent variables.
We use ANOVA to assess the dependence between the
six sets of dependent variables, as we have six different
groups affected to the different treatments.
22 / 37

23. Antipatterns and
Comprehension Study Results
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol Summary of the Collected Data
Introduction Systems C. A.T. (s.) A.E. A.C.A.
Related Work
System 1: B 261 43.42 71%
YAMM B 149 27.99 71%
Experimental System 2: B 251 43.30 67%
Study
JVerFileSystem B 206 33.68 79%
Study Results System 3: B 189 33.31 58%
Discussions Aura B 271 42.55 66%
Conclusion and System 1: S 190 47.31 67%
Future Work GanttProject S 194 53.44 51%
Questions System 2: S 215 39.27 52%
Xerces S 182 34.48 52%
References
System 3: S 195 42.37 52%
JFreeChart S 218 45.84 45%
System 1: BS 187 45.36 44%
GanttProject BS 107 27.32 83%
System 2: BS 184 42.83 73%
Xerces BS 140 29.33 86%
System 3: BS 208 48.68 43%
JFreeChart BS 138 31.27 78%
23 / 37

24. Antipatterns and
Comprehension Study Results
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
p-values and Cohen’s d effect size results
Introduction Times
Related Work M.-W. t-Test ANOVA Cohen
Experimental
p p p d
Study Experiment 1 0.46 0.30 0.02 0.18
Study Results
Experiment 2 0.89 0.97 0.87 0.01
Experiment 3 <0.01 <0.01 <0.01 1.54
Discussions
Efforts
Conclusion and M.-W. t-Test ANOVA Cohen
Future Work
p p p d
Questions Experiment 1 0.25 0.17 0.29 0.24
References Experiment 2 0.64 0.57 0.41 0.09
Experiment 3 <0.01 <0.01 0.01 1.20
Answers
M.-W. t-Test ANOVA Cohen
p p p d
Experiment 1 0.41 0.21 0.58 0.27
Experiment 2 0.26 0.24 0.60 0.35
Experiment 3 <0.01 <0.01 <0.01 1.61
24 / 37

25. Antipatterns and
Comprehension
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study
Study Results
Discussions
Conclusion and
Future Work
Questions
References
25 / 37

26. Antipatterns and
Comprehension Discussions
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol Results
Introduction The results of Experiment 1 show an increase in
Related Work subjects’ average time and effort on systems with Blob
Experimental and a decrease in their average percentages of correct
Study
Study Results
answers.
Discussions Experiment 2 reveals no significant difference between
Conclusion and systems without/with a Spaghetti Code.
Future Work
Questions Experiment 3 shows strong statistically significant
References differences between systems without/with a Blob and a
Spaghetti Code.
ANOVA tests confirm that these differences are
significant across the six groups of subjects.
Cohen’s d effect size values (for efforts, times, and
percentages of correct answers) are large; suggesting a
strong relation.
26 / 37

27. Antipatterns and
Comprehension Discussions
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Mitigating Variables
Introduction
We investigate if three mitigating variables impacted
Related Work
our results: knowledge of Java, Eclipse, and in software
Experimental
Study engineering.
Study Results
We use 5-level Likert scales (bad, neutral, good,
Discussions
excellent, expert).
Conclusion and
Future Work As the groups are non-equal in terms of size, we used
Questions
the Mann-Whitney test.
References
We found no significant differences between the
different levels; p-values are high, expressing a lack of
correlation.
We perform an ANOVA test to assess the impact of the
mitigating variables on the three measured variables,
which shows that they do not impact our results.
27 / 37

28. Antipatterns and
Comprehension Discussions
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study
Threats to Validity
Study Results Some threats limit the validity of our study results:
Discussions
Construct Validity: Measurement errors, subjectivity,
Conclusion and
Future Work “seriousness” of the antipattern occurrences.
Questions Internal Validity: Learning, selection, measures (again),
References diffusion of the treatments.
External Validity: Generalisability.
28 / 37

29. Antipatterns and
Comprehension Conclusion and Future Work
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work Conclusion
Experimental
Study
Antipatterns are conjectured in the literature to
Study Results
negatively impact the quality of systems.
Discussions We performed three experiments to study whether
Conclusion and systems with antipattern(s) are more difficult to
Future Work
Questions
understand than systems without any antipattern.
References Each experiment was performed with 24 subjects and
on three different Java systems.
We measured the subjects’ performance in terms of
efforts, times, and percentages of correct answers.
29 / 37

30. Antipatterns and
Comprehension Conclusion and Future Work
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction Conclusion
Related Work
Collected data showed that the occurrence of one
Experimental
Study antipattern in the source code of a system does not
Study Results significantly make its understandability.
Discussions
The combination of two occurrences of antipatterns
Conclusion and
Future Work impacted negatively and significantly understandability.
Questions Developers can cope with antipatterns in isolation
References but combinations thereof should be avoided,
possibly through detection and refactorings.
Consequently, developers and quality assurance
personnel should be wary with growing numbers of
antipatterns in their systems.
30 / 37

31. Antipatterns and
Comprehension Conclusion and Future Work
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental Future Work
Study
Study Results
We want to investigate whether these
Discussions statistically-significant differences are due to the density
Conclusion and
˝
of antipatterns in the system andU-or to the
Future Work
occurrences of specific antipatterns.
Questions
References
We also plan to replicate this study in other contexts,
with other subjects, other questions, other antipatterns,
and other systems.
31 / 37

32. Antipatterns and
Comprehension Any Question?
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study
Study Results
Discussions
Conclusion and
Future Work
Questions
References
32 / 37

33. Antipatterns and
Comprehension References
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
B. F. Webster, Pitfalls of Object Oriented Development,
Related Work 1st ed. M & T Books, February 1995. [Online].
Experimental Available:
Study
www.amazon.com/exec/obidos/ASIN/1558513973
Study Results
Discussions W. J. Brown, R. C. Malveau, W. H. Brown, H. W.
Conclusion and McCormick III, and T. J. Mowbray, Anti Patterns:
Future Work
Questions
Refactoring Software, Architectures, and Projects in
References
Crisis, 1st ed. John Wiley and Sons, March 1998.
[Online]. Available: www.amazon.com/exec/obidos/tg/
detail/-/0471197130/ref=ase theantipatterngr/
103-4749445-6141457
M. Fowler, Refactoring – Improving the Design of
Existing Code, 1st ed. Addison-Wesley, June 1999.
33 / 37

34. Antipatterns and
Comprehension References
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Naouel Moha, Y.-G. Gu´h´neuc, L. Duchien, and
e e
Related Work
A.-F. L. Meur, “DECOR: A method for the specification
Experimental
Study and detection of code and design smells,” Transactions
Study Results on Software Engineering (TSE), 2009. [Online].
Discussions Available: http://www-etud.iro.umontreal.ca/˜ptidej/
Conclusion and Publications/Documents/TSE09.doc.pdf
Future Work
Questions M. J. Munro, “Product metrics for automatic
References identification of “bad smell” design problems in java
source-code,” in Proceedings of the 11th International
Software Metrics Symposium. IEEE Computer Society
Press, September 2005. [Online]. Available: http://doi.
ieeecomputersociety.org/10.1109/METRICS.2005.38
34 / 37

35. Antipatterns and
Comprehension References
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
R. Marinescu, “Detection strategies: Metrics-based rules
Related Work
for detecting design flaws,” in Proceedings of the 20th
Experimental
Study International Conference on Software Maintenance.
Study Results IEEE CS Press, 2004, pp. 350–359.
Discussions
St´phane Vaucher, Foutse Khomh, Naouel Moha, and
e
Conclusion and
Future Work Y.-G. Gu´h´neuc, “Prevention and cure of software
e e
Questions defects: Lessons from the study of god classes,” in
References Proceedings of the 16th Working Conference on Reverse
Engineering (WCRE), G. Antoniol and A. Zaidman, Eds.
IEEE Computer Society Press, October 2009, 10 pages.
[Online]. Available: http://www-etud.iro.umontreal.ca/
˜ptidej/Publications/Documents/WCRE09b.doc.pdf
35 / 37

36. Antipatterns and
Comprehension References
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
S. Olbrich, D. S. Cruzes, V. Basili, and N. Zazworka,
Introduction “The evolution and impact of code smells: A case study
Related Work of two open source systems,” in Third International
Experimental Symposium on Empirical Software Engineering and
Study
Study Results
Measurement, 2009.
Discussions D. Ignatios, S. Ioannis, A. Lefteris, R. Manos, and
Conclusion and
Future Work
S. Martin, “A controlled experiment investigation of an
Questions
object oriented design heuristic for maintainability,”
References
Journal of Systems and Software, vol. 65, no. 2,
February 2003.
B. D. Bois, S. Demeyer, J. Verelst, T. Mens, and
M. Temmerman, “Does god class decomposition affect
comprehensibility?” in Proceedings of the IASTED
International Conference on Software Engineering.
IASTED/ACTA Press, 2006, pp. 346–355.
36 / 37

37. Antipatterns and
Comprehension References
Abbes, Khomh,
Gu´h´neuc, and
e e
Antoniol
Introduction
Related Work
Experimental
Study
Study Results
Discussions
Conclusion and
Future Work
Questions
References
37 / 37