Paper: Structural Conformance Checking with Design Tests: An Evaluation of Usability and Scalability. Authors: João Brunet, Dalton Dario Serey Guerrero and Jorge Figueiredo. Session: Research Track 5: Traceability

1. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Design Tests
João Brunet
Dalton Guerrero and Jorge Figueiredo
Federal University of Campina Grande (UFCG), Brazil.
09/28/2011
João Brunet 1/ 26

2. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Agenda
1 Motivation
2 Design Test
3 DesignWizard
4 Evaluation
API Usability
5 Conclusion
João Brunet 2/ 26

3. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Motivation
Check code against design rules is an important activity to
guarantee quality on source code
Design Review
Manual Process
Error-prone
Does not scale
João Brunet 3/ 26

4. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Motivation
Check code against design rules is an important activity to
guarantee quality on source code
Design Review
Manual Process
Error-prone
Does not scale
João Brunet 3/ 26

5. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
The Problem
Gap between state-of-the-art and the state-of-the-practice
Approaches from the state-of-the-art require learning different
languages to express design rules
João Brunet 4/ 26

6. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
The Problem
Lack of structural conformance checking
Design decisions are violated
João Brunet 5/ 26

7. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Design Test
Design Test
A test that checks whether an implementation complies with a given
design rule expressed as an algorithm.
João Brunet 6/ 26

8. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
What’s the big deal?
It is written in the target programming language
Automated test
João Brunet 7/ 26

9. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Example of Design Rule
Only controller package shall access dao package of a given
component.
João Brunet 8/ 26

10. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Example of Design Test
João Brunet 9/ 26

11. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Example of Design Test
Facts about the code (an API)
João Brunet 10/ 26

12. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Example of Design Test
Facts about the code (an API)
Assertion routines
João Brunet 11/ 26

13. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
DesignWizard
João Brunet 12/ 26

14. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Some methods of DesignWizard’s API
getAllPackages()
getAllSubClasses()
getStaticMethods()
getImplementedInterfaces()
...
João Brunet 13/ 26

15. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Process
João Brunet 14/ 26

16. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Process
João Brunet 15/ 26

17. Motivation
Design Test
DesignWizard API Usability
Evaluation
Conclusion
Methodology
“It is the comparison between what developers expect and what the
API provides that is interesting when evaluating the usability of an API
(Steven Clarke)”
Think Aloud Protocol For APIs to capture expectations, goals,
difficulties and strategies
João Brunet 16/ 26

18. Motivation
Design Test
DesignWizard API Usability
Evaluation
Conclusion
Experiment
11 developers
5 real design tests (selected by the designer of the OurGrid)
OurGrid
Eclipse
API
Pseudocode vs. Code
João Brunet 17/ 26

19. Motivation
Design Test
DesignWizard API Usability
Evaluation
Conclusion
Experiment
João Brunet 18/ 26

20. Motivation
Design Test
DesignWizard API Usability
Evaluation
Conclusion
Experiment
Only one developer did not write one instance of design test.
João Brunet 19/ 26

21. Motivation
Design Test
DesignWizard API Usability
Evaluation
Conclusion
Experiment
...but, 10 tests were incorrectly written. The good news is that all
developers found the right method, but used it incorrectly.
João Brunet 20/ 26

22. Motivation
Design Test
DesignWizard API Usability
Evaluation
Conclusion
Results
Two weaknesses of API:
getAllClasses()
We decided to follow the Java Reflection API pattern
getDeclaredMethod (String name)
An appropriated name of the parameter would be signature
João Brunet 21/ 26

23. Motivation
Design Test
DesignWizard API Usability
Evaluation
Conclusion
Results
Think Aloud Protocol revealed that devolopers had no difficulties
to find the right method to achieve a given task
Results suggested changes to DesignWizard API and
documentation.
João Brunet 22/ 26

24. Motivation
Design Test
DesignWizard API Usability
Evaluation
Conclusion
Threats to validity
Coding and designing expertise of developers
More complex design rules may produce different observations
The API was not fully evaluated
João Brunet 23/ 26

25. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Final Remarks
Approach to check conformance between code and design rules
Evaluation regarding its usability
Design Tests and maintenance
Executable documentation of design rules
Enables developers changing functionality keeping in mind
structural constraints
João Brunet 24/ 26

26. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Thank you
www.designwizard.org
João Brunet 25/ 26

27. Motivation
Design Test
DesignWizard
Evaluation
Conclusion
Related Work
ArchJava
Reflexion Models
FindBugs
...
João Brunet 26/ 26