Testing plug-in based systems entails testing a complex net of multiple plug-ins which not only interact with, but also enhance the functionality of each other by extension mechanisms. In the Eclipse Testing Study, testing practices of successful Eclipse projects have been studied. Among others, Eclipsers stress integration testing as an important testing activity for plug-in systems, and explain that understanding complex plugin test suites can become a challenging task. To remedy the problem of understanding plug-in test suites, we developed the Eclipse Plug-in Test Suite Exploration (ETSE) tool. ETSE combines static and dynamic information on plug-in dependencies, extension initialization, and extension or service usage during a test run. This information is used to create five architectural views of the system under test and its test suite, which developers can use to understand how the integration of multiple plug-ins is tested. In this presentation, we will talk about plug-in test suites and present the architectural views, which help answering questions like: “which plug-ins are tested by this test suite?”, “where are test harness and test utilities located?”, “which extensions influence the state of the test system?”, and many more. We will end this session with a short demonstration of ETSE by visualizing the test suite of Mylyn.

1. Understanding Plug-in Test Suites
from an Integration Perspective
Michaela Greiler, Arie van Deursen
Software Engineering Research Group
Delft University of Technology

2. Understanding Plug-in Test Suites

5. Need more Information?
Which Information?

7. Code Dependencies
which plug-ins are using each other
Service Relations
which plug-ins are using services each other
Extension Relations
which plug-ins are extending each other

9. Interviewed 25 Eclipse practitioners
Understanding
test cases
Understanding
test suites

10. Interviewed 25 Eclipse practitioners
Understanding
test cases
Coding standards, naming conventions.
Explanation and Motivation.
“Tests are also like other parts of code. Sometimes people give bad
names to their methods and variables. Then it is hard to understand.”

11. Interviewed 25 Eclipse practitioners
Understanding
test suites
“What one specific test does, that’s quite
easy to understand. What’s difficult is to see
where the blank spots on the map are.”

12. Interviewed 25 Eclipse practitioners
Understanding
test suites
Challenges keeping an Overview:
“It is difficult. You have to read it all. If you skip
one test, you do not know one part of the system.”
Test Organization and Structure.
Nested Test Suites.

13. Interviewed 25 Eclipse practitioners
Understanding
test suites
“We realized: either we work on a particular point, and
we run only one test case, or we run all of them.
We worry very often about not being able to run the test
suite in a more fine-grained way..”

15. Which plug-ins are
How are the plug-in
tested by which test-
extensions and services
component?
tested?
How are the plug-ins
extending each other?
Which extensions are
used during the tests?

16. Manual Inspection!

17. Look at…
… extension relations
Only Static
…code

18. Understanding Plug-in Test Suites
from an Integration Perspective

19. 5 Architectural View
Extension Service
Plug-in Test-Suite Extension
Usage View Usage View
Modularization Modularization Initialization
View View View
Plug-in Level Test Case Level Extension Level Method Level Method Level
Plug-in A Plug-in A Plug-in A Plug-in A
Plug-in A
is tested by loads invokes invokes
uses
test case t extension e method m of method m of
Plug-in B
of Plug-in B of Plug-in B extension e service s
in plug-in B in plug-in B

20. Ingredients: Data Behind
Static Data & Dynamic Trace Data
plugin.xml & schema Test suite
manifest class files
class files
Fact Extraction Instrumentation
Facts Traces
Reconstruction
VIEWS
Recipe: Reconstructing The Views

21. Recipe: Reconstructing the Extension Usage View
STATIC: plug-in.xml, schema, .class DYNAMIC: Method Calls
Fact Extraction Instrumentation
Facts Traces
Reconstruction
Extension Method Set: Heuristic
(all methods of an extension)
Inspect the Trace File
Extension Usage View
which “extension methods” are
invoked

22. ETSE: Eclipse Test Suite Explorer

23. Empirical Evaluation

24. Case Study Set-Up
Connector
27 plug-ins 5 main plug-ins 3 main plug-ins
11 test-components 2 test components 1 test component
2 test suites 4 test suites 2 test suites
200,000 LOCs 100,000 LOCs 8,500 LOCs
30,000 LOCs tests 14,000 lines tests 3,400 lines tests

26. ETSE: Eclipse Test Suite Explorer

27. Static
Dynamic
Which plug-ins are
tested by which test-
component?

28. Dynamic & Static
System Level
How are the plug-in
extension-relations tested?
How are the plug-ins
extending each other?

29. Plug-in Level
Dynamic & Static
How are the plug-in
extension-relations tested?
How are the plug-ins
extending each other?

30. When and how have
the extensions been
used?
Which extensions are
used during the tests?

31. ETSE: Eclipse Test Suite Explorer

32. Want to use ETSE in your project?
Contactus!
m.s.greiler@tudelft.nl
arie.vandeursen@tudelft.nl
Update site:
http://swerl.tudelft.nl/etse/ETSE_UpdateSite/site.xml