If this Giant Must Walk: A Manifesto for a New Nigeria
Effective and Efficient API Misuse Detection via Exception Propagation and Search-based Testing
1. Effective and Efficient API Misuse Detection via
Exception Propagation and Search-Based Testing
Maria Kechagia
Univ. College London,
London, United Kingdom
Xavier Devroey
Delft Univ. of Technology
Delft, Netherlands
Annibale Panichella
Delft Univ. of Technology
Delft, Netherlands
Georgios Gousios
Delft Univ. of Technology
Delft, Netherlands
Arie van Deursen
Delft Univ. of Technology
Delft, Netherlands
2. APIs everywhere
• An Application Programming Interface (API)
is a set of functions and procedures offering
services
• Applications are built by combining API
calls
• Example: Maven dependency graph with
186,384 Nodes and 1,229,083 Edges
• Usage constraints
• Explicit
• Implicit
• API misuses
• Violation of usage constraints
!2
Maven dependency graph using Force Atlas 2
https://github.com/ogirardot/meta-deps
7. Catcher
!7
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
8. Catcher
!8
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
1. Spotting
potential runtime
exceptions throws
API
9. Catcher
!9
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
10. Catcher
!10
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
11. Catcher
!11
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
4. Use the candidate misuses
as test objectives for a search-
based test case generation
algorithm
tests
12. Catcher
!12
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
4. Use the candidate misuses
as test objectives for a search-
based test case generation
algorithm
tests
5. Expose the misuses
13. Static exception propagation
!13
Static exception propagation analysis
API Client
public void a1(…){
…
throw new IllegalStateException();
…
}
public void a2(…){
…
}
public void a3(…){
…
a1(…)
…
}
public void c1(…){
…
a3(…);
…
}
public void c2(…){
…
}
public void c3(…){
…
a1(…);
…
}
14. Filtering candidate misuses
!14
Static exception propagation analysis
API Client
public void a1(…){
…
throw new IllegalStateException();
…
}
public void a2(…){
…
}
public void a3(…){
…
a1(…)
…
}
public void c1(…){
…
a3(…);
…
}
public void c2(…){
…
}
public void c3(…){
…
a1(…);
…
}
Candidate misuses
IllegalStateException
at a1()
at a3()
at c1()
IllegalStateException
at a1()
at c3()
15. Search-based test case generation
!15
Candidate misuses
IllegalStateException
at a1()
at a3()
at c1()
IllegalStateException
at a1()
at c3()
Focused search-based test generation
EvoSuite
public void c1(…){
…
a3(…);
…
}
public void c2(…){
…
}
public void c3(…){
…
a1(…);
…
}
public void testC1(…){
…
}
public void testC3(…){
…
}
16. Static exception propagation analysis
• Builds the API call graph
• Annotated with the list of runtime exceptions
• Builds the global call graph
• Link the client call graph to the API call graph
• Propagate exceptions from the API to the client
• Maximal propagation depth (≤4)
• Filter exceptions handled by the client
• Each propagation is a candidate misuse
• Relies on Soot (http://sable.github.io/soot/)
!16
a1() a2() a3()
a4() a5()
c1() c2() c3()
c4()
API
Client
17. Focused Search-based test generation
• Dynamic Many-Objective Sorting Algorithm (DynaMOSA)
• Objectives
• For each misuse
• Reaching propagation point (branch coverage)
• Client method input diversity
• Client method output diversity
• Misuse detected if and only if
• Exception is thrown by the test case
• Propagation is the same as in the candidate misuse
• Relies on EvoSuite (http://www.evosuite.org)
!17
Random test
cases tests
Evolutionary
process
Test cases
exposing misuses tests
18. Evaluation on the JDK API
• RQ1: How do existing unit level coverage-based test
generation tools perform in discovering API misuses?
• RQ2: Does Catcher improve the performance of existing
test coverage-based approaches on detecting API
misuses?
• RQ3: What types of API misuses does Catcher expose?
• Comparison with (plain) EvoSuite with…
• DynaMOSA + default coverage criteria
• On all classes of the client apps
• 25 runs x 3 minutes x each class under test
!18
Client project Files
(#)
LOC (K)
bcel-6.2 489 39
commons-cli-1.4 50 7
commons-codec-1.12 124 10
commons-collections-4.2 535 63
commons-compress-1.17 352 43
commons-lang-3.7 323 76
commons-math-3.6.1 1617 209
easymock-3.6 204 14
gson-2.8.5 206 25
hamcrest-core-1.3 152 7
jackson-databind-2.9.6 919 114
javassist-3.23.1 527 82
jcommander-1.71 139 6
jfreechart-1.5.0 990 134
jodat-time-2.10 330 86
jopt-simple-5.0.4 192 9
natty-0.13 27 3
neo4j-java-driver-1.6.2 510 52
shiro-core-1.3.2 653 31
xwiki-commons-job-10.6 67 3
xwiki-commons-text-10.6 3 101
19. Evaluation results
!19
EvoSuite (RQ1) Catcher (RQ2)
Effectiveness
165 (mean=123)
unique misuses
243 (mean=207) unique
misuses
Efficiency
~15 days to produce
the test cases for all
the projects (~17
hours per project)
~2 days to produce the
test cases for all the
projects (~2 hours per
project)
Detected misuses overlap:
1632 80
EvoSuite Catcher
21. Implications
!21
Research
New research directions towards focused search-based testing.
Automated generation of the test oracle.
Further examination of the benefits of combining static analysis and testing.
Practical
Easily deployable in a continuous integration pipelines.
Improvement of the reliability of APIs leading to fewer client application crashes.
24. !24
Catcher
!X
The combination of static exception propagation analysis and
search-based test case generation for
the effective and efficient detection of API misuses.
Client
1. Spotting
potential runtime
exceptions throws
2. See which ones
of those exceptions
can propagate to
the client
API
3. Filter trivial false
positives (e.g. try/
catch)
4. Use the candidate misuses
as test objectives for a search-
based test case generation
algorithm
tests
5. Expose the misuses
Evaluation on the JDK API
• RQ1: How do existing unit level coverage-based test
generation tools perform in discovering API misuses?
• RQ2: Does Catcher improve the performance of existing
test coverage-based approaches on detecting API
misuses?
• RQ3: What types of API misuses does Catcher expose?
• Comparison with (plain) EvoSuite
• DynaMOSA + default coverage criteria
• On all classes of the client apps
• 25 runs x 25 minutes x each class under test
!X
Client project Files
(#)
LOC (K)
bcel-6.2 489 39
commons-cli-1.4 50 7
commons-codec-1.12 124 10
commons-collections-4.2 535 63
commons-compress-1.17 352 43
commons-lang-3.7 323 76
commons-math-3.6.1 1617 209
easymock-3.6 204 14
gson-2.8.5 206 25
hamcrest-core-1.3 152 7
jackson-databind-2.9.6 919 114
javassist-3.23.1 527 82
jcommander-1.71 139 6
jfreechart-1.5.0 990 134
jodat-time-2.10 330 86
jopt-simple-5.0.4 192 9
natty-0.13 27 3
neo4j-java-driver-1.6.2 510 52
shiro-core-1.3.2 653 31
xwiki-commons-job-10.6 67 3
xwiki-commons-text-10.6 3 101
Evaluation results
!X
EvoSuite (RQ1) Catcher (RQ2)
Effectiveness
166 (mean=123)
unique misuses
243 (mean=207) unique
misuses
Efficiency
~15 days to produce
the test cases for all
the projects (~17
hours per project)
~2 days to produce the
test cases for all the
projects (~2 hours per
project)
Detected misuses overlap:
1632 80
EvoSuite Catcher
Implications
!X
Research
New research directions towards focused search-based testing.
Automated generation of the test oracle.
Further examination of the benefits of combining static analysis and testing.
Practical
Easily deployable in a continuous integration pipelines.
Improvement of the reliability of APIs leading to fewer client application crashes.
25. !25
Eective and Eicient API Misuse Detection via Exception
Propagation and Search-Based Testing
Maria Kechagia∗
m.kechagia@ucl.ac.uk
University College London
London, United Kingdom
Xavier Devroey
x.d.m.devroey@tudelft.nl
Delft University of Technology
Delft, Netherlands
Annibale Panichella
a.panichella@tudelft.nl
Delft University of Technology
Delft, Netherlands
Georgios Gousios
g.gousios@tudelft.nl
Delft University of Technology
Delft, Netherlands
Arie van Deursen
arie.vandeursen@tudelft.nl
Delft University of Technology
Delft, Netherlands
ABSTRACT
Application Programming Interfaces (APIs) typically come with
(implicit) usage constraints. The violations of these constraints (API
misuses) can lead to software crashes. Even though there are sev-
eral tools that can detect API misuses, most of them suer from a
very high rate of false positives. We introduce Catcher, a novel API
misuse detection approach that combines static exception propa-
gation analysis with automatic search-based test case generation
to eectively and eciently pinpoint crash-prone API misuses in
client applications. We validate Catcher against 21 Java applications,
targeting misuses of the Java platform’s API. Our results indicate
that Catcher is able to generate test cases that uncover 243 (unique)
API misuses that result in crashes. Our empirical evaluation shows
that Catcher can detect a large number of misuses (77 cases) that
would remain undetected by the traditional coverage-based test
case generator EvoSuite. Additionally, on average, Catcher is eight
times faster than EvoSuite in generating test cases for the identi-
ed misuses. Finally, we nd that the majority of the exceptions
ACM Reference Format:
Maria Kechagia, Xavier Devroey, Annibale Panichella, Georgios Gousios,
and Arie van Deursen. 2019. Eective and Ecient API Misuse Detection via
Exception Propagation and Search-Based Testing. In Proceedings of the 28th
ACM SIGSOFT International Symposium on Software Testing and Analysis
(ISSTA ’19), July 15–19, 2019, Beijing, China. ACM, New York, NY, USA,
12 pages. https://doi.org/10.1145/3293882.3330552
1 INTRODUCTION
Developers use external libraries to increase the velocity and reduce
the production cost of software projects [38]. While increasing pro-
ductivity, this form of software reuse comes with several challenges:
dependencies need to be kept up to date [15], developers must learn
the intricacies of each imported Application Programming Interface
(), and resulting client programs should be robust, ecient, and
responsive. Correctly using third-party s is not an easy task;
many s are millions of lines of code large, interact with various
https://github.com/mkechagia/Catcher
26. Effective and Efficient API Misuse Detection via
Exception Propagation and Search-Based Testing
Maria Kechagia, Xavier Devroey, Annibale Panichella,
Georgios Gousios and Arie van Deursen
