Ning Chen's Survey for his PhD Qualifying Examination.

1. 1
A SURVEY ON
AUTOMATIC TEST GENERATION
AND CRASH REPRODUCTION
PhD Qualifying Examination
Ning Chen

2. 2
Outline
• Automatic Test Generation
• Feedback-directed Random Generation
• Captured-Object based Generation
• Genetic Evolution based Generation
• Symbolic Execution
• Dynamic Symbolic Execution
• Automatic Crash Reproduction
• Record-and-replay Approaches
• Static analysis based Approaches
• Summary

3. 3
Automatic Test Generation

4. 4
Automatic Test Generation
Automatic Test
Generation
Symbolic
Search
Execution
Based
Based
Feedback- Object- Genetic Symbolic Dynamic Symbolic
directed capturing Evolution Execution Execution
Common Challenge:
My Approach
Object Creation Challenge

5. 5
Automatic Test Generation
Categories Representatives
Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11
Captured-Object Based MSeqGen’09, OCAT’10
Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09
Symbolic Execution King’76, Clarke’76, JPF’04
Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08

6. 6
Feedback-directed Random Test Generation
Randomly
select a new
method call
Classes to
test
Error-
Generate Execute Examine revealing test
Method Method Execution cases
Sequences Sequences Result
Time Limit
Regression
test cases
Properties
Method
to check Sequences
Method Arguments If Sequence Legal
Generated

7. 7
Feedback-directed Random Test Generation
Randomly
select a new
Classes to method call
test
Error-
Generate Execute revealing test
Method Method cases
Sequences Sequences
Time Limit
Regression
test cases
Properties
Method
to check Sequences
Feedback to guide If Sequence Legal
Generated

8. 8
Feedback-directed Random Test Generation
• Advantages:
• Scalable to large subject programs
• Known to be easy to implement
• Generates no or little redundant method call sequences
• Challenges:
• Generates too many test cases (thousands of test cases)
• Cannot generate complex method sequences, thus may not
achieve good coverage for programs with complex structures

9. 9
Feedback-directed Random Test Generation
01 void execute(tinySQLStatement stmt, String sqlStr) {
02 if (stmt.execute(sqlStr)) {
03 // target branch
04 }
05 }
• cannot cover the target branch because it is difficult to
generate the proper sql statement randomly
• In short, Random approach fails to create test inputs that
can reach deeper part of the subject program.

10. 10
Follow-up works
Combining Dynamic Exec. with Random Generation:
S. Artzi, M. D. Ernst, A. Kie˙zun, C. Pacheco, and J. H. Perkins. Finding the
needles in the haystack: Generating legal test inputs for object-oriented
programs. In 1st Workshop on Model-Based Testing and Object-Oriented,
2006. (Palulu)
Combining Static Exec. with Random Generation:
W. Zheng, Q. Zhang, M. Lyu, and T. Xie. Random unit-test generation with mut-
aware sequence recommendation. In ASE, 2010. (RecGen)
Combining Dynamic + Static Exec. with Random Generation:
S. Zhang, D. Saff, Y. Bu, and M. D. Ernst, Combined static and dynamic
automated test generation, in ISSTA, 2011 (Palus)

11. 11
Follow-up works
• The common goal: more diverse and complex test inputs
that can reach deeper part of the subject program.
• However, there is an alternative…

12. 12
Automatic Test Generation
Categories Representatives
Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11
Captured-Object Based MSeqGen’09, OCAT 10
Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09
Symbolic Execution King’76, Clarke’76, JPF’04
Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08

13. 13
Object Captured Based
Randomly
select a new
method call
Classes to
test
Generate Execute Error-
Method Method revealing test
Sequences Sequences cases
Time Limit
Arguments
Regression
Method test cases
Sequences
Properties If Sequence Legal
Generated
to check
Runtime Existing Test
Captured Cases / Normal
Objects Executions

14. 14
Object Captured Based
• Advantages:
• Can generate test inputs that are complex for random approaches
• Can be integrated with other test generation approaches.
• E.g. Seed objects for Random generation.
• Challenges:
• Fail to demonstrate an object generating method sequence.
• Making a test case less useful.
• Rely heavily on the quality and quantity of existing test cases.
• Captured objects may contain sensitive private information.
• James Clause et. al - ICSE2011

15. 15
Object Captured Based
01 void test_execute() {
02 SQLStatement stmt = load(“…/SQLStatement/001”);
03 String sqlStr = load(“…/String/123”);
02 if (stmt.execute(sqlStr)) {
03 // target branch
04 }
05 }
• Apparently, such a test case would be less useful as it
does not provide much information.

16. 16
Automatic Test Generation
Categories Representatives
Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11
Captured-Object Based MSeqGen’09, OCAT’10
Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09
Symbolic Execution King’76, Clarke’76, JPF’04
Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08

17. 17
Genetic Evolution Based
• Adopts evolutionary computation techniques to synthesize
method sequences for reaching higher code coverage.
• Method sequences (chromosome) are crossed-over and
mutated to produce different varieties of offspring.
• The fitness score is computed according to the code
coverage.
• Method sequences that can achieve higher code coverage will be
selected for future evolution.

18. 18
Genetic Evolution Based
• Advantages:
• Have stronger capabilities of generating complex test inputs over
random approaches.
• Challenges:
• Might not scale to large subject programs.
• Factors such as fitness functions, mutation operators, evolution
algorithms can greatly affect the quality of the generations.

19. 19
Automatic Test Generation
Categories Representatives
Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11
Captured-Object Based MSeqGen’09, OCAT’10
Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09
Symbolic Execution King’76, Clarke’76, JPF’04
Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
Search Based Test Generation Approaches

20. 20
Automatic Test Generation
Categories Representatives
Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11
Captured-Object Based MSeqGen’09, OCAT’10
Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09
Symbolic Execution King’76, Clarke’76, JPF’04
Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
Symbolic Execution Based Test Generation Approaches

21. 21
Symbolic Execution
• Proposed as early as the 70s but become realistic only in
recent years
• Generate tests by symbolically executing all paths in the
program
• Rely on constraint solvers to compute test inputs (i.e.
models)

22. 22
Symbolic Execution
01 void foo(int x, int y) {
02 if (x > y + 1) {
03 // branch 1
04 }
05 else {
06 // branch 2
07 }
08 }

23. 23
Symbolic Execution
01 void foo(int x, int y) { Symbolic Input: X, Y
02 if (x > y + 1) {
Path Condition: X > Y + 1
03 // branch 1
04 }
05 else { Constraint Solver -> X = 2, Y = 0
06 // branch 2
07 } Generate actual test case
08 }

24. 24
Symbolic Execution
01 void foo(int x) { Symbolic Input: X, Y
02 if (x > y + 1) {
Path Condition: X <= Y + 1
03 // branch 1
04 }
05 else { Constraint Solver -> X = 0, Y = 0
06 // branch 2
07 } Generate actual test case
08 }

25. 25
Symbolic Execution
• Advantages:
• Feasible in recent years with the increase of computation power.
• Capable of achieving higher structural coverage than search based
techniques.

26. 26
Symbolic Execution
• Challenges:
• Constraint Solving Limitations:
• Non-linear
• Floating point,
• Bit-vector, etc.
• External Method Calls:
• External Libraries, System Level Calls, etc.
• Cannot Symbolically execution these external method calls.
• The Object Creation Problem:
• In OO, test inputs (objects) needs to be constructed in sequence of
method calls.

27. 27
Follow-up work
Complex Constraint Solving Challenge:
C. S. P˘as˘areanu, N. Rungta, and W. Visser. Symbolic execution with mixed
concrete-symbolic solving. In ISSTA ‟11
I. Erete and A. Orso. Optimizing constraint solving to better support symbolic
execution. In CSTVA‟11
The Object Creation Problem:
S. Thummalapenta, T. Xie, N. Tillmann, P. de Halleux, and W. Schulte.
MSeqGen: Object-oriented unit-test generation via mining source code. In
ESEC/FSE‟09

28. 28
Automatic Test Generation
Categories Representatives
Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11
Captured-Object Based MSeqGen’09, OCAT’10
Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09
Symbolic Execution King’76, Clarke’76, JPF’04
Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08

29. 29
Dynamic Symbolic Execution
• Symbolic execution Ideally can achieve complete
coverage, but usually not the case in reality
• Recent years, Dynamic Symbolic Execution is proposed.
• EXE/KLEE „05
• DART „05
• CUTE „05, JCUTE „06
• PEX „08
• DSE is designed to handle two major challenges in
Symbolic Execution:
• 1) Constraint Solving, and
• 2) External Method Calls

30. 30
Dynamic Symbolic Execution
01 void foo(int x, int y) {
02 if (external (x) == y) { No source access for external()
03 // branch 1
04 }
05 else if (hash(x) > y) { hash() is complex arithmetic
06 // branch 2
07 }
08 }
Key Idea:
Combines Dynamic execution + Symbolic execution

31. 31
Dynamic Symbolic Execution
01 void foo(int x, int y) { Dynamic Exec.
02 if (external(x) == y) { external(0) == 1
03 // branch 1
04 }
05 else if (hash(x) < y) { hash(0) == 10
06 // branch 2
07 }
08 }
First Iteration (Random Input):
Dynamic Exec: x = 0 and y = 0

32. 32
Dynamic Symbolic Execution
01 void foo(int x, int y) { Symbolic Exec.
02 if (external(x) == y) { external(X) == Y
03 // branch 1
04 } external(0) == 1
05 else if (hash(x) < y) {
06 // branch 2 Constraint Solver:
07 } X == 0, Y == 1
08 }
Second Iteration:
Path Condition: external(X) == Y

33. 33
Dynamic Symbolic Execution
01 void foo(int x, int y) { Symbolic Exec.
02 if (external(x) == y) { external(X) != Y
03 // branch 1
04 } hash(X) < Y
05 else if (hash(x) < y) {
06 // branch 2 external(0) == 1
hash(0) == 10
07 }
08 } Constraint Solver:
X == 0, Y == 11
Third Iteration:
Path Condition: external(x) == y && hash(x) < y

34. 34
Dynamic Symbolic Execution
• Advantage:
• Can handle situations where traditional symbolic executions cannot
handle. i.e. Constraint Solving, External Method Calls
• Achieve higher coverage compared to only symbolic execution.

35. 35
Dynamic Symbolic Execution
• Challenges:
• A recent paper by Xiao ‟11. investigates the major challenges faced
by a state-of-the-art DSE framework for C#, Pex.
Project LOC Cov % Object Creation Problem
SvnBridge 17.1K 56.26 11 (42.31%)
xUnit 11.4K 15.54 8 (72.73%)
Math.Net 3.5K 62.84 17 (70.83%)
QuickGraph 8.3K 53.21 10 (100%)
Total 40.3K 49.87 46 (64.79%)
• The object creation challenge accounts for 65% of all cases that
cause code blocks to be not covered by Pex.

36. 36
The Object Creation Problem
void foo(Container container) {
if (container.size >= 10) { Model:
// not covered branch container.size = 10
…
}
The Object Creation Problem :
How do we create and mutate a Container object into size = 10?

37. 37
Dynamic Symbolic Execution
• In object-oriented programs, modifying a non-public field
requires a set of legal method sequence
• e.g. Container.size == 10
• Possible method sequence:
Container.add(Object);
…
Container.add(Object);
• But as the structure of the classes become more complex,
so will the object creation problem.
• Therefore, the object creation problem is an important, yet
unsolved problem in automatic test generation.

38. 38
Demand-driven Object Generation
• To address this OCP, I am working on a project for
Demand-Driven Object Generation.
• Goal: Given a target state -> generate a legal method
sequence that can create and mutate object that satisfy
this target state
• Target state: container.size == 10
• Possible Output: container.add(new Object()); ten times

39. 39
Demand-driven Object Generation
• The approach should be demand-driven, so only legal
sequence satisfying the target state can be generated.
• It is helpful to solve the object creation problem
mentioned.
• Thus, it can improve the coverage for dynamic symbolic
execution approaches for object-oriented programs.

40. 40
Automatic Crash Reproduction

41. 41
Automatic Crash Reproduction
Automatic Crash
Reproduction
Static
Record-
Analysis
and-replay
Based
Common Challenges My Approach

42. 42
Automatic Crash Reproduction
Categories Representatives
Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11
Captured-Object Based MSeqGen’09, OCAT’10
Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09
Symbolic Execution King’76, Clarke’76, JPF’04
Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
Categories Representatives
Record-and-replay Jrapture’00, BugNet’05, ReCrash’08, LEAP’10
Static Analysis Approaches PSE’04, XYLem’09, SnuggleBug’09

43. 43
Record-and-Replay
• Approach:
• Monitoring Phase: Captures/Stores runtime heap & stack objects.
• Test Generation Phase: Generates tests that loads the correct
objects with the crashed methods.
Original Program Execution
Store from heap & stack
Stored
Objects
Load as crashed method params
Recreated Test Case

44. 44
Record-and-Replay
• Advantage:
• Can usually reproduce target crashes reliability
• Challenges:
• Require instrumentation in advance
• Extra runtime overhead, i.e. ReCrash:
• 13-64% performance overhead, and
• 3-90% memory overhead
• Extensive amount of data from users: objects in heap, running
context, etc
• Difficult to obtain ( > 100M )
• Privacy issues: James Clause et. al @ ICSE2011

45. 45
Record-and-Replay
• Existing Frameworks:
• Jrapture ‟00,
• BugNet ’05,
• ReCrash/ReCrashJ ‟08
• LEAP/LEAN ’10
• Existing approaches have been trying various techniques
to reduce the runtime overhead yet maintain a good
reproduction result.

46. 46
Automatic Crash Reproduction
Categories Representatives
Feedback-directed Random Based Randoop’06, Palulu’06, RecGen’10, Palus’11
Captured-Object Based MSeqGen’09, OCAT’10
Genetic Evolution Based GADGET’01, eToc’04, STGP’06, Harman’09
Symbolic Execution King’76, Clarke’76, JPF’04
Dynamic Symbolic Execution KLEE’05, DART’05, CUTE’05, JCUTE’06, PEX’08
Categories Representatives
Record-and-replay Jrapture’00, BugNet’05, ReCrash’08, LEAP’10
Static Analysis Approaches PSE’04, XYLem’09, SnuggleBug’09

47. 47
Static Analysis Approaches
• Microsoft PSE ‘04
• Postmortem static analysis framework.
• Explains the potential cause of a crash through symbolic execution.
• XyLem ’09
• Null-dereference analysis tool for Java language.
• Identifies the precise paths along which null values may flow to de-
reference.
• IBM SnuggleBug ‘09
• Demand-driven backward symbolic execution framework.
• Computes the weakest preconditions of crashes.
• But does not generate actual crash triggering objects from the
computed preconditions.

48. 48
STAR: automatically reproduces failures
• All of the above approaches either only finds error path, or
computes the error triggering preconditions.
• They do not support the full reproduction of a crash from the
crash information.
• We propose STAR, an automatic crash reproduction framework
• Requires only minimal information (crash stack trace, crash type)
• No client site deployment and recording necessary
• No performance overhead
• No need to collect any private data from user.
• Supports both finding out the failure preconditions as well as
recreating the actual failures.

49. 49
STAR: automatically reproduces failures
crash precondition
stack trace processing crash condition preparation
computation
1 2 3
bug report x() at nnn {var == null}
y() at yyy {true}
z() at zzz …
crash
4 test case generation reproduction
program
test
TestGen: A novel test generation technique
cases

50. 50
Backward Symbolic Execution
• Goal: Given a crash location, compute at a public method
entry, the precondition that can trigger the crash
• There have been several previous work in this direction
• XYLem ICSE ‟09
• SnuggleBug PLDI ‟09
• STAR has a more efficient backward symbolic execution
engine that improves existing approaches with heuristics:
• Pruning irrelevant branches
• Smarter backtracking
• Early detection of inner contradictions

51. 51
STAR: automatically reproduces failures
crash precondition
stack trace processing crash condition preparation
computation
1 2 3
bug report x() at nnn {var == null}
y() at yyy {true}
z() at zzz …
crash
4 test case generation reproduction
program
test
TestGen: A novel test generation technique
cases

52. 52
Crash Test Case Generation
• Essentially, this is the same problem as the demand-
driven method sequence generation we‟ve discussed.
• Thus, we combine our efficient backward symbolic
execution approach with the demand-driven method
sequence generation to achieve automatic crash
reproduction.

53. 53
Summary – Automatic Test Generation
Categories Efficiency Coverage Complex Input
Generation
Feedback-directed Random Based
Captured-Object Based
Genetic Evolution Based
Symbolic Execution
Dynamic Symbolic Execution

54. 54
Summary – Automatic Test Generation
Categories Efficiency Coverage Complex Input
Generation
Feedback-directed Random Based
Captured-Object Based
Genetic Evolution Based
Symbolic Execution
Dynamic Symbolic Execution
Demand Driven Object Generation

55. 55
Summary – Crash Reproduction
Categories Efficiency Coverage Complex Input
Generation
Feedback-directed Random Based
Captured-Object Based
Genetic Evolution Based
Symbolic Execution
Dynamic Symbolic Execution
Categories Reproducibility Minimal No Performance
Information Overhead
Record-and-replay
Static Analysis Approaches

56. 56
Summary – Crash Reproduction
Categories Efficiency Coverage Complex Input
Generation
Feedback-directed Random Based
Captured-Object Based
Genetic Evolution Based
Symbolic Execution
Dynamic Symbolic Execution
Categories Reproducibility Minimal No Performance
Information Overhead
Record-and-replay
Static Analysis Approaches
STAR