Automatic patch generation learned from human written patches

Automatic Patch Generation Learned
from Human-Written Patches
Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim
The Hong Kong University of Science and Technology, China
24 May 2013
the 35th International Conference on Software Engineering (ICSE 2013)

3
GenProg
C. Le Goues, M. Dewey-Vogt, S. Forrest, and W.Weimer,“A systematic study of automated
program repair: Fixing 55 out of 105 bugs for $8 each,” in ICSE ’12.

3
GenProg
State-of-the-art

3
GenProg
State-of-the-art
Genetic Programming

3
GenProg
State-of-the-art
Genetic Programming
Random Mutation

3
GenProg
State-of-the-art
Genetic Programming
Random Mutation
Systematically Evaluated

1500 num = state.parenCount;
1501 int kidMatch = matchRENodes(state, (RENode)ren.kid,
1502 stop, index);
1503 if (kidMatch != 1) return kidMatch;
1504 for (int i = num; i < state.parenCount; i++)
1505 state.parens[i].length = 0;
1506 state.parenCount = num;
4
in Interpreter.java
reported as Mozilla Bug #76683
Buggy Code

1502 stop, index);
4
in Interpreter.java
reported as Mozilla Bug #76683
Null Pointer Exception
Buggy Code

1502 stop, index);
5
GenProg repairs bugs

1502 stop, index);
1502 stop, index);
1502 stop, index);
1505 {
1506 // deleted.
1507 }
5
1502 stop, index);
Buggy
Code
GenProg

1502 stop, index);
1502 stop, index);
1502 stop, index);
1505 {
1506 // deleted.
1507 }
5
1502 stop, index);
Buggy
Code
GenProg
This patch passes ALL test cases.

6
GenProg repairs bugs1500 num = state.parenCount;
1502 stop, index);
1502 stop, index);
1505 {
1506 // deleted.
1507 }

6
1502 stop, index);
1502 stop, index);
1505 {
1506 // deleted.
1507 }
1502 stop, index);
1502 stop, index);
1505 {
1506 // deleted.
1507 }

1502 stop, index);
6
1502 stop, index);
1502 stop, index);
1505 {
1506 // deleted.
1507 }
1502 stop, index);
1502 stop, index);
1505 {
1506 // deleted.
1507 }

7
1502 stop, index);
1502 stop, index);
1505 {
1506 // do nothing.
1507 }
Would you accept?

7
1502 stop, index);
1502 stop, index);
1505 {
1506 // do nothing.
1507 }
Would you accept?
17
Students
68
Developers

8
1502 stop, index);
1502 stop, index);
1505 {
1506 // do nothing.
1507 }
Would you accept?
17
Students
68
Developers

8
1502 stop, index);
1502 stop, index);
1505 {
1506 // do nothing.
1507 }
Would you accept?
9.4%
90.6%
17
Students
68
Developers

9
•Random
•Unnatural
•Harder
to
understand

10
Random
Evolution
Explores the large
search space

10
Random
Evolution
Explores the large
search space
but sometimes
produces
random results

10
Random
Evolution
Intelligent
Design
Explores the large
search space
but sometimes
produces
random results

10
Random
Evolution
Intelligent
Design
Explores the large
search space Leverages human
knowledgebut sometimes
produces
random results

11
Human-written Patches
Readable

11
Readable
Natural

11
Readable
Natural
Easy to understand

11
Readable
Natural
Easy to understand
We can learn how to generate
patches from human knowledge.

12
Manual
Classiﬁcation
>60,000
Patches
JDT

12
Manual
Classiﬁcation
#patches
Patterns
>60,000
Patches
JDT

12
Manual
Classiﬁcation
#patches
Patterns
Top frequent
patterns
account for >20~30%
>60,000
Patches
JDT

13
Common Fix Patterns
Altering method parameters
obj.method(v1,v2)0→0obj.method(v1,v3)
Altering method parameters
obj.method(v1,v2)0→0obj.method(v1,v3)

13
Common Fix Patterns
Adding a checker
obj.m1())→)if(obj'!='null)){obj.m1()}
Adding a checker
obj.m1())→)if(obj'!='null)){obj.m1()}

14
PAR
Pattern-based Automatic Program Repair

if(lhs == DBL_MRK) lhs = ...;
if(lhs == undefined) {
lhs = strings[pc + 1];
}
Scriptable calleeScope = ...;
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Patch Candidate Generation
Fail
Pass
(c) Patch Evaluation
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
15

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
15
+
-
+
Human Knowledge

16
+
-
+
Using Human
Knowledge
for patch generation

16
+
-
+
Fix Templates
Using Human
Knowledge

16
+
-
+
Fix Templates
Program Edit Script
Using Human
Knowledge

16
+
-
+
Fix Templates
Program Edit Script
Using Human
Knowledge
10

16
+
-
+
Fix Templates
Program Edit Script
Manually created from ﬁx patterns
Using Human
Knowledge
10

16
+
-
+
Fix Templates
Program Edit Script
Manually created from ﬁx patterns
Highly reusable
Using Human
Knowledge
10

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
17

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
18
}
Fault Localization

19
Fault
Locations
J.A. Jones, M. J. Harrold, and J. Stasko,“Visualization of test information to assist fault localization,” in Proceedings
of the 24th International Conference on Software Engineering, NewYork, NY, USA, 2002, pp. 467–477.

19
TP TF
Fault
Locations

19
TP TF
Fault
Locations
Fault
locations

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
20
}
Fault Localization

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
21

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
22
Template-based
+
-
+
-
+
+
Fix
Template
Patch
Candidate
Fault
Location

+
-
+
23
How a Fix Template works

+
-
+
23
AST Analysis
→Collects necessary AST nodes

+
-
+
23
AST Analysis
Context Check
→Examines applicability

+
-
+
23
AST Analysis
Context Check
→Examines applicability
Program Editing
→Modiﬁes the source code

Using a Fix Template:An Example
24
1502 stop, index);

+
-
+Null Pointer Checker
24
1502 stop, index);

+
-
24
obj ref.: state, parens[i], ...
1502 stop, index);

+
-
24
Check obj ref.: PASS
1502 stop, index);

+
-
24
Edit: Insert
...
...
+ if( ) {
state.parens[i].length = 0;
+ }
...
...
state != null && state.parens[i] != null
1502 stop, index);

+
-
24
Edit: Insert
...
...
+ if( ) {
+ }
...
...
1502 stop, index);
1503 if (kidMatch != -1) return kidMatch;
1502 stop, index);
1505 {
1506 // deleted.
1507 }
1502 stop, index);
1502 stop, index);
1505 {
1506 if( state != null && state.parens[i] != null)
1508 }
1502 stop, index);

+
-
24
Edit: Insert
...
...
+ if( ) {
+ }
...
...
1502 stop, index);
1502 stop, index);
1505 {
1508 }
1502 stop, index);
1502 stop, index);
1505 {
1508 }
1502 stop, index);

25
List of Templates
Parameter Replacer
Method Replacer
Parameter Adder
and Remover
Expression Replacer
Expression Adder and
Remover
Object Initializer
Range Checker
Collection Size
Checker
Null Pointer Checker
Class Cast Checker

26
Fault location #1
Fault location #2
Fault location #3
Fault location #4
Fault location #N
...
Creating Patch Candidates

26
Fault location #1
Fault location #2
Fault location #3
Fault location #4
Fault location #N
...
+
-
+
...
+
-
+
+
-
+
+
-
+
+
-
+
Fix
Templates

26
Fault location #1
Fault location #2
Fault location #3
Fault location #4
Fault location #N
...
+
-
+
...
+
-
+
+
-
+
+
-
+
+
-
+
Fix
Templates
...
Patch
Candidates

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
27
Template-based
+
-
+
-
+
+
Fix
Template
Patch
Candidate
Fault
Location

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
28
Patch Evaluation

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
29
Patch Evaluation

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
29
Patch Evaluation
T
Repaired
Test
Cases

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
29
Patch Evaluation
T
Repaired
PassTest
Cases

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
29
Patch Evaluation
T
Repaired
T
Repaired
PassTest
Cases

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
29
Patch Evaluation
T
Repaired
T
Repaired
Pass
Fail
Test
Cases

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
29
Patch Evaluation
T
Repaired
T
Repaired
+
-
+
-
+
+
Fix
Template
Patch
Candidate
Fault
Location
Pass
Fail
Test
Cases

}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
30

31
Evaluation:
Experiment Design

31
Evaluation:
Experiment Design
PAR GenPro

31
Evaluation:
Experiment Design
PAR GenPro
# #

32
RQ1(Fixability): How many bugs
are ﬁxed successfully?
RQ2(Acceptability):Which approach
can generate more acceptable bug
patches?
Evaluation:
Research Questions
#

33
Subject # bugs LOC # test cases
Rhino 17 51,001 5,578
AspectJ 18 180,394 1,602
log4j 15 27,855 705
Math 29 121,168 3,538
Lang 20 54,537 2,051
Collections 20 48,049 11,577
Total 119 351,406 25,051
Experiment Subjects

34
RQ1: Fixability
PAR GenProg
0
6
12
18
24
30

34
RQ1: Fixability
PAR GenProg
0
6
12
18
24
30
27

34
RQ1: Fixability
PAR GenProg
0
6
12
18
24
30
27
16

34
RQ1: Fixability
PAR GenProg
0
6
12
18
24
30
27
16
PAR GenProg
27 16
>

35
RQ2: Acceptability
User Study #1: Ranking between
PAR and GenProg

37
User Study #1
Ranking Patches

37
User Study #1
Bug Description
Ranking Patches

37
User Study #1
Bug Description
Buggy Code
Ranking Patches

37
User Study #1
Bug Description
Buggy Code
Anonymized Patch #1
Ranking Patches

37
User Study #1
Bug Description
Buggy Code
Anonymized Patch #1
Anonymized Patch #2
Ranking Patches

37
User Study #1
Bug Description
Buggy Code
Anonymized Patch #1
Anonymized Patch #2
Anonymized Patch #3
Ranking Patches

37
User Study #1
Bug Description
Buggy Code
Anonymized Patch #1
Anonymized Patch #2
Anonymized Patch #3
Rank patches
17
Students
68
Developers
Ranking Patches

37
User Study #1
Bug Description
Buggy Code
Anonymized Patch #1
Anonymized Patch #2
Anonymized Patch #3
2
1
3
Rank patches
17
Students
68
Developers
Ranking Patches

User Study #1: Results
Student
group
(avg.
ranking)
38
(the lower the better)
Developer
group
(avg.
ranking)

Student
group
(avg.
ranking)
38
0
0.75
1.5
2.25
3
1.72 1.57
2.67
PAR GenProgHuman
Developer
group
(avg.
ranking)

Student
group
(avg.
ranking)
38
0
0.75
1.5
2.25
3
1.72 1.57
2.67
PAR GenProgHuman
Developer
group
(avg.
ranking)
Signiﬁcantly
Different

Student
group
(avg.
ranking)
38
0
0.75
1.5
2.25
3
1.72 1.57
2.67
PAR GenProgHuman
Developer
group
(avg.
ranking)
1
1.35
1.7
2.05
2.4
1.81 1.82
2.36
PAR
GenPro
g
Human
Signiﬁcantly
Different

Student
group
(avg.
ranking)
38
0
0.75
1.5
2.25
3
1.72 1.57
2.67
PAR GenProgHuman
Developer
group
(avg.
ranking)
1
1.35
1.7
2.05
2.4
1.81 1.82
2.36
PAR
GenPro
g
Human
Signiﬁcantly
Different
Signiﬁcantly
Different

Student
group
(avg.
ranking)
38
0
0.75
1.5
2.25
3
1.72 1.57
2.67
PAR GenProgHuman
Developer
group
(avg.
ranking)
1
1.35
1.7
2.05
2.4
1.81 1.82
2.36
PAR
GenPro
g
Human
Signiﬁcantly
Different
Signiﬁcantly
Different
PAR generates better ranking
patches than GenProg

39
RQ2: Acceptability
PAR and GenProg

39
RQ2: Acceptability
PAR and GenProg
User Study #2: Pair-wise Comparison
between
Vs.
PAR or GenProg

41
User Study #2
Pair-wise Comparison

41
User Study #2
Bug DescriptionPair-wise Comparison

41
User Study #2
Bug Description
Buggy Code

41
User Study #2
Bug Description
Buggy Code
Anonymized Patch #1

41
User Study #2
Bug Description
Buggy Code
Anonymized Patch #1
Anonymized Patch #2

41
User Study #2
Bug Description
Buggy Code
Anonymized Patch #1
Anonymized Patch #2
Choose acceptable
patch(es)
72
Students
96
Developers

41
User Study #2
Bug Description
Buggy Code
Anonymized Patch #1
Anonymized Patch #2
Patch #1 Patch #2
Both Not sure
Choose acceptable
patch(es)
72
Students
96
Developers

GenProg
42
PAR

GenProg
42
0
10
20
30
40
21
28
37
14
responses(%)
PAR HumanBoth Not
Sure
PAR

GenProg
42
0
10
20
30
40
21
28
37
14
responses(%)
PAR HumanBoth Not
Sure
PAR
0
15
30
45
60
20
12
51
17responses(%)
GenProg HumanBoth Not
Sure

GenProg
42
0
10
20
30
40
21
28
37
14
responses(%)
PAR HumanBoth Not
Sure
PAR
0
15
30
45
60
20
12
51
17responses(%)
Sure
49%

GenProg
42
0
10
20
30
40
21
28
37
14
responses(%)
PAR HumanBoth Not
Sure
PAR
0
15
30
45
60
20
12
51
17responses(%)
Sure
49%
32%

GenProg
42
0
10
20
30
40
21
28
37
14
responses(%)
PAR HumanBoth Not
Sure
PAR
0
15
30
45
60
20
12
51
17responses(%)
Sure
49%
32%
PAR generates more
acceptable patches than GenProg

43
Limitations
• Fix templates are written manually.
• But it is one-time cost, these are highly reusable.
•We entirely re-implemented GenProg by Java.
•All subjects are collected from open source projects.
• Some participants may not be thoroughly qualiﬁed.

44
Summary
Observed common patches
#patches
Patterns

44
Summary
#patches
Patterns
Fix Templates and PAR
}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location

44
Summary
Can ﬁx more bugs with more acceptability
#patches
Patterns
}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
0
6
12
18
24
30
16
27
0
0.75
1.5
2.25
3
1
1.35
1.7
2.05
2.4
0
10
20
30
40
PAR HumanBoth Not
Sure
0
15
30
45
60
Sure
49%
32%

45
Future Work
Automatic Fix Template Identiﬁcation
• More templates can ﬁx more bugs.
More Test Cases
• More test cases may lead us to better patches.

46
Acknowledgement
Special Thanks to:
Westley Weimer,
Claire Le Goues,
Thomas Zimmermann,
and Christian Bird

47
Summary
Can ﬁx more bugs with more acceptability
#patches
Patterns
}
Buggy
Program
(a) Fault
Localization
+
-
+
-
+
+
(b) Template-based
Fail
Pass
T
Repaired
Fix
Template
Patch
Candidate
Repaired
Program
Fault
Location
0
6
12
18
24
30
16
27
0
0.75
1.5
2.25
3
1
1.35
1.7
2.05
2.4
0
10
20
30
40
PAR HumanBoth Not
Sure
0
15
30
45
60
Sure
49%
32%

Automatic patch generation learned from human written patches

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (14)

Más de Sung Kim

Más de Sung Kim (13)

Automatic patch generation learned from human written patches