2. Automated Debugging
• Gupta and colleagues ’05
• Jones and colleagues ’02
• Korel and Laski ’88
• Liblit and colleagues ’05
• Nainar and colleagues ’07
• Renieris and Reiss ’03
• Seward and Nethercote ’05
• Tucek and colleagues ’07
• Weiser ’81
• Zhang and colleagues ’05
• Zhang and colleagues ’06
• ...
3. Automated Debugging
Code-centric
• Gupta and colleagues ’05
• Jones and colleagues ’02
• Korel and Laski ’88
• Liblit and colleagues ’05
• Nainar and colleagues ’07
• Renieris and Reiss ’03
• Seward and Nethercote ’05
• Tucek and colleagues ’07
• Weiser ’81
• Zhang and colleagues ’05
• Zhang and colleagues ’06
• ...
4. Automated Debugging
Code-centric
• Gupta and colleagues ’05
• Jones and colleagues ’02
• Korel and Laski ’88
• Liblit and colleagues ’05
• Nainar and colleagues ’07
• Renieris and Reiss ’03
• Seward and Nethercote ’05
• Tucek and colleagues ’07
• Weiser ’81
• Zhang and colleagues ’05
• Zhang and colleagues ’06
• ...
What about inputs which cause the failure?
5. • Chan and Lakhotia ’98
• Zeller and Hildebrandt ’02
• Misherghi and Su ’06
Data-centric Techniques
6. • Chan and Lakhotia ’98
• Zeller and Hildebrandt ’02
• Misherghi and Su ’06
Delta Debugging
Data-centric Techniques
7. • Chan and Lakhotia ’98
• Zeller and Hildebrandt ’02
• Misherghi and Su ’06
Delta Debugging
Data-centric Techniques
Requires:
1. Multiple executions
2. Large amounts of manual
effort (oracle creation, setup)
8. • Chan and Lakhotia ’98
• Zeller and Hildebrandt ’02
• Misherghi and Su ’06
Delta Debugging
Data-centric Techniques
Requires:
1. Multiple executions
2. Large amounts of manual
effort (oracle creation, setup)
Penumbra
9. • Chan and Lakhotia ’98
• Zeller and Hildebrandt ’02
• Misherghi and Su ’06
Delta Debugging
Data-centric Techniques
Requires:
1. Multiple executions
2. Large amounts of manual
effort (oracle creation, setup)
Penumbra
Comparable
performance
10. • Chan and Lakhotia ’98
• Zeller and Hildebrandt ’02
• Misherghi and Su ’06
Delta Debugging
Data-centric Techniques
Requires:
1. Multiple executions
2. Large amounts of manual
effort (oracle creation, setup)
Requires:
1. Single execution
2. Reduced manual effort
Penumbra
Comparable
performance
13. Intuition and Terminology
Failure-revealing input vector
Failure-relevant subset
(inputs which are useful for investigating the failure)
Approximate failure-relevant subsets by
identifying inputs that reach the failure along
program dependencies.
29. fileinfo
Penumbra Overview
foo: 512 ... bar: 1024 ... baz: 150... total: 150...
Foo
512B
Bar
1KB
Baz
1.5GB
Relevant context:
1. When the failure occurs.
2. Which data are involved in
the failure.
30. fileinfo
Penumbra Overview
foo: 512 ... bar: 1024 ... baz: 150... total: 150...
Foo
512B
Bar
1KB
Baz
1.5GB
13. strcat(out, pview);
In general, it is chosen using
traditional debugging methods.
42. 1: Tainting Inputs
Assign a taint mark to each input as it enters the application.
Per-byte Per-entity Domain specific
43. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
Per-byte Per-entity Domain specific
44. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
Precise
identification
Per-byte Per-entity Domain specific
45. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
Precise
identification
Unnecessarily
expensive
Per-byte Per-entity Domain specific
46. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
Precise
identification
Unnecessarily
expensive
Per-byte Per-entity Domain specific
47. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
Precise
identification
Unnecessarily
expensive
Maintains per -
byte precision
Per-byte Per-entity Domain specific
48. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
Precise
identification
Unnecessarily
expensive
Maintains per -
byte precision
Increases
scalability
Per-byte Per-entity Domain specific
49. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
Precise
identification
Unnecessarily
expensive
Maintains per -
byte precision
Increases
scalability
Per-byte Per-entity Domain specific
50. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
Precise
identification
Unnecessarily
expensive
Maintains per -
byte precision
Increases
scalability
Per-byte Per-entity Domain specific
Maintains per -
byte precision
51. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
Precise
identification
Unnecessarily
expensive
Maintains per -
byte precision
Increases
scalability
Per-byte Per-entity Domain specific
Maintains per -
byte precision
Further increases
scalability
52. 1: Tainting Inputs
Assign a unique
taint mark to each
byte.
(read from files)
Assign the same
taint mark to
related bytes.
(argv, argc, fstat, ...)
Assign taint marks
based on user-
provided
information.
Assign a taint mark to each input as it enters the application.
When a taint mark is assigned to an input, log the
input’s value and where the input was read from.
Precise
identification
Unnecessarily
expensive
Maintains per -
byte precision
Increases
scalability
Per-byte Per-entity Domain specific
Maintains per -
byte precision
Further increases
scalability
54. 2: Propagating Taint Marks
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
55. 2: Propagating Taint Marks
Taint marks flow along only
data dependencies.
Taint marks flow along data and
control dependencies.
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
56. 2: Propagating Taint Marks
Taint marks flow along only
data dependencies.
Taint marks flow along data and
control dependencies.
C = A + B;
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
57. 2: Propagating Taint Marks
Taint marks flow along only
data dependencies.
Taint marks flow along data and
control dependencies.
C = A + B;
1 2
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
58. 2: Propagating Taint Marks
Taint marks flow along only
data dependencies.
Taint marks flow along data and
control dependencies.
C = A + B;
1 21 2
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
59. 2: Propagating Taint Marks
Taint marks flow along only
data dependencies.
Taint marks flow along data and
control dependencies.
C = A + B;
1 21 2
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
60. 2: Propagating Taint Marks
Taint marks flow along only
data dependencies.
Taint marks flow along data and
control dependencies.
C = A + B;
if(X) {
C = A + B;
}
1 21 2
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
61. 2: Propagating Taint Marks
Taint marks flow along only
data dependencies.
Taint marks flow along data and
control dependencies.
C = A + B;
if(X) {
C = A + B;
}
1 21 2
1 2
3
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
62. 2: Propagating Taint Marks
Taint marks flow along only
data dependencies.
Taint marks flow along data and
control dependencies.
C = A + B;
if(X) {
C = A + B;
}
1 21 2
1 2
3
1 2 3
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
63. 2: Propagating Taint Marks
Taint marks flow along only
data dependencies.
Taint marks flow along data and
control dependencies.
C = A + B;
if(X) {
C = A + B;
}
1 21 2
1 2
3
1 2 3
The effectiveness of each option depends on the particular failure.
Data-flow
Propagation (DF)
Data- and control-flow
Propagation (DF + CF)
64. 3: Identifying Relevant-inputs
1. Relevant context indicates
which data is involved in the
considered failure.
2. Identify which taint marks as
associated with the data
indicated by the relevant
context.
3. Use recorded logs to
reconstruct inputs that are
identified by the taint marks.
Baz
1.5GB
71. Evaluation
Study 1: Effectiveness for debugging real failures
Study 2: Comparison with Delta Debugging
Application KLoC Fault location
bc 1.06 10.5 more_arrays : 177
gzip 1.24 6.3 get_istat : 828
ncompress 4.24 1.4 comprexx : 896
pine 4.44 239.1 rfc822_cat : 260
squid 2.3 69.9 ftpBuildTitleUrl : 1024
Subjects:
72. Evaluation
Study 1: Effectiveness for debugging real failures
Study 2: Comparison with Delta Debugging
Application KLoC Fault location
bc 1.06 10.5 more_arrays : 177
gzip 1.24 6.3 get_istat : 828
ncompress 4.24 1.4 comprexx : 896
pine 4.44 239.1 rfc822_cat : 260
squid 2.3 69.9 ftpBuildTitleUrl : 1024
Subjects:
We selected a failure-revealing input vector for each subject.
73. Data Generation
Penumbra Delta Debugging
Setup
(manual)
Execution
(automated)
Choose a relevant
context
Create an automated
oracle
Use prototype tool to
identify failure-relevant
inputs (DF and DF +
CF)
Use the standard Delta
Debugging
implementation to
minimize inputs.
74. Data Generation
Penumbra Delta Debugging
Setup
(manual)
Execution
(automated)
Choose a relevant
context
Create an automated
oracle
Use prototype tool to
identify failure-relevant
inputs (DF and DF +
CF)
Use the standard Delta
Debugging
implementation to
minimize inputs.
75. Data Generation
Penumbra Delta Debugging
Setup
(manual)
Execution
(automated)
Choose a relevant
context
Create an automated
oracle
Use prototype tool to
identify failure-relevant
inputs (DF and DF +
CF)
Use the standard Delta
Debugging
implementation to
minimize inputs.
• Location: statement where
the failure occurs.
• Data: any data read by such
statement
76. Data Generation
Penumbra Delta Debugging
Setup
(manual)
Execution
(automated)
Choose a relevant
context
Create an automated
oracle
Use prototype tool to
identify failure-relevant
inputs (DF and DF +
CF)
Use the standard Delta
Debugging
implementation to
minimize inputs.
77. Data Generation
Penumbra Delta Debugging
Setup
(manual)
Execution
(automated)
Choose a relevant
context
Create an automated
oracle
Use prototype tool to
identify failure-relevant
inputs (DF and DF +
CF)
Use the standard Delta
Debugging
implementation to
minimize inputs.
78. Data Generation
Penumbra Delta Debugging
Setup
(manual)
Execution
(automated)
Choose a relevant
context
Create an automated
oracle
Use prototype tool to
identify failure-relevant
inputs (DF and DF +
CF)
Use the standard Delta
Debugging
implementation to
minimize inputs.
• Use gdb to inspect stack
trace and program data.
• One second timeout to
prevent incorrect results.
79. Data Generation
Penumbra Delta Debugging
Setup
(manual)
Execution
(automated)
Choose a relevant
context
Create an automated
oracle
Use prototype tool to
identify failure-relevant
inputs (DF and DF +
CF)
Use the standard Delta
Debugging
implementation to
minimize inputs.
80. Study 1: Effectiveness
Is the information that
Penumbra provides helpful for
debugging real failures?
81. Study 1 Results: gzip & ncompress
Crash when a file name is longer than 1,024 characters.
82. Study 1 Results: gzip & ncompress
Contents
&
Attributes
Contents
&
Attributes
bar
Contents
&
Attributes
foo./gzip
Crash when a file name is longer than 1,024 characters.
# Inputs: 10,000,056
long
filename[ ]
83. Study 1 Results: gzip & ncompress
Contents
&
Attributes
Contents
&
Attributes
bar
Contents
&
Attributes
foo./gzip
Crash when a file name is longer than 1,024 characters.
# Inputs: 10,000,056 # Relevant (DF): 1
long
filename[ ]
84. Study 1 Results: gzip & ncompress
Contents
&
Attributes
Contents
&
Attributes
bar
Contents
&
Attributes
foo./gzip
Crash when a file name is longer than 1,024 characters.
# Relevant (DF + CF): 3
# Inputs: 10,000,056 # Relevant (DF): 1
long
filename[ ]
85. Study 1 Results: pine
Crash when a “from” field contains 22 or more double quote characters.
86. Study 1 Results: pine
# Inputs: 15,103,766
...
From clause@boar Tue Feb 20 11:49:53 2007
Return-Path: <clause@boar>
X-Original-To: clause
Delivered-To: clause@boar
Received: by boar (Postfix, from userid 1000)
id 88EDD1724523; Tue, 20 Feb 2007 11:49:53 -0500 (EST)
To: clause@boar
Subject: test
Message-Id: <20070220164953.88EDD1724523@boar>
Date: Tue, 20 Feb 2007 11:49:53 -0500 (EST)
From: """"""""""""""""""""""""""""""""@host.fubar
X-IMAPbase: 1172160370 390
Status: O
X-Status:
X-Keywords:
X-UID: 5
...
Crash when a “from” field contains 22 or more double quote characters.
87. Study 1 Results: pine
# Inputs: 15,103,766
...
From clause@boar Tue Feb 20 11:49:53 2007
Return-Path: <clause@boar>
X-Original-To: clause
Delivered-To: clause@boar
Received: by boar (Postfix, from userid 1000)
id 88EDD1724523; Tue, 20 Feb 2007 11:49:53 -0500 (EST)
To: clause@boar
Subject: test
Message-Id: <20070220164953.88EDD1724523@boar>
Date: Tue, 20 Feb 2007 11:49:53 -0500 (EST)
From: """"""""""""""""""""""""""""""""@host.fubar
X-IMAPbase: 1172160370 390
Status: O
X-Status:
X-Keywords:
X-UID: 5
...
… …" " " " " " " " " " " "
Crash when a “from” field contains 22 or more double quote characters.
88. Study 1 Results: pine
# Inputs: 15,103,766 # Relevant (DF): 26
...
From clause@boar Tue Feb 20 11:49:53 2007
Return-Path: <clause@boar>
X-Original-To: clause
Delivered-To: clause@boar
Received: by boar (Postfix, from userid 1000)
id 88EDD1724523; Tue, 20 Feb 2007 11:49:53 -0500 (EST)
To: clause@boar
Subject: test
Message-Id: <20070220164953.88EDD1724523@boar>
Date: Tue, 20 Feb 2007 11:49:53 -0500 (EST)
From: """"""""""""""""""""""""""""""""@host.fubar
X-IMAPbase: 1172160370 390
Status: O
X-Status:
X-Keywords:
X-UID: 5
...
… …" " " " " " " " " " " "
Crash when a “from” field contains 22 or more double quote characters.
89. Study 1 Results: pine
# Relevant (DF + CF):15,100,344
# Inputs: 15,103,766 # Relevant (DF): 26
...
From clause@boar Tue Feb 20 11:49:53 2007
Return-Path: <clause@boar>
X-Original-To: clause
Delivered-To: clause@boar
Received: by boar (Postfix, from userid 1000)
id 88EDD1724523; Tue, 20 Feb 2007 11:49:53 -0500 (EST)
To: clause@boar
Subject: test
Message-Id: <20070220164953.88EDD1724523@boar>
Date: Tue, 20 Feb 2007 11:49:53 -0500 (EST)
From: """"""""""""""""""""""""""""""""@host.fubar
X-IMAPbase: 1172160370 390
Status: O
X-Status:
X-Keywords:
X-UID: 5
...
… …" " " " " " " " " " " "
Crash when a “from” field contains 22 or more double quote characters.
91. Study 1: Conclusions
1. Data-flow propagation is always effective,
data- and control-flow propagation is sometimes
effective.
➡ Use data-flow first then, if necessary, use control-flow.
92. Study 1: Conclusions
1. Data-flow propagation is always effective,
data- and control-flow propagation is sometimes
effective.
➡ Use data-flow first then, if necessary, use control-flow.
2. Inputs identified by Penumbra correspond to the
failure conditions.
➡ Our technique is effective in assisting the debugging of
real failures.
93. Study 2: Comparison with Delta Debugging
RQ1: How much manual effort
does each technique require?
RQ2: How long does it take to
fix a considered failure given
the information provided by
each technique?
95. RQ1: Manual effort
Use setup-time as a proxy for manual (developer) effort.
5,400
12,600
1,8001,800
1259731470163
ncompress bc pine
Setup-time(s)
gzip
Penumbra
Delta Debugging
squid
96. RQ1: Manual effort
Use setup-time as a proxy for manual (developer) effort.
5,400
12,600
1,8001,800
1259731470163
ncompress bc pine
Setup-time(s)
gzip
Penumbra
Delta Debugging
squid
97. RQ1: Manual effort
Use setup-time as a proxy for manual (developer) effort.
5,400
12,600
1,8001,800
1259731470163
ncompress bc pine
Setup-time(s)
gzip
Penumbra
Delta Debugging
squid
98. RQ1: Manual effort
Use setup-time as a proxy for manual (developer) effort.
5,400
12,600
1,8001,800
1259731470163
ncompress bc pine
Setup-time(s)
gzip
Penumbra
Delta Debugging
squid
Penumbra requires considerably less setup time than Delta Debugging
(although more time time overall for gzip and ncompress).
100. RQ2: Debugging Effort
Subject PenumbraPenumbra Delta Debugging
DF DF + CF
bc 209 743 285
gzip 1 3 1
ncompress 1 3 1
pine 26 15,100,344 90
squid 89 2,056 —
Use number of relevant inputs as a proxy for debugging effort.
101. RQ2: Debugging Effort
Subject PenumbraPenumbra Delta Debugging
DF DF + CF
bc 209 743 285
gzip 1 3 1
ncompress 1 3 1
pine 26 15,100,344 90
squid 89 2,056 —
Use number of relevant inputs as a proxy for debugging effort.
• Penumbra (DF) is comparable to (slightly better than) Delta Debugging.
102. RQ2: Debugging Effort
Subject PenumbraPenumbra Delta Debugging
DF DF + CF
bc 209 743 285
gzip 1 3 1
ncompress 1 3 1
pine 26 15,100,344 90
squid 89 2,056 —
Use number of relevant inputs as a proxy for debugging effort.
• Penumbra (DF) is comparable to (slightly better than) Delta Debugging.
• Penumbra (DF + CF) is likely less effective for bc, pine, and squid
103. Conclusions & Future Work
• Novel technique for identifying failure-relevant
inputs.
• Overcomes limitations of existing approaches
• Single execution
• Minimal manual effort
• Comparable effectiveness
• Combine Penumbra with existing code-centric
techniques.