During the past ten years, static analysis tools have become a vital part of software development for many organizations. However, the question arises, “Can we quantify the benefits of static analysis?” William Oliver presents the results of a Lawrence Livermore National Laboratory study that first measured the cost of finding software defects using formal testing on a system without static analysis; then, they integrated a static analysis tool into the process and, over a period of time, recalculated the cost of finding software defects. Join William as he shares the results of their study and discusses the value and benefits of static testing. Learn how commercial and open source analysis tools can perform sophisticated source code analysis over large code bases. Take back proof that employing static analysis can not only reduce the time and cost of finding defects and their subsequent debugging but ultimately can reduce the number of defects making their way into your releases.
1. Lawrence Livermore National Laboratory
Quantifying the Value of Static
Analysis
Date 5/19/2011
William B. Oliver
Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA 94551
This work performed under the auspices of the U.S. Department of Energy by
Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344
LLNL-PRES-490136
2. What is Static Analysis
The use of tools during pre-testing to remove structural
defects
Software Developer
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
2
3. What is Static Analysis
Static analysis tools provide an in-depth analysis of
source code to find defects.
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
3
4. What is Static Analysis
Finds Defects that compilers and traditional testing miss
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
4
5. What is Static Analysis
Defect types include but not limited to
• Use of uninitialized variables
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
5
6. What is Static Analysis
Defect types include but not limited to
• Use of uninitialized variables
• Memory leaks
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
6
7. What is Static Analysis
Defect types include but not limited to
• Use of uninitialized variables
• Memory leaks
• Null Pointer dereferences
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
7
8. What is Static Analysis
Defect types include but not limited to
• Use of uninitialized variables
• Memory leaks
• Null Pointer dereferences
• Array Bounds Overflows (and many others)
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
8
9. Why Incorporate Static Analysis
Static Analysis finds additional defects
Better Code Coverage
Reduced Developer Debug Time
Advanced Tools find Defects Inter-Procedurally
Uncovers structural defects that can cause Functional
Defects
Finds defects missed during code reviews/walk thrus
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
9
11. Why Incorporate Static Analysis
Enhances Dynamic Testing
• Dynamic testing does not
generally uncover memory
leaks and other structural
defects
• Static Analysis provides 100 %
code coverage
• Structural defects such as
Array Bounds Overflows can
cause Functional Defects
“They are best used in combination
with traditional dynamic testing
techniques, and can even reduce the
cost to create and manage test cases
for stringent run-time coverage..”
Dr. Paul Anderson PhD
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
11
12. Structural Defects vs Functional Defects
Relate to conformance to the
Programming Language rules and
syntax
SD
FD
Uninitialized Data
Memory/Resource Leaks
Array Bounds Overflows
Null Pointer Dereferences
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
12
13. Structural Defects vs Functional Defects
Associated with Features,
Performance, Availability
etc.
SD
FD
Found During Dynamic Testing
Some Causes Include:
Solving the wrong problem
Code Logic Errors
System Integration Issues
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
13
14. Uninitialized Data
Correct Set of Values
Set of Right Answers
y
z
x
Random Set of Values
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
14
17. NULL Pointer Dereference
This memory location contains the address of this memory location
Pointer
Value of the contents of address pointed to by the pointer
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
17
19. Typical Static Analysis Work Flow
Perform
Static
Analysis
Add
Features
Test Fixes
Analyze
Defects
Fix Defects
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
19
20. Advantages for Testers
o Less Wasted Time
o Allows more time for test case development
o Better Test Cases
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
20
21. Assumptions About Time to Find Defects
One Million Lines of Code
Static Analysis
1000 Defects
20 Percent False Positives
800 Valid Defects
Time to Run Code Thru Tool Negligable
Ten Minutes Per Defect to Triage
Dynamic Testing
Automated Testing: 1 hour per defect
Includes Test Case Development
Test Evaluation
Test Report Generation
Manual Testing: 2 hours per Defect
1000 Defects
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
21
22. Automated Testing
TD = Total Defects = SD + FD
Time = Time to Find SD + Time to find FD
TD = 800 + 1000 = 1800
SD Time = (1000 Defects * 10 min/defect)/60min/hour = 166.67 hours
Time = 166.67 + 1000 = 1166.67 hours
Time/TD = 1166.67/1800 = .65 hours/defect = 39 minutes per defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
22
23. Test Case: Automated Testing
Code Type:
Programming Language:
Number of Developers:
Source Lines of Code Analyzed:
Scientific Simulation
C++
4
161,880
Total Number SD found:
528
Total Number SD Analyzed:
190
Number of False Positives:
55
Average Analysis Time/Defect:
8.9 minutes
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
23
24. Test Case: Automated Testing
TD = Total Defects = SD + FD
FD = 297 for dynamic testing and 1 hour per defect
TD = 135 + 297 = 432
SD Time = (190 Defects * 8.9 min/defect)/60min/hour = 28 hours
Time = 28 + 297 = 325 hours
Time/TD = 325/432 = .75 hours/defect = 45 minutes per defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
24
25. Just For Fun What If All 528 defects were triaged
Assuming 28 % False Positive Rate
Total Number SD found:
528
Total Number SD Analyzed:
528
Number of False Positives:
148
Average Analysis Time/Defect:
8.9 minutes
Estimated number of real defects = 380
Estimated Time = (528 * 8.9) / 60 = 78 hours
TD = 380 + 297 = 677
Time = 78 + 297 = 375 hours
Time/TD = 375/677 = .55 hours/defect = 33 minutes per defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
25
26. Manual Testing
Code Type:
Programming Language:
Security Access
C#
Number of Developers:
Total Number SD found:
76
Total Number SD Analyzed:
35
Number of False Positives:
0
Average Analysis Time/Defect:
3.4 minutes
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
26
27. Manual Testing
FD = 339 for dynamic testing and 5 hours per defect
TD = 35 + 339 = 374
SD Time = (35 Defects * 3.4 min/defect)/60min/hour = 2 hours
Time = 2 + 1695 = 1697 hours
Time/TD = 1697/374 = 4.5 hours/defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
27
28. Bottom Line
Combined with dynamic testing
Static Analysis results in finding
more Defects
And the organization spends less
time per defect in the process
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
28
29. Summary
For Static Analysis the time to Find
a defect is less than or equal to 10
minutes
Dynamic Testing:
Automated: 1 hour per Defect
Manual: 4 – 5 hours per Defect
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
29
30. New Breed of Tester
Perform
Static
Analysis
Analyze
Defects
Lawrence Livermore National Laboratory
Option:UCRL#
Option:Additional Information
30
Perform Static Analysis: Check out code and build with static analysis tool (Usually automated via nightly run after all code has been checked into the repository) Supports continuous integrationAnalyze Defects: Developers review results and mark defects as either False Positives or Defects Needed to be Fixed and sets prioritiesFix Defects: Developers fix defects from the analysis phaseTest Fixes: Developers perform necessary unit tests to verify that the code provides the required functionality. Add Functionality: Developers continue the development process. Code check-in occurs here.
Less Wasted Time: By applying static analysis the code is more testable the first time it is delivered to the test team. Allowing the test team to test more functionality early on in the test cycle. Also by fixing structural defects during software development some functional defects get fixed.Allows more time for test case development: Testers can now focus on what they do best, develop test cases.Better Test cases: Testers have more time to add test cases that improve code coverage.
Time Per Defect varied considerable among developers from a little over 52 minutes per defect to as low as 3.58 minutes per defect. Attributable to the learning curve.Above number reflect the top three severity levels (Critical, Severe, Error) only