Promise 2011: "Are Change Metrics Good Predictors for an Evolving Software Product Line?" Sandeep Krishnan, Chris Strasburg, Robyn Lutz and Katerina Goseva-Popstojanova.

1. Are Change Metrics Good Predictors for an Evolving Software Product Line? Sandeep Krishnan, ISU Chris Strasburg, ISU & Ames Laboratory Robyn R. Lutz, ISU & JPL, California Institute of Technology Katerina Goseva-Popstojanova, WVU 1 This research is supported by NSF grants 0916275 and 0916284 Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

3. JDT, PDE, Mylyn, Webtools, etc.

4. Reused in more than three products and for more than six years.

5. Low-reuse variation

6. CDT, Datatools, Java EE tools.

7. Reused in three or fewer products and for more than four years.2 Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

8. Related Work Eclipse as a product line. [Chastek, McGregor, and Northrop, 2007], [Linden, 2009], [Krishnan et al., 2011]. Summary of previous work 3 Failure-prone file -A file with one or more non-trivial post-release bugs recorded in the Eclipse Bugzilla database. Important/Good predictor – Predictor providing high information gain for classification of failure-prone files Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

9. Product Line Evolution Product line evolution in two dimensions 4 New Releases P1 R1 P1 R2 P1 R3 P1 Rn P2 R1 P2 R2 P2 R3 P2 Rn New Products Pn R1 Pn R2 Pn R3 Pn Rn Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

10. Motivation Can we leverage the reducing amount of change in product lines to better predict failure-prone files? 5 Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

11. Eclipse case study 6 Blocker Eclipse Classic Critical Eclipse Java Major Eclipse C/C++ Normal Eclipse JavaEE Minor

12. Research Questions As a product evolves, do any change metrics serve as good predictors of failure-prone files? Is there a subset of change metrics which are good predictors across all product line members? Does our ability to predict failure-prone files improve as product line evolves? 7 Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

13. Findings The change metrics provide good classification of the failure-prone files in the Eclipse product line. As each product evolves, there is a stable set of change metrics that are prominent predictors of failure-prone files across its releases. There is a subset of change metrics that is among the prominent predictors of all the products across most of the releases. As the product line matures, prediction performance improves for each of the four Eclipse products. 8 Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

14. Data Source 9 Data Timeline Data Timeline Source of failure reports- Source of change reports – CVS repository of Eclipse. Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

15. Approach 10 Get prediction results + best predictors Weka J48 decision tree learner Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

16. Replication Results 11 Learner performance compared to previous results Classification performance comparison for Eclipse Classic 2.0, 2.1, and 3.0 PC- Percentage of correctly classified instances TPR- True positive rate FPR- False positive rate Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

18. 13 Learner performance improves as single product evolves Extension Results Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

19. 14 Top five predictors for later releases of Eclipse Classic Extension Results Revisions is good predictor for later releases also. Max_changeset is a good predictor also. Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

20. 15 Learner performance improves as product line evolves Extension Results Percentage of correctly classified instances increases across releases for each product Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

21. 16 Learner performance improves as product line evolves Extension Results Percentage of true positives shows improvement across releases for each product Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

22. 17 Learner performance improves as product line evolves Extension Results Percentage of false positives shows reduces across releases for each product Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

23. 18 Top five predictors for four products of Eclipse Product Line Extension Results No common set of predictors across each product and each release. Max_changeset, Revisions and Authors are prominent predictors for all products. Some predictors are prominent for only one product. Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

24. 19 Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011

25. Thank You! 20 Our data is available at http://www.cs.iastate.edu/~lss/PROMISE11Data.tar.gz Dept. of Computer Science, Iowa State University, PROMISE, September 20, 2011