Presentation of research on recommendation systems for bug management in large software engineering projects. Contains a background and some highlights from my own research. Target audience: industry practitioners

1. Lund University - cs.lth.se/markus_borg/
Recommendation Systems for Issue
Management
MARKUS BORG, SOFTWARE ENGINEERING RESEARCH GROUP

2. Lund University - cs.lth.se/markus_borg/
• PhD student 2010-
• Software engineering
• Requirements-test alignment
• Member of SWELL
• Supervisor: Per Runeson
Markus Borg
Per Runeson
• MSc CS and Eng. 2007
• Developer, ABB (2007-2010)
• Compiler and editor development
• Safety-critical systems
• Embedded development

3. Lund University - cs.lth.se/markus_borg/
Agenda
• Background
– Bug report inflow in large projects
– Recommendation systems
• Overview of some ongoing bug research
– Duplicate detection
– Team assignment
– Effort prediction
– Change impact analysis

4. Lund University - cs.lth.se/markus_borg/
There Will Be Bugs!

5. Lund University - cs.lth.se/markus_borg/
Issue Reports as Batons in the
Development “Relay Race”
Release
Change Board
Project manager
Tester
Issue handover
Developer
Developer Tester
Developer
Architect
Analyst
Designer

6. Lund University - cs.lth.se/markus_borg/
Challenges in the Bug Overflow
• 10 000s of bug reports in large projects
– Continous inflow of bug reports
– Limited resources to deal with it
• Q1. Which issues do we already know about?
• Q2. Who should investigate the issue?
• Q3. How long time will it take to correct this bug?
• Q4. If we make a corrective change, what is the impact?

7. Lund University - cs.lth.se/markus_borg/
Recommendation System
• General definition
“a software application that aim to support users in
their decision-making while interacting with large
information spaces”
– Amazon, Netflix, RateBeer, Filmtipset...
• In software engineering
”a software application that provides information items
estimated to be valuable for a software engineering task
in a given context”

8. Lund University - cs.lth.se/markus_borg/
Two main approaches to recommendations
Bug
example

9. Lund University - cs.lth.se/markus_borg/
Q1. Which issues do we already
know about?
Duplicate Detection

10. Lund University - cs.lth.se/markus_borg/
Issue Duplicate Detection

11. Lund University - cs.lth.se/markus_borg/
Network Analysis of Bug Reports

12. Lund University - cs.lth.se/markus_borg/
Subgraph Analysis in Bug Networks
Bug star
One central bug report
pointing at several others
Dense ring
Most bug reports are
connected

13. Lund University - cs.lth.se/markus_borg/
Textual Similarity Analysis
1. Study natural
language content
2. Calculate overlap
3. Recommend
duplicates

14. Lund University - cs.lth.se/markus_borg/
Calculating Textual Similarity
Issue
reports
Represent
Calculate angles in
vector space

15. Lund University - cs.lth.se/markus_borg/
Reported Results
• Results typically reported as:
– ”% found among top 10 recommendations”
• Sony Ericsson 40%
• Firefox 60%, 90%
• Eclipse 40%, 70%
• Open Office 60%, 65%
• Mozilla Foundation 45%, 65%
• Android 60% (Jens?)
• Are these good results? Good question.

16. Lund University - cs.lth.se/markus_borg/
Q2. Who should investigate the
issue?
Team assignment

17. Lund University - cs.lth.se/markus_borg/
Bug Assignment
Bug
tossing!

18. Lund University - cs.lth.se/markus_borg/
Machine Learning to Find Patterns
• Bugs found in India late in testing phase are severe?
• Bugs reports containing term ”termination” are costly?
• Bugs reported on fridays with attached stack traces tend to
be deferred to later releases?

19. Lund University - cs.lth.se/markus_borg/
Automated Bug Assignment –
Supervised Machine Learning
Machine
learning
Train
Recommend team

20. Lund University - cs.lth.se/markus_borg/
How to Represent a Bug Report?
• Function
• Severity
• System Version
• Submit Date
• Submitter Location
etc.
• … And the text! Title and description.

21. Lund University - cs.lth.se/markus_borg/
Experimental Setup
Ericsson
Company A
Machine
learning
4 x
Pre-processing
& feature
selection

22. Lund University - cs.lth.se/markus_borg/
Results
• Prediction accuracy using state-of-the-art machine learning:
– Ericsson 30-70%
– Company A 30%
• At Ericsson
– In line with accuracy of human assignment
– Deploying tool in selected projects
• Textual features improve predictions, but training times
increase a lot!
– Working on finding a good balance

23. Lund University - cs.lth.se/markus_borg/
Q3. How long time will it take to
correct this bug?
Effort prediction

24. Lund University - cs.lth.se/markus_borg/
Bug Report Clustering
– Unsupervised Machine Learning
Automatically cluster
bug reports based on
natural language in
report

25. Lund University - cs.lth.se/markus_borg/
Prediction of Bug Resolution Time
1. A new bug is submitted
2. Put it in the right cluster
3. Predict its resolution time

26. Lund University - cs.lth.se/markus_borg/
Results
• Clustered bug reports based on textual content
– Android
– Eclipse
– Company A
• Average resolution times in bug clusters are different
– Statistically significant
• Is it a good way to predict resolution times?
– No, at least not yet…

27. Lund University - cs.lth.se/markus_borg/
Q4. If we make a corrective
change, what is the impact?
Change impact analysis

28. Lund University - cs.lth.se/markus_borg/
Recommendations for Impact Analysis
Mandated by safety standards!
Reqs.
Tests

29. Lund University - cs.lth.se/markus_borg/
Networks of Impacted Artifacts
Data mining in archive of previous
impact analysis reports

30. Lund University - cs.lth.se/markus_borg/
Identification of Key Artifacts

31. Lund University - cs.lth.se/markus_borg/
Recommend Possible Impact
1. Textual similarity analysis
2. Follow links
in network

32. Lund University - cs.lth.se/markus_borg/
Recommend Possible Impact
3. Rank recommendations
– Network statistics
– Textual similarity

33. Lund University - cs.lth.se/markus_borg/

34. Lund University - cs.lth.se/markus_borg/
Results
• Experiments in a laboratory setting
– 30% of previous impact among top 5 recommendations
– 40% among top 10 recommendations
• Prototype evaluation in industrial setting ongoing...

35. Lund University - cs.lth.se/markus_borg/
Summary

36. Lund University - cs.lth.se/markus_borg/
Summary
• Issue inflow in large projects can be daunting
• On the other hand, issue reports contain
important information!
• With the right tools, the information overflow
can be tamed to recommendations!