Presentation of the paper "Towards using Data to Inform Decisions in Agile Software Development: Views of Available Data" held on July 28th 2019 at the 14th International Conference on Software Technologies (ICSOFT'19) in Prague. Paper authors: Christoph Matthies, Guenter Hesse. Hasso Plattner Institute, University of Potsdam, Germany

1. Hasso Plattner Institute
University of Potsdam, Germany
christoph.matthies@hpi.de
@chrisma0
Using Data to Inform Decisions in Agile
Software Development
Christoph Matthies, Guenter Hesse
Prague, July ’19

2. Motivation
2
for using data in software engineering
Software engineering must shed the folkloric advice and
anecdotal evidence [...] and replace them with
convincing results, established with [...] empirical methods
– Bertrand Meyer [Meyer, 2013]
“
”[Meyer, 2013] B. Meyer, H. Gall, M. Harman, and G. Succi, “Empirical Answers to Fundamental Software
Engineering Problems (Panel),” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software
Engineering, ser. ESEC/FSE 2013. New York, USA: ACM, 2013, pp. 14–18.
Picture: https://commons.wikimedia.org/wiki/File:Bertrand_Meyer_recent.jpg

3. Importance of Decision-Making
3
in software development projects
■ Software project success hinges on many factors
□ Budget, scope, team organization, technology, architecture
□ Customer engagement, requirements analysis
[Drury, 2012] Drury, M., Conboy, K., and Power, K. (2012). Obstacles to decision making
in Agile software development teams. Journal of Systems and Software, 85(6):1239–1254.
[Taherdoost, 2018] Taherdoost, H. and Keshavarzsaleh, A. (2018). A Theoretical Review on IT Project Success/Failure
Factors and Evaluating the Associated Risks. In Mathematical and Computational Methods in Electrical Engineering

4. Importance of Decision-Making
4
in software development projects
■ Software project success hinges on many factors
□ Budget, scope, team organization, technology, architecture
□ Customer engagement, requirements analysis
■ Correct project decisions critical for success [Drury, 2012]
□ Overruns of budget and schedule
□ Project cancellation [Taherdoost, 2018]
[Drury, 2012] Drury, M., Conboy, K., and Power, K. (2012). Obstacles to decision making
in Agile software development teams. Journal of Systems and Software, 85(6):1239–1254.
[Taherdoost, 2018] Taherdoost, H. and Keshavarzsaleh, A. (2018). A Theoretical Review on IT Project Success/Failure
Factors and Evaluating the Associated Risks. In Mathematical and Computational Methods in Electrical Engineering

5. Motivation
5
for using data in Agile software development processes
Scrum is founded on empirical process control theory [...]
knowledge comes from experience and making decisions
based on what is known. [...]
– The Scrum Guide [Schwaber, 2017]
[Schwaber, 2017] K. Schwaber, J. Sutherland, “The Scrum Guide - The Definitive Guide to Scrum,” 2017,
[online] http://scrumguides.org/docs/scrumguide/v2017/2017-Scrum-Guide-US.pdf
Picture: https://www.scrum.org/resources/2017-scrum-guide-update-ken-schwaber-and-jeff-sutherland
“
”

6. Decision-Making in Agile
6
Evidence-based decision-making in teams
■ Agile teams should make use of available data to enable
“evidence-based decision making” [Fitzgerald, 2014]
■ However, few details available on concrete implementations
[Fitzgerald, 2014] Fitzgerald, B., Musiał, M., and Stol, K.-J. (2014). Evidence-based decision making in lean software project management. In
Companion Proceedings of the 36th International Conference on Software Engineering - ICSE Companion 2014, pages 93–102, New York,
New York, USA. ACM Press.

7. Decision-Making in Agile
7
Evidence-based decision-making in teams
■ Agile teams should make use of available data to enable
“evidence-based decision making” [Fitzgerald, 2014]
■ However, few details available on concrete implementations
Main Research Questions
■ What data is available and relevant to Agile teams?
■ What methods are transferable from other domains?
[Fitzgerald, 2014] Fitzgerald, B., Musiał, M., and Stol, K.-J. (2014). Evidence-based decision making in lean software project management. In
Companion Proceedings of the 36th International Conference on Software Engineering - ICSE Companion 2014, pages 93–102, New York,
New York, USA. ACM Press.

8. Related Work [Svensson, 2019]
8
Unfulfilled Potential of DDDM in Agile
[Svensson, 2019] Svensson, R.B., Feldt, R., & Torkar, R. “The Unfulfilled Potential of Data-Driven Decision Making in Agile Software Development”,
20th International Conference on Agile Software Development (XP), 2019 (preprint), https://arxiv.org/abs/1904.03948

9. Potential of DDDM
9
Lots of potential, little usage [Svensson, 2019]
■ Survey of software practitioners
■ How is data used in the company for making decisions?
[Svensson, 2019] Svensson, R.B., Feldt, R., & Torkar, R. “The Unfulfilled Potential of Data-Driven Decision Making in Agile Software Development”,
20th International Conference on Agile Software Development (XP), 2019 (preprint), https://arxiv.org/abs/1904.03948

10. Data-Driven Decision-Making
10
In business contexts
■ DDDM contrasted with “gut feeling” [Provost, 2013]
□ Standard in business in the absence of analysis technologies
[Provost, 2013] Provost, F. and Fawcett, T. (2013). Data Science and its Relationship to Big Data
and Data-Driven Decision Making. Big Data, 1(1):51–59.
[Brynjolfsson, 2011] Brynjolfsson, E., Hitt, L. M., and Kim, H. H. (2011). Strength in Numbers:
How Does Data-Driven Decisionmaking Affect Firm Performance? SSRN Electronic Journal.
[Schrage, 2016] Schrage, M. (2016). How the big data explosion has changed decision making.
Harvard Business Review Digital Articles, pages 2–5

11. Data-Driven Decision-Making
11
In business contexts
■ DDDM contrasted with “gut feeling” [Provost, 2013]
□ Standard in business in the absence of analysis technologies
■ Technology use in business has fueled DDDM adoption
□ Increased company productivity [Brynjolfsson, 2011]
□ Transparency & real-time information intensifies tensions,
management must adapt [Schrage, 2016]
[Provost, 2013] Provost, F. and Fawcett, T. (2013). Data Science and its Relationship to Big Data
and Data-Driven Decision Making. Big Data, 1(1):51–59.
[Brynjolfsson, 2011] Brynjolfsson, E., Hitt, L. M., and Kim, H. H. (2011). Strength in Numbers:
How Does Data-Driven Decisionmaking Affect Firm Performance? SSRN Electronic Journal.
[Schrage, 2016] Schrage, M. (2016). How the big data explosion has changed decision making.
Harvard Business Review Digital Articles, pages 2–5

12. Agile Software Development
12
Development Processes and their connection to data
■ Plan-driven, rational decision-making [Moe, 2012]
□ Sets of rules that govern behavior
□ Assumption: outcome probabilities are known [Zannier, 2007]
[Moe, 2012] Moe, N. B., Aurum, A., and Dybå, T. (2012). Challenges of shared decision-making:
A multiple case study of agile software development. Information and Software Technology, 54(8):853–865.
[Zannier, 2007] Zannier, C. and Maurer, F. (2006). Foundations of Agile Decision Making from Agile Mentors
and Developers. In Abrahamsson, P., Marchesi, M., and Succi, G., editors, Extreme Programming and
Agile Processes in Software Engineering, pages 11–20, Berlin, Heidelberg. Springer Berlin Heidelberg.

13. Agile Software Development
13
Development Processes and their connection to data
■ Plan-driven, rational decision-making [Moe, 2012]
□ Sets of rules that govern behavior
□ Assumption: outcome probabilities are known [Zannier, 2007]
■ Data enables teams to implement Agile feedback cycles
■ Agile, naturalistic decision-making
□ Situational behavior, context-dependent rules
[Moe, 2012] Moe, N. B., Aurum, A., and Dybå, T. (2012). Challenges of shared decision-making:
A multiple case study of agile software development. Information and Software Technology, 54(8):853–865.
[Zannier, 2007] Zannier, C. and Maurer, F. (2006). Foundations of Agile Decision Making from Agile Mentors
and Developers. In Abrahamsson, P., Marchesi, M., and Succi, G., editors, Extreme Programming and
Agile Processes in Software Engineering, pages 11–20, Berlin, Heidelberg. Springer Berlin Heidelberg.

14. Levels of Decisions
14
The impact of Agile “self-organizing” teams
■ Strategic plans define long-term goals
■ Tactical plans involve managing projects to achieve goals
■ Operational plans define milestones, and the actual work

15. Data Use in Business
15
A wealth of data to use for decisions
■ Businesses gather detailed statistics
□ Users, suppliers, partners & competitors
□ ERP systems for collection, BI for analysis [Chen, 2012]
[Chen, 2012] Chen, H., Chiang, R. H. L., and Storey, V. C. (2012). Business intelligence and analytics:
From big data to big impact. MIS quarterly, 36(4).
[Svensson, 2019] Svensson, R. B., Feldt, R., and Torkar, R. (2019). The Unfulfilled Potential of Data-Driven
Decision Making in Agile Software Development. In Agile Processes in Software Engineering and Extreme
Programming, pages 69–85. Springer International Publishing.

16. Data Use in Business
16
A wealth of data to use for decisions
■ Businesses gather detailed statistics
□ Users, suppliers, partners & competitors
□ ERP systems for collection, BI for analysis [Chen, 2012]
■ Continuing digitization allows additional data collection
□ “Reality mining”: cars, smart devices, etc. [Svensson, 2019]
□ Clickstream data, UI interactions, high time-resolution
[Chen, 2012] Chen, H., Chiang, R. H. L., and Storey, V. C. (2012). Business intelligence and analytics:
From big data to big impact. MIS quarterly, 36(4).
[Svensson, 2019] Svensson, R. B., Feldt, R., and Torkar, R. (2019). The Unfulfilled Potential of Data-Driven
Decision Making in Agile Software Development. In Agile Processes in Software Engineering and Extreme
Programming, pages 69–85. Springer International Publishing.

17. Data in Software Development
17
Much more than code
■ Software development process itself: source of valuable data
□ Intrinsically relevant to making decisions
□ Not only code, but a multitude of supporting artifacts
[de Souza, 2005] de Souza, C., Froehlich, J., and Dourish, P. (2005). Seeking the Source: Software Source Code as a Social and Technical Artifact. In
Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work - GROUP ’05, page 197, New York, New York, USA.
ACM, ACM Press.
[Ziftci & Reardon] Ziftci, C. and Reardon, J. (2017). Who broke the build? Automatically identifying changes that induce test failures in continuous
integration at google scale. In Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in
Practice Track, ICSE-SEIP 2017, pages 113–122.

18. Data in Software Development
18
Much more than code
■ Software development process itself: source of valuable data
□ Intrinsically relevant to making decisions
□ Not only code, but a multitude of supporting artifacts
■ Processes are “inscribed” into software artifacts [de Souza, 2005]
□ E.g. VCS: who, when, what, why, status
□ Also includes evidence of failures [Ziftci & Reardon, 2017]
[de Souza, 2005] de Souza, C., Froehlich, J., and Dourish, P. (2005). Seeking the Source: Software Source Code as a Social and Technical Artifact. In
Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work - GROUP ’05, page 197, New York, New York, USA.
ACM, ACM Press.
[Ziftci & Reardon] Ziftci, C. and Reardon, J. (2017). Who broke the build? Automatically identifying changes that induce test failures in continuous
integration at google scale. In Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in
Practice Track, ICSE-SEIP 2017, pages 113–122.

19. Software Project Data
19
[Guo et al., 2016] Guo, J., Rahimi, M., Cleland-Huang, J., Rasin, A., Hayes, J. H., & Vierhauser, M, “Cold-start software analytics”, In Proceedings of the
13th International Workshop on Mining Software Repositories - MSR ’16, pp. 142–153, New York, New York, USA, ACM Press, 2016.
“a gold-mine of actionable information” [Guo et al., 2016]

20. Software Project Data
20
“a gold-mine of actionable information” [Guo et al., 2016]
Primary purpose: communication
[Guo et al., 2016] Guo, J., Rahimi, M., Cleland-Huang, J., Rasin, A., Hayes, J. H., & Vierhauser, M, “Cold-start software analytics”, In Proceedings of the
13th International Workshop on Mining Software Repositories - MSR ’16, pp. 142–153, New York, New York, USA, ACM Press, 2016.

21. Software Project Data
21
Use this data for reflection
[Guo et al., 2016] Guo, J., Rahimi, M., Cleland-Huang, J., Rasin, A., Hayes, J. H., & Vierhauser, M, “Cold-start software analytics”, In Proceedings of the
13th International Workshop on Mining Software Repositories - MSR ’16, pp. 142–153, New York, New York, USA, ACM Press, 2016.
“a gold-mine of actionable information” [Guo et al., 2016]

22. Scrum Process
22
Where Agile decision-making happens [Schwaber & Sutherland, 2017]
1
2
3
4
5
6
7
Sprint
Planning
Sprint
BacklogBacklog
Sprint
Daily
Stakeholder
Input
Product
4 iterations
Sprint
Retrospective Finished
Increment
Sprint Review
[Schwaber & Sutherland, 2017] Schwaber, K., & Sutherland, J, “The Scrum Guide - The Definitive Guide to Scrum: The Rules of the Game”, 2017,
[online] Available: http://scrumguides.org/docs/scrumguide/v2017/2017-Scrum-Guide-US.pdf

23. Sprint Planning
23
1
2
3
4
5
6
7
Sprint
Planning
Sprint
BacklogBacklog
Sprint
Daily
Stakeholder
Input
Product
4 iterations
Sprint
Retrospective Finished
Increment
Sprint Review
Building a backlog of work for the next Sprint

24. Sprint Planning
24
Building a backlog of work for the next Sprint
■ Selection and prioritization of features for next iteration
□ Based on stakeholders’ experiences & opinions [Svensson, 2019]
□ Lack explanation and links to evidence

25. Sprint Planning
25
Building a backlog of work for the next Sprint
■ Selection and prioritization of features for next iteration
□ Based on stakeholders’ experiences & opinions [Svensson, 2019]
□ Lack explanation and links to evidence
■ Enable short feedback cycles between devs and real customers
□ Metrics from deployed software, real use cases
□ Observe actual (mis)usage, engineering relevant metrics
□ Base changes on results

26. Daily Scrum
26
Quick status update
1
2
3
4
5
6
7
Sprint
Planning
Sprint
BacklogBacklog
Sprint
Daily
Stakeholder
Input
Product
4 iterations
Sprint
Retrospective Finished
Increment
Sprint Review

27. Daily Scrum
27
Quick status update
■ Short daily status meeting
□ “What did I do?”, “What will I do?”, “Blockers?”
□ Strictly time-boxed

28. Daily Scrum
28
Quick status update
■ Short daily status meeting
□ “What did I do?”, “What will I do?”, “Blockers?”
□ Strictly time-boxed
■ Development artifacts hold information on what was done
□ Concrete items that were completed
□ Rely less on personal recollection
□ Optimize for speed

29. Sprint Retrospective
29
Reflecting on past work and improving
1
2
3
4
5
6
7
Sprint
Planning
Sprint
BacklogBacklog
Sprint
Daily
Stakeholder
Input
Product
4 iterations
Sprint
Retrospective Finished
Increment
Sprint Review

30. Retrospectives
30
Reflecting on past work and improving
What should be
improved?
Retrospective
Meeting

31. 31
Did we improve
what we planned?
Retrospective
Meeting
Retrospectives
Reflecting on past work and improving

32. 32
Did we improve
what we planned?
Retrospective
Meeting
Project DataEvidence of last
iteration’s work
code
docs
stats
Retrospectives
Reflecting on past work and improving

33. 33
Did we improve
what we planned?
Retrospective
Meeting
Project DataEvidence of last
iteration’s work
code
docs
stats
Retrospectives
Reflecting on past work and improving

34. Related Research Fields
34
Mining Software Repositories
■ Draw from MSR techniques [Prana, 2018]
■ MSR: Focus on large amounts of repositories
□ “most widely used open source license?” [Dyer, 2013]
■ Little research: Few repositories,
intricate knowledge of creators / users
[Prana, 2018] Prana, G. A. A., Treude, C., Thung, F., Atapattu, T., & Lo, D. “Categorizing the Content of GitHub
README Files”. Empirical Software Engineering. 2018. https://doi.org/10.1007/s10664-018-9660-3
[Dyer, 2013] Dyer, R., Nguyen, H. A., Rajan, H., & Nguyen, T. N. “Boa: A language and infrastructure for
analyzing ultra-large-scale software repositories”. In Proceedings - International Conference on Software
Engineering. pp. 422–431. 2013. IEEE.

35. Conclusion
35
■ Agile requires teams to make decisions based on
best information available [Schuh, 2004]
■ Definition of “best information” ongoing research challenge
[Dybå and Dingsøyr, 2008] Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development:
A systematic review. Information and Software Technology, 50(9-10):833–859.
[Schuh, 2004] Schuh, P. (2004). Integrating Agile Development In The Real World. Charles River Media, Inc.,
Rockland, MA, USA, 1st edition.

36. Conclusion
36
■ Agile requires teams to make decisions based on
best information available [Schuh, 2004]
■ Definition of “best information” ongoing research challenge
■ Individuals can ignore analyses or own preferences due to rules,
traditions or influence of others [Dybå and Dingsøyr, 2008]
[Dybå and Dingsøyr, 2008] Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development:
A systematic review. Information and Software Technology, 50(9-10):833–859.
[Schuh, 2004] Schuh, P. (2004). Integrating Agile Development In The Real World. Charles River Media, Inc.,
Rockland, MA, USA, 1st edition.

37. Conclusion
37
■ Agile requires teams to make decisions based on
best information available [Schuh, 2004]
■ Definition of “best information” ongoing research challenge
■ Individuals can ignore analyses or own preferences due to rules,
traditions or influence of others [Dybå and Dingsøyr, 2008]
Data analysis and interpretation by Agile teams (supported by
context changes) can enable better informed decision
[Dybå and Dingsøyr, 2008] Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development:
A systematic review. Information and Software Technology, 50(9-10):833–859.
[Schuh, 2004] Schuh, P. (2004). Integrating Agile Development In The Real World. Charles River Media, Inc.,
Rockland, MA, USA, 1st edition.

38. Summary
38

39. Image Credits
39
In order of appearance
■ Data directed Decision Making by H Alberto Gongora from the Noun Project (CC BY 3.0 US)
■ Project by useiconic.com from the Noun Project (CC BY 3.0 US)
■ Question by unlimicon from the Noun Project (CC BY 3.0 US)
■ Business by David Glöckler from the Noun Project (CC BY 3.0 US)
■ Agile Development by Bê Sabino from the Noun Project (CC BY 3.0 US)
■ Digitization by priyanka from the Noun Project (CC BY 3.0 US)
■ Developer by becris from the Noun Project (CC BY 3.0 US)
■ Plan by Gregor Cresnar from the Noun Project (CC BY 3.0 US)
■ Daily Scrum by Swen-Peter Ekkebus from the Noun Project (CC BY 3.0 US)
■ Documents by LUFTFI GANI AL ACHMAD from the Noun Project (CC BY 3.0 US)
■ Idea Generation by priyanka from the Noun Project (CC BY 3.0 US)