To detect integration errors as quickly as possible, organizations use automated build systems. Such systems ensure that (1) the developers are able to integrate their parts into an executable whole; (2) the testers are able to test the built system; (3) and the release engineers are able to leverage the generated build to produce the upcoming release. The flipside of automated builds is that any incorrect change can break the build, and hence testing and releasing, and (even worse) block other developers from continuing their work, delaying the project even further. To measure the impact of such build breakage, this empirical study analyzes 3,214 builds produced in a large software company over a period of 6 months. We found a high ratio of build breakage (17.9%), and also quantified the cost of such build breakage ranging from 904.64 to 2034.92 man-hours. Interviews with 28 software engineers from the company helped to understand the circumstances under which builds are broken and the effects of build breakages on the collaboration and coordination of teams. We quantitatively investigated the main factors impacting build breakage and found that build failures correlate with the number of simultaneous contributors on branches, the type of work items performed on a branch, and the roles played by the stakeholders of the builds (for example developers vs. integrators). http://mcis.polymtl.ca/publications/2014/icsm14_noureddine.pdf

1. Why do Automated Builds Break?
An Empirical Study M
C IS
Noureddine
Kerzazi
Foutse
Khomh
Bram
Adams
http://imgs.steps.dragoart.com/how-to-draw-a-flying-dragon-http://www.printed-editions.com/upload/standard/Harold_Edgerton_Pigeon_Release_69.jpg dragon-in-flight-step-8_1_000000102347_5.gif

2. Imagine 2,000 developers working in the same
branch, and one breaks the build, then you end
up with 1,999 unhappy people!
[T. Zimmermann @ SCAM 2014]

3. 3
What is "breaking the build"?

4. 3
What is "breaking the build"?
trunk ...

5. branch 1
3
What is "breaking the build"?
trunk ...

6. branch 1
branch 2
3
What is "breaking the build"?
trunk ...

7. branch 1
branch 2
3
What is "breaking the build"?
trunk ...

8. test build
branch 1
branch 2
3
What is "breaking the build"?
integrate
trunk
...

9. test build
branch 1
branch 2
3
What is "breaking the build"?
integrate
trunk
...

10. branch 1
branch 2
3
What is "breaking the build"?
trunk
integrate
test build
...

11. branch 1
branch 2
3
What is "breaking the build"?
trunk
integrate
test build
...

12. branch 1
branch 2
3
What is "breaking the build"?
trunk
integrate
test build
integrate
test build
down-time
...

13. branch 1
branch 2
3
What is "breaking the build"?
trunk
down-time
team-level
breakage
...

14. branch 1
branch 2
3
What is "breaking the build"?
trunk
down-time
integrate
test build
team-level
breakage
...

15. branch 1
branch 2
3
What is "breaking the build"?
trunk
down-time
merge
team-level
breakage
...

16. branch 1
branch 2
3
What is "breaking the build"?
trunk
down-time
merge
team-level
breakage
integrate
test build
...

17. branch 1
branch 2
3
What is "breaking the build"?
trunk
down-time
merge
down-time
team-level
breakage
integrate
test build
...

18. branch 1
branch 2
3
What is "breaking the build"?
trunk
merge
sync
down-time
down-time
team-level
breakage
X
integrate
test build
...

19. branch 1
branch 2
3
What is "breaking the build"?
trunk
merge
sync
down-time
down-time
team-level
breakage
X
integrate
test build
integrate
test build
...

20. branch 1
branch 2
3
What is "breaking the build"?
trunk
merge
sync
down-time
down-time
down-time
team-level
breakage
X
integrate
test build
integrate
test build
...

21. branch 1
branch 2
3
What is "breaking the build"?
trunk
merge
sync
down-time
down-time
down-time
team-level
breakage
X
integrate
test build
integrate
test build
integrate
test build
...

22. branch 1
branch 2
3
What is "breaking the build"?
trunk
merge
sync merge
down-time
down-time
down-time
team-level
breakage
X
integrate
test build
...

23. branch 1
branch 2
3
What is "breaking the build"?
trunk
merge
sync merge
down-time
down-time
down-time
team-level
breakage
X
integrate
test build
integrate
test build
...

24. branch 1
branch 2
3
What is "breaking the build"?
trunk
merge
sync merge
down-time
down-time
down-time
team-level
breakage
project-level
breakage
X
...

25. 4
Build Breakage should
be Avoided!
you"are"stuck
your"team"
members"are"stuck
testers"are"
stuck
(a0er"merge)
other+teams"
deadlines"slip are"stuck
morale"drops

26. 5
RQ1. How does
Build Breakage
Impact a Project?
RQ2. Why does
the Build Break?
RQ3. Which
Factors Impact
Build Breakage?

27. Web-Based, Financial System
6
200 employees
>1.5 MLOC .NET
(8,524 files)
25 branches
6 inter-dependent
projects
2 geographical
sites
agile development
across different
branches
6 months of data

28. triangula9on
measure interview
model 7

29. 8
RQ1. How does
Build Breakage
Impact a Project?
RQ2. Why does
the Build Break?
RQ3. Which
Factors Impact
Build Breakage?

30. Median of 8.3% Broken Builds
per Day (avg. of 22.3%)
9
%broken builds
100
100"
90
90"
80
80"
70
70"
60
60"
50
50"
40
40"
30
30"
20
20"
10
10"
0"
18,07,13"
25,07,13"
01,08,13"
08,08,13"
15,08,13"
22,08,13"
29,08,13"
05,09,13"
12,09,13"
19,09,13"
26,09,13"
03,10,13"
10,10,13"
17,10,13"
24,10,13"
31,10,13"
07,11,13"
14,11,13"
21,11,13"
28,11,13"
05,12,13"
12,12,13"
19,12,13"
26,12,13"
02,01,14"
09,01,14"
16,01,14"
23,01,14"
30,01,14"
0

31. 1000
1000"
900
900"
800
800"
700
700"
600
600"
500
500"
400
400"
300
300"
200
200"
100"
0"
Architecture)
Branch-2)
Branch-3)
Branch-4)
Branch-5)
Branch-6)
Branch-7)
Branch-8)
Branch-9)
QuickFix)
Branch-18)
PreRelease)
Branch-10)
Branch-11)
Branch-12)
Branch-13)
Branch-14)
Branch-15)
Branch-16)
Branch-20)
Branch-21)
Trunk)
Branch-23)
Branch-24)
Branch-25)
DownDme)by)branch)(min))
Average Median Downtime
of 1 Hour per Branch
10
#min.
re-architecting can
cause very long
downtime
100

32. Estimating Cost of Team-Level
Build Breakage for Branch
11
X
integrate
test build
integrate
test build time
integrate
test build
integrate
+
test build time
+
...
$$$(branch)
in man-hours=

33. Estimating Cost of Team-Level
Build Breakage for Project
12
$$$(project)
in man-hours =
$$$(branch 1)
in man-hours
+
$$$(branch 2)
in man-hours
+
...

34. Total of
1.25 to
2.83
Man-
Months
Wasted
(in 6 months)
13
0 100 200 300 400 500
Branch−18
Branch−2
Branch−5
QuickFix
Branch−12
Branch−21
Branch−10
Branch−25
Branch−3
Branch−11
Branch−23
Branch−15
Branch−7
Branch−14
Branch−24
PreRelease
Branch−4
Branch−6
Branch−20
Branch−13
Branch−16
Branch−8
Arch.
Branch−9
Trunk
re-architecting
fixing trunk
upper bound lower
bound
M/H

35. 14
1 hour of downtime
(avg.) per branch
>=1.25 man-months
lost

36. RQ2. Why does
the Build Break? 1 hour of downtime
(avg.) per branch
14
>=1.25 man-months
lost

37. Semi-Structured Interviews
15
architects
technical leads
integrator
front-end developers
back-end developers
testers
>3 years of
experience
20th percentile
most successful/
broken builds
50 minutes

38. Analysis of Interviews
16
awareness (Q1-Q4)
causes (Q5-Q6)
prevention (Q7-Q8)
X
Q1.
Q2.
...
card sort

39. 25% of Interviewees Track
the Build Results
I switched off the Build notifications
to avoid any distraction. I check the
results of my builds only in predefined
time windows within the day.
I’m always aware about the build notification,
especially the red ones. When a build notification for
a branch shows up, I examine not only that build, but
also the history of build failure on that branch.
17

40. Awareness of a Breakage
Takes on Average 3 Hours
When we see in the work items system tracker that the
developer is done and we do not see the build result,
then we have to communicate with the concerned
developer to verify where the latest code to test is.
18

41. Breakage Takes on Average
1 Hour to Resolve
>
19
concerned about
project's overall
consistency
concerned about
own feature/bug fix

42. 3 Major Causes of Build
Breakage
20
forgetting to commit
certain files
(libraries, templates,
configuration files, ...)
committing from
the wrong
workspace (branch)
feature
libA 1.0
libB 2.1
feature
libA 1.0
libB 1.5
build scripts do not
specify transitive
library dependencies

43. Developers Active in 4 to
15 Branches in Parallel
I was assigned bugs on a specific
project within the overall solution. I
unload all other projects from the
solution to focus only on one project.
I am a nomad moving from a branch
to another. I map one workspace by
branch to avoid check-in work in
progress related to other branch.
21

44. Only 12% has Routine for
Avoiding Breakages
Before I check- in, I always compare diligently
my pending changes with the latest version
from the server. In this way I am not wasting
team members’ time with broken builds.
New developers joining a development project
should be required to learn the basic practices
of code committing, as part of the process
known as onboarding
22

45. 23
1 hour of downtime
(avg.) per branch
>=1.25 man-months
lost
25% track CI results
detection takes 3 h
caused by missing file
(dependencie)s and
accidental commits
active in 4-15
branches in parallel

46. 23
RQ3. Which
Factors Impact
Build Breakage?
1 hour of downtime
(avg.) per branch
>=1.25 man-months
lost
25% track CI results
detection takes 3 h
caused by missing file
(dependencie)s and
accidental commits
active in 4-15
branches in parallel

47. Statistical Model of Impact
24
vs. vs. vs. vs. vs.
local vs. remote
code/file churn
commit vs. merge
feature vs. bug fix vs. integration
P1 P2 P3 P4
day of week/time of day
breakage?

48. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
Working_Day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
●
●
●
●
●
10 20 30 40 50
25
larger team
%incMSE
AUC:
0.78

49. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
Working_Day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
●
●
●
●
●
10 20 30 40 50
25
P1 P2 P3 P4
larger team
P1
%incMSE
AUC:
0.78

50. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
Working_Day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
●
●
●
●
●
10 20 30 40 50
25
P1 P2 P3 P4
time of day
larger team
P1
late commits, longer downtime
%incMSE
AUC:
0.78

51. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
Working_Day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
●
●
●
●
●
10 20 30 40 50
25
vs. vs. vs. vs. vs.
P1 P2 P3 P4
time of day
larger team
P1
late commits, longer downtime
%incMSE
architects
AUC:
0.78

52. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
Working_Day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
larger team
●
●
●
●
●
10 20 30 40 50
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integration
P1 P2 P3 P4
time of day
integration > feature > fix
P1
late commits, longer downtime
%incMSE
architects
AUC:
0.78

53. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
Working_Day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
larger team
●
●
●
●
●
10 20 30 40 50
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integration
commit vs. merge
P1 P2 P3 P4
time of day
integration > feature > fix
merge > commit
P1
late commits, longer downtime
%incMSE
architects
AUC:
0.78

54. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
Working_Day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
larger team
●
●
●
●
●
10 20 30 40 50
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integration
commit vs. merge
P1 P2 P3 P4
code churn
file churn
time of day
integration > feature > fix
merge > commit
more churn
P1
late commits, longer downtime
%incMSE
architects
AUC:
0.78

55. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
Working_Day
time of day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
larger team
●
●
●
●
●
10 20 30 40 50
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integration
commit vs. merge
P1 P2 P3 P4
code churn
file churn
local vs. remote
integration > feature > fix
merge > commit
more churn
local teams
P1
late commits, longer downtime
%incMSE
architects
AUC:
0.78

56. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
day of week
time of day
Working_Day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
larger team
●
●
●
●
●
10 20 30 40 50
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integration
commit vs. merge
P1 P2 P3 P4
code churn
file churn
local vs. remote
integration > feature > fix
merge > commit
more churn
local teams
P1
late commits, longer downtime
%incMSE
architects
AUC:
0.78

57. Factors having Higher Breakage
Role
NumberofContributors
Work_Items
Build_Type
Timeline.P1_P4.
Churn..Code.
Churn..Files.
day of week
time of day
Working_Day
Working_Hour
Geog_Distance
●
●
mtcars.rf
●
●
●
larger team
●
●
●
●
●
10 20 30 40 50
25
vs. vs. vs. vs. vs.
feature vs. bug fix vs. integration
commit vs. merge
P1 P2 P3 P4
code churn
file churn
local vs. remote
integration > feature > fix
merge > commit
more churn
local teams
P1
late commits, longer downtime
%incMSE
architects
AUC:
0.78

58. 26
1 hour of downtime
(avg.) per branch
>=1.25 man-months
lost
25% track CI results
detection takes 3 h
caused by missing file
(dependencie)s and
accidental commits
active in 4-15
branches in parallel
ro l e
team size
in branch
type of
work item/
build
timeline

59. branch 1
branch 2
Trunk- vs. Branch-based
Development
3
What is "breaking the build"?
trunk
merge
sync merge
down-time
down-time
down-time
team-level
breakage
project-level
breakage
X
...
triangula9on
measure interview
model 7
Total of
1.25 to
2.83
Man-
Months
Wasted
(in 6 months)
upper bound lower
13
0 100 200 300 400 500
bound
Branch−18
Branch−2
Branch−5
QuickFix
Branch−12
Branch−21
Branch−10
Branch−25
Branch−3
Branch−11
Branch−23
Branch−15
Branch−7
Branch−14
Branch−24
PreRelease
Branch−4
Branch−6
Branch−20
Branch−13
Branch−16
Branch−8
Arch.
Branch−9
Trunk
26
1 hour of downtime
(avg.) per branch
>=1.25 man-months
lost
25% track CI results
detection takes 3 h
caused by missing file
(dependencie)s and
accidental commits
active in 4-15
branches in parallel
role
team size
in branch
type of
work item/
build
timeline