Three years ago Netflix migrated all of its source code to Git for version control. This change furthered the independence of our engineering teams enabling them to manage, build, test and deploy with little friction. But the more distributed our source code becomes, the harder it can be to assess the impact of a change or understand the state of the world. Making a simple change to a shared library could have impact to dozens of critical services so the more feedback engineers can gain, the better. But how do you run an expensive operation against every single git repository within your organization? In this talk, Mike McGarr (Engineering Manager, Developer Productivity) and Ed Bukoski (Senior Software Engineer, Developer Productivity) will share the story of Niagara, a service built to test code changes across all potentially impacted Git repositories at Netflix. We will also share our approach to scaling up Atlassian Bitbucket to handle Niagara’s significant demands.

1. Continuously Integrating
Distributed Code
Ed Bukoski
Mike McGarr

2. Before we begin

3. I have a problem…

4. …that impacts every engineer at Netflix.

5. Affects you too.

6. I don’t know the solution.

7. direction
Image attribution: http://bit.ly/2n0C8NT

8. So what is the problem?

9. SHARED
CODE

10. cloud microservices
Image attribution: http://bit.ly/2wfdGLC

11. client libraries

13. consuming libraries
should be
easy

14. software isn’t static
Image attribution: http://bit.ly/2xbF55b

15. semantic versions

17. (*) 3.1.13
3.1.14
3.1.15
3.1.16
3.2.0
3.2.1
3.2.2
3.2.3
4.0.0-RC1

18. Which version is good?

19. (*) 3.1.13
3.1.14
3.1.15
3.1.16
3.2.0
3.2.1
3.2.2
3.2.3
4.0.0-RC1

20. (*) 3.1.13
3.1.14
3.1.15
3.1.16
3.2.0
3.2.1
3.2.2
3.2.3
4.0.0-RC1 ?

21. Semantic versioning is insufficient

23. Which version of guava
will you get?

26. Just give me a
version that
works!

27. consuming libraries
is actually
hard

28. Well, at least
publishing
is easy, right?

29. Changing foobar-client:3.2.2
• Bug fix? 3.2.3
• Behavior change? 3.2.0
• API change? 4.0.0

30. Who will I break?

31. Who is consuming
my library?

32. Who is using this
API
that I want to change?

33. publishers lack
organization-wide
visibility

34. There are solutions out there…

35. Option #1:
Just deal with pain

36. Option #2:
Share nothing

37. Option #2:
Share nothing little

38. Option #3:
Monorepo

39. • all code in a single repository
• source integration
• head is production
• no library versions
• atomic API changes
• atomic deployments

40. monorepo depends on gates
Image Attribution: http://bit.ly/2vTBIka

42. Our approach…

43. Can we get the benefits
of a monorepo,
without the constraints?

44. Distributed refactoring
Publisher feedback
Managed source

45. Distributed Refactoring

46. How do I
change my API
across hundreds of
repos?

47. 1. find all repositories
2. find all API calls
3. refactor all API calls
4. issue pull requests en masse

48. experimental
Image attribution: http://bit.ly/2wPq8DP

49. Publisher Feedback

50. Who will I break?

51. Astrid
Niagara

54. I now know who I will
break!

55. Managed Source

56. eliminate reduce
consumer pain!

59. Give me a
version that
works!

60. Shorten adoption
curve

62. Surviving the solution

63. Bitbucket req/sec
150 —

64. 150 —
450 —
Bitbucket req/sec

65. Bitbucket req/sec

66. Bitbucket req/sec
Niagara

67. Niagara
Bitbucket req/sec

68. Niagara
Bitbucket req/sec
2000 —

69. Now we have two problems…

70. What did we do first
when we found out about
Niagara ?

71. Niagara ?

72. But we can’t keep Niagara
shutdown forever…

73. How can we scale Bitbucket Server?

74. How can we scale Bitbucket Server?
Scaling Bitbucket is scaling git.

75. Step 1 is look at metrics.

76. JMX —>

78. 4x Load
< 5 minutes

80. Adaptive Throttling

81. Adaptive Throttling
Too optimistic

82. Adaptive Throttling
Too optimistic
Adapted too slow

83. Fixed hosting tickets

84. Fixed hosting tickets
Stopped outages

85. But Niagara is still
dominating requests…

86. JMX —>

87. JMX —>
Niagara

88. JMX —>
Niagara
25% of avail tickets

89. JMX —>
Niagara
25% of avail tickets
Run more often!

90. How can we scale Bitbucket Server?

91. How can we scale Bitbucket Server?
Stopped outages.

92. How can we scale Bitbucket Server?
Stopped outages.
Used JMX metrics to throttle demand.

93. How can we scale Bitbucket Server?

94. How can we scale Bitbucket Server?
Scaling Bitbucket is scaling git.

95. Increase memory and cpu.

96. Current Bitbucket Server
1 Server
AWS
32 CPU
60 GB

97. New Bitbucket Server
1 Server
AWS
32 64 CPU
60 244 GB

98. 32 cpu
60 gb

99. 32 cpu
60 gb
64 cpu
244 gb

100. 32 cpu
60 gb
64 cpu
244 gb
2x

101. How can we scale Bitbucket Server?
Increase memory and cpu.

102. Bitbucket
Data Center

103. Current Bitbucket Server
1 Server
AWS
32 CPU
60 GB

104. Current Bitbucket Server Data Center
1 Server
AWS
32 CPU
60 GB
1 Server
AWS
32 CPU
60 GB
1 Server
AWS
32 CPU
60 GB
1 Server
AWS
32 CPU
60 GB
1 Server
AWS
32 CPU
60 GB
1 Server
AWS
32 CPU
60 GB

105. 192 CPU
360 GB

106. Server

107. Server Data Center

108. How can we scale Bitbucket Server?

109. How can we scale Bitbucket Server?
Stopped outages.

110. How can we scale Bitbucket Server?
Stopped outages.
Used JMX metrics to throttle demand.

111. How can we scale Bitbucket Server?
Stopped outages.
Used JMX metrics to throttle demand.
Increased cpu and memory.

112. How can we scale Bitbucket Server?
Stopped outages.
Used JMX metrics to throttle demand.
Increased cpu and memory.
Switched to Data Center.

113. Now we have two problems…

114. Now we have two one problem…

115. direction
Image attribution: http://bit.ly/2n0C8NT

116. Thank you.