Bulletproof Networks provides managed hosting services to some of the largest companies in Australia. Bulletproof implements strong isolation of customer environments, and this can present unique challenges when re-using Puppet code across our customer base. Additionally, the environments range in size from small to very large, and our tools + processes need to be able to handle both uses cases equally well. In this talk Lindsay + Mick will cover how Bulletproof's approach to these problems has evolved over the last 4 years, and some of the tools Bulletproof has developed and built upon to provide an awesome service to our customers.

1. Islands

3. Who are
these guys?

4. Mick Pollard
@aussielunix
&
Lindsay Holmwood
@auxesis

5. Puppet users
for > 5 years

6. BULLETPROOF

7. IaaS &
Managed
Services

8. Movember Australian Museum Rebel Group
Blackmores Angus & Robertson Telstra
Perisher BlueScope Steel Woolworths
DMG Radio Clive Peters Deloitte
Clemenger budget.gov.au Nissan
AOC Nova Sydney Airports
Whirlpool Smooth Theiss
Borders Fosters Country Road
Midas Australian Geographic Sensis
Tourism Victoria ABS AusPost
DET FWA Vodafone

9. Using Puppet
since 2008

11. Unique
challenges

12. Strong isolation

13. Network
segregation
with VLANs

14. Central
Puppetmaster
isnʼt an option

15. Thorough
change control

16. Rapid growth

18. How do we
use Puppet?

19. Standalone
systems
(puppetmaster-less)

20. Internal
infrastructure

21. Full customer
environments

23. Standalone
systems
(puppetmaster-less)

24. Campaign
driven
business

25. budget.gov.au

26. movember.com

27. mamamia.com.au

28. Reverse
proxies

29. Nginx
with customisation

30. Rump

31. More detail in
John Ferlitoʼs
talk at 14.00

33. Internal
infrastructure

34. Vanilla

35. One
Puppetmaster

36. Ubuntu
(Lucid or Precise)

38. Full customer
environments

39. Every customer
has their own
puppetmaster

40. “Islands of
Puppet”

41. Copypasta

42. Configuration
drift

44. apache

45. apache

46. apache
customer-a

47. apache
customer-a

48. apache
customer-b
customer-a

49. apache
customer-b
customer-a

50. apache
customer-b
customer-a
customer-c

51. apache
customer-b
customer-a
customer-c

52. apache
customer-b
customer-a customer-d
customer-c

54. Poor code
share

55. What if
customers
edit the code?

56. How do
we maintain
common code?

58. Commonalities

59. Mix of
Puppet
versions

60. 0.25
(as provided by
Ubuntu)

61. 2.7
(as provided by
Puppet Labs)

62. Mix of
Operating
Systems

63. lucid precise
2.7 internal infrastructure some customers
0.25 most customers

65. Passenger > webrick

66. --no-daemonize

67. Default behaviour is
orthogonal to
change control

68. We don't want
systems to change
without control

69. All changes
initiated by
an engineer

70. nodes + roles

71. node 'stlyqy-lvs02.cust.bulletproof.net' {
server { $fqdn: }
include snmp::server::lvs
include sysctl::lvs
include keepalive::lvs
include network::conntrack::modules
include network::conntrack::hashsize
include network::bonding::activebackup
include network::type::bonded_vlan
include ript
}

72. node 'stlyqy-lvs02.cust.bulletproof.net' {
server { $fqdn: }
include snmp::server::lvs
include sysctl::lvs
include keepalive::lvs
include network::conntrack::modules
include network::conntrack::hashsize
include network::bonding::activebackup
include network::type::bonded_vlan
include ript
}

73. define server($collectd_client_report_to='collectd.bulletproof.net') {
include motd
include augeas
include apt
include utils
include puppet::client
include ssh::server
include ssh::authorized_keys
include ntp::client
include postfix::satellite
include ruby::dev
include ruby::rubygems
include bzr::client
include git::common
include git::github
include snmp::server
include vmware::tools
include apparmor::disable
collectd::client { "${fqdn}":
report_to => $collectd_client_report_to
}
}

74. app_server
database_server
file_server
management_server
memcache_server
monitor_server
proxy_server
puppetmaster_server
redis_server
single_server
sphinx_server
static_server

75. Heira
is the future

77. We use
Capistrano

78. ssh
in-a-parallel-for-loop

79. Why cap and
not mcollective?

80. We deploy
everything
with cap

81. Monitoring configuration
Firewall configuration
Web applications
Internal tools

82. Consistent
deployment tool
across all projects

83. Principle of
least surprise

84. Engineers learn
1 tool

85. Puppet is no
different to the
rest of our stack

87. How do we use
cap + Puppet?

88. Puppet changes

89. cap puppet:go ROLES=lvs options="--noop"
cap puppet:go ROLES=lvs

91. Smoke tests

92. $ cap puppet:go options="--noop"
# ...
infmon hosts
serves a Nagios page over https
Finished in 1.67 seconds
1 example, 0 failures

93. Works out all roles
that hosts within a
run belong to

94. Runs tests tagged
with those roles

95. Fast feedback
on
change success

97. Bootstrapping

98. cap node:bootstrap HOSTFILTER=lvs-08.bp.net

99. Takes VM in
unknown state

100. Brings into
known state for
Puppet 2.7 run

102. Limitations

103. Singling out
hosts is tricky

104. Re-using data
across commands
requires... creativity

105. servers = []
run "mysql -e "SHOW MASTER STATUS;" | tail -n 1" do |channel, type, data|
hostname = channel[:host]
filename = data.split(/s+/).first
position = data.split(/s+/).last
servers << {
:hostname => hostname,
:filename => filename,
:position => position
}
end

106. ROLES
and
HOSTFILTER

108. TLDR;

109. There are
edge cases

110. It does the job

112. How have we
tried to solve
them?

113. First iteration

114. Modules

115. modules/apache/
| manifests/init.pp
| files/
| templates/
| lib/
| README.markdown

116. Stored on
GitHub

117. Drink from the
firehose

119. puppet-module-tool

120. github.com
puppetlabs/puppet-module-tool

121. gem install puppet-module

122. Modulefile

123. puppet-module build

124. Turns

125. modules/apache/
| manifests/init.pp
| files/
| templates/
| lib/
| README.markdown

126. into

127. bulletproofnetworks-apache-1.3.0.tar.gz

128. Puppet forge

130. Public
by default

131. Not great if you don't
want to open source
all your secret sauce

132. But puppet-
module-tool is
interesting...

133. ...can we fake
the forge?

135. Pain points

136. Arduous
release
workflow

137. Better suited
for infrequent
changes

138. High barrier of
entry for customers
to submit patches

139. Sharing bugfixes &
improvements
requires significant
refactoring

140. Limited
reporting on
customer lag

141. Second iteration

142. Bundler

144. Gemfile

145. #!/usr/bin/env ruby
source :rubygems
gem 'capistrano', '2.9.0'
gem 'capistrano-ext', '1.2.1'
gem 'colorize'
gem 'puppet', '2.7.13'
gem 'puppet-module'
group :test do
gem 'rspec', '2.8.0'
gem 'mechanize'
gem 'puppet-lint'
end

146. Can we
reuse it?

147. Tim Sharpe
@ GitHub
(@rodjek)

148. Messy working
prototype

149. pre-alpha
quality

150. More research
required

151. Librarian

152. “A framework for
bundlers”

153. librarian-chef
manages Chef
repositories

154. by Jay Feldblum
github.com
applicationsonline/librarian

156. librarian-puppet

157. “You can all stop using git
submodules now”

158. gem install librarian-puppet --pre

159. github.com
rodjek/librarian-puppet

160. Bundler-like
behavior
for Puppet

161. Puppetfile

163. #!/usr/bin/env ruby
forge "http://forge.puppetlabs.com"
mod "puppetlabs/razor"
mod "puppetlabs/ntp", "0.0.3"

164. #!/usr/bin/env ruby
forge "http://forge.puppetlabs.com"
mod "puppetlabs/razor"
mod "puppetlabs/ntp", "0.0.3"
mod "stdlib",
:git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"

165. #!/usr/bin/env ruby
forge "http://forge.puppetlabs.com"
mod "puppetlabs/razor"
mod "puppetlabs/ntp", "0.0.3"
mod "stdlib",
:git => "git://github.com/puppetlabs/puppetlabs-stdlib.git"
mod "apt",
:git => "git://github.com/puppetlabs/puppetlabs-apt.git"
:ref => 'feature/master/dans_refactor'

166. One canonical
module

167. modules/
in
.gitignore

168. librarian-puppet outdated
tells you what
needs to be updated

169. librarian-puppet update
updates your modules

170. Demo!
http://aussielunix.github.com/jenkins-appliance/

172. Policy

173. Always use
a module's
master

174. Make changes
to modules as
usual

175. git commit &&
git push
the module

176. Make module
changes generic
by default

177. Only branch
a module when:

178. 1. something is super
client specific

179. 2. there is an
unmissable deadline

180. 3. testing ideas

181. Set a reminder to
merge changes into
master

182. Use pull requests
on GitHub for
dangerous changes

184. Take aways:

185. Puppet modules
are pretty neat

186. Keep feedback
loops short

187. Code share
is king

189. Do you have
similar
problems?

190. How do you
solve them?

191. Thank you!