The document provides an overview of how to build a fully automated server deployment system using open source tools such as Request Tracker, Nictool/djbdns, dhcpd, PXEboot, Httpd, a yum repository, and Puppet. Key aspects include using an asset tracker (Request Tracker) to store server information and trigger automated builds. A PXE boot script generates configuration files using data from the asset tracker. A CGI script generates customized Kickstart files which install servers. Puppet then configures and deploys applications to servers based on their roles defined in the asset tracker. The goal is to achieve repeatable, consistent server builds from bare metal to a live application server within an hour with no

1. How to build a fully automated (or nearly so) deployment system using open source tools. Russell Miller SCALE 8x 2010 (16,080, in case you were wondering) [email_address]

3. The methods discussed in this talk will use only open source tools. You don't have to pay for anything I mention here unless you want to. In which case...

4. My paypal address is...

6. Worked on many different kinds of environments

7. Maintained deployment system for 4,000 server environment for leading Internet Shopping Comparison company in West Los Angeles.

8. Instrumental in bringing up two datacenters of about 300 servers each from scratch for leading Internet Advertising company in Irvine, including building the entire deployment system from metal to application.

9. The second datacenter, once all the physical hardware was there, took live traffic in less than two weeks from bare metal.

11. Powerpoint/OpenOffice Impress presentations

12. Funny jokes.

13. Keeping an audience awake

14. And, of course, self-deprecating humor.

16. Deploying servers manually usually mean many cycles of going back and forth with the application owners to validate. For each server. This eats up time and means bringing up servers can take up to or more than two weeks...

17. And consistency is a lost cause.

18. You've already lost control the minute the OS is installed.

19. It's too easy to miss little things.

20. System Administrators are TERRIBLE at documentation.

21. Admit it... you are. No use denying it.

22. What about server spec sheets?

23. NO!

25. It's not their fault. Their focus is on the application. They are not System Administrators. That is why you are paid.

26. They do not accurately represent server builds

27. They often contain incorrect or useless information

28. Frankly, they serve no function other than to waste time and sow confusion, while providing an easily shattered veneer of repeatability.

29. So, naturally, they're usually the first thing tried.

31. The code is the documentation. Server specs can never go out of sync because the server specs are actively deployed.

32. Very tight control over anything that is deployed to the servers.

33. (This means that even if someone installs something you don't want them to, you can simply have it removed within 10 minutes with no manual intervention.)

34. Repeatability.

35. And most importantly, astonished and very happy managers.

36. So do I have your attention? :)

38. Nictool/djbdns (Bind and any other manager will work too, but this is what I use, because nictool has a simple schema and is scriptable)

39. dhcpd

40. PXEboot

41. Httpd (you will see why in a moment)

42. A yum repository

43. And... puppet (cfengine or another configuration management tool will probably work, but I prefer puppet.)

45. Make sure the DNS info is properly entered into your DNS server in whatever way.

46. Tell your DHCP server you want to allow the server to PXE boot. This can happen manually or automatically.

47. I prefer manually simply because if you set the server up to boot automatically – you can get into a situation where the server accidentally reboots and rebuilds itself. This tends to make app owners unhappy.

48. And.... let it install. An hour later you have a full build with no further manual intervention.

50. No?

51. Guess I'll have to tell you the super-secret explanation.

53. For example, at a minimum you'll want to put the MAC Address into the RT system. You may even want to populate DNS from an IP field. Every step of this process uses the info from RT.

54. There may be site-specific stuff you need to use. Don't be afraid to add or use it. This is only a framework.

57. The kickstart file is not a file at all. It is a CGI script. It goes to RT and DNS and gathers all of the information required, makes decisions on how to build the servers, and then custom generates a kickstart file.

58. It should at minimum take one argument – the RT asset ID. This is a unique identifier and allows all the information to be pulled out of the asset tracker to be used in the script.

60. I recommend something like Template::Toolkit to make it more manageable.

61. Build maintainability in from the beginning! You may not get another chance!

63. DO NOT USE EXTERNAL REPOSITORIES.

64. Pull down an internal mirror and use that.

65. The reason for this is: control. If you use external repositories, you are putting control of releases and upgrades into their hands, not yours.

66. And while Centos, Fedora, etc., are fairly good about it, they make mistakes – and you do not want your production site to go down because of someone else's mistake. It's still your fault for not taking my advice. :)

68. I am a control freak.

69. At least when it comes to System Administration.

70. But this is a good thing...

71. Because if you are in complete control of your environment, you reduce the chances of surprises.

72. And surprises are your worst enemy.

73. ... well, maybe not your WORST...

75. A pxelinux.cfg generation script that will point to a custom kickstart script, which pulls the necessary info from RT..

76. ... which is generated by a CGI script, which pulls the necessary info from RT..

77. And a yum repository which has all of the packages you need for a kickstart install.

79. But what about configuration management and application deployment?

80. Oops. Looks like there's more to do.

82. Each server will likely have a base config that every server needs. For example, snmp, ntp, etc., etc.

83. But each server also has an individual role. Application server, database server, facebook browser, Quake Server, pr0n datastore...

84. Can you deploy these roles automatically too?

85. YES!

87. I use puppet.

88. Puppet is a configuration management system

89. It controls what is deployed and what is NOT deployed.

90. It can deploy a package to one or a thousand servers at the same time.

91. And it slices, it dices, it writes bad checks...

93. Facter executes little bits of ruby code in order to determine facts about the system. OS release is one example, etc.

94. But the facts that it determines are not limited to that...

95. The snippets of ruby code can also call AT and pull facts out of AT and make them available to puppet.

98. Put server in DNS, and in AT.

99. Set the fields appropriately. For example, Server Function is DB server, build is Centos 5.4, network role is vmware, mac address is correct.

100. Run a command to add server to dhcp server (pulling info from DNS and AT).

101. Reboot server.

102. Wait for it to build out.

103. Sign puppet certificate

104. Wait for puppet to run

105. Hand off fully built server .

108. You could have AT automatically populate DNS using the SOAP client for nictool...

109. It's not out of the realm of possibility that you could set up a server from bare metal all the way to application deployment simply by setting the fields correctly in AT and then setting a special field using this infrastructure.

110. Questions?