Batou ist ein Open-Source Werkzeug um mit Python das Deployment von komplexen Anwendungen zu beschreiben und zu automatisieren. Das Konzept besteht darin eine passende allgemeine Modellsprache in Form einer API zu definieren. Dabei stehen Wiederverwendung, Toleranz gegenüber unbekannten Zuständen und Ausdrucksstärke im Vordergrund. Der Vortrag zeigt den Werdegang und unsere Motivation zur Entwicklung von batou auf und zeigt die konzeptionellen Stärken anhand praktischer Beispiele. Wir haben in den letzten Jahren eine Reihe großer und kleiner Projekte beim Deployment betreut. Im Rahmen unserer Plattform "gocept.net" bereiten wir unsere Erfahrungen in Dokumentation und offenen Werkzeugen auf und haben daher batou entwickelt. batou ist sowohl in Python geschrieben und nutzt Python um Deployment-Strukturen zu beschreiben. Wir möchten Batou gerne vorstellen um unsere Erfahrungen weiterzugeben und (auch auf den Sprints) anzuregen sich mit dem Werkzeug auseinanderzusetzen, Feedback zu bekommen, und daran weiterzuarbeiten. Siehe auch: http://bitbucket.org/gocept/batou

1. BATOU
Multi-(Component|Environment|Host|Platform) deployment
Christian Theune <ct@gocept.com>
Wednesday, October 31, 12

2. Wednesday, October 31, 12

3. Wednesday, October 31, 12

4. INHALT
• Motivation
• Hello World
• Tiefer in den Kaninchenbau
• Ausblick
Wednesday, October 31, 12

5. WARUM?
Wednesday, October 31, 12

6. Wednesday, October 31, 12

7. Wednesday, October 31, 12

8. Wednesday, October 31, 12

9. Wednesday, October 31, 12

10. Wednesday, October 31, 12

11. PARTY LIKE ITS ...
• 1999: PHP FTP-Uploads
• 2001: Shell-Skripte um Zope-Instanzen zu verwalten
• 2003: Zope 2.4 “mkinstance”
• 2006: zc.buildout
• 2009: Fabric
Wednesday, October 31, 12

12. BATOU
VS
Puppet, Chef Fabric, Salt JuJu
Systemkonfiguration
versus Service Modellierung Laufzeitumgebung
deployment
Wednesday, October 31, 12

13. HELLO WORLD
http://www.flickr.com/photos/oskay/472097903
Wednesday, October 31, 12

14. ./.hg
./buildout.cfg
./bootstrap.py
./components
./components/hello
./components/hello/component.py
./environments
./environments/dev.cfg
./environments/prod.cfg
$ find .
Wednesday, October 31, 12

15. from batou.component import Component
from batou.lib.file import File
class Hello(Component):
def configure(self):
self += File('hello',
content='Hello world')
components/hello/component.py
Wednesday, October 31, 12

16. [hosts]
localhost = hello
environments/dev.cfg
Wednesday, October 31, 12

17. $ bin/batou-local dev localhost
Updating Hello > File(hello) > Presence(hello)
Updating Hello > File(hello) > Content(hello)
Wednesday, October 31, 12

18. ./
./components
./environments
...
./work
./work/hello
./work/hello/hello
$ find .
Wednesday, October 31, 12

19. $ bin/batou-local dev localhost
Wednesday, October 31, 12

20. $ cat "foo" > work/hello/hello
$ bin/batou-local dev localhost
Updating Hello > File(hello) > Content(hello)
Wednesday, October 31, 12

21. [environment]
service_user = test
host_domain = gocept.net
[hosts]
test01 = hello
test02 = hello
environments/prod.cfg
Wednesday, October 31, 12

22. $ bin/batou-remote prod
test02.gocept.net: connecting
test01.gocept.net: connecting
test01.gocept.net: bootstrapping
test02.gocept.net: bootstrapping
OK
OK
Deploying test01.gocept.net/hello
Updating Hello > File(hello) > Presence(hello)
Updating Hello > File(hello) > Content(hello)
OK
Deploying test02.gocept.net/hello
Updating Hello > File(hello) > Presence(hello)
Updating Hello > File(hello) > Content(hello)
OK
Wednesday, October 31, 12

23. $ bin/batou-remote prod
test02.gocept.net: connecting
test01.gocept.net: connecting
test01.gocept.net: bootstrapping
test02.gocept.net: bootstrapping
OK
OK
Deploying test01.gocept.net/hello
OK
Deploying test02.gocept.net/hello
OK
Wednesday, October 31, 12

24. HELLO WORLD
• Komponenten und Environments
• batou-local im Konfiguration eines Hosts lokal zu realisieren
• batou-remote um eine ganze Umgebung zu deployen
• Konvergenz und Idempotenz
Wednesday, October 31, 12

25. IN DEN KANINCHENBAU
• ein Programm
• verwaltet mit Supervisor
Wednesday, October 31, 12

26. DAS PROGRAMM
#!/usr/bin/env python
import time
while True:
print 'Hello, world!'
time.sleep(1)
Wednesday, October 31, 12

27. KOMPONENTE
from batou.component import Component
from batou.lib.file import File
class Hello(Component):
def configure(self):
self += File('hello',
source='hello.py',
mode=0o755)
components/hello/component.py
Wednesday, October 31, 12

28. ENVIRONMENT
[hosts]
localhost = hello
environments/dev.cfg
Wednesday, October 31, 12

29. INSTALLATION
$ bin/batou-local dev localhost
Updating Hello > File(hello) > Presence(hello)
Updating Hello > File(hello) > Content(hello)
Updating Hello > File(hello) > Mode(hello)
Wednesday, October 31, 12

30. LÄUFT?
$ ./work/hello/hello
Hello, world!
Hello, world!
^C
Wednesday, October 31, 12

31. Wednesday, October 31, 12

32. BUILDOUT
[buildout]
parts = supervisor
[supervisor]
recipe = collective.recipe.supervisor
components/supervisor/buildout.cfg
Wednesday, October 31, 12

33. BUILDOUT
from batou.component import Component
from batou.lib.buildout import Buildout
class Supervisor(Component):
def configure(self):
self += Buildout('supervisor',
python='2.7')
components/supervisor/component.py
Wednesday, October 31, 12

34. SERVER-ADRESSE
[buildout]
parts = supervisor
[supervisor]
recipe = collective.recipe.supervisor
port = {{component.address.listen}}
serverurl =
http://{{component.address.listen}}
components/supervisor/buildout.cfg
Wednesday, October 31, 12

35. SERVER-ADRESSE
from batou.component import Component
from batou.lib.buildout import Buildout
from batou.utils import Address
class Supervisor(Component):
port = 9000
def configure(self):
self.address = Address(
self.host.fqdn, self.port)
self += Buildout('supervisor',
python='2.7')
components/supervisor/component.py
Wednesday, October 31, 12

36. NETZWERKADRESSEN
a = Address('localhost:631')
a = Address('localhost', '631')
str(a.connect) == 'localhost:631'
a.connect.host == 'localhost'
a.connect.port == '631'
str(a.listen) == '127.0.0.1:631'
a.listen.host == '127.0.0.1'
a.listen.port == '631'
Wednesday, October 31, 12

37. SERVER-ADRESSE
from batou.component import Component
from batou.lib.buildout import Buildout
from batou.utils import Address
class Supervisor(Component):
port = 9000
def configure(self):
self.address = Address(
self.host.fqdn, self.port)
self += Buildout('supervisor',
python='2.7')
components/supervisor/component.py
Wednesday, October 31, 12

38. SERVER-ADRESSE
[hosts]
localhost = program, supervisor
[component:supervisor]
port = 6000
environments/dev.cfg
Wednesday, October 31, 12

39. WELCHE PROGRAMME
STARTEN?
from batou.component import Component
from batou.lib.file import File
class Hello(Component):
def configure(self):
self += File('hello',
source='hello.py', mode=0o755)
self.provide('program',
'{}/hello'.format(self.workdir))
components/hello/component.py
Wednesday, October 31, 12

40. WELCHE PROGRAMME
STARTEN?
class Supervisor(Component):
def configure(self):
self.programs = self.require(
w
'program', host=self.host)
self += Buildout('supervisor',
python='2.7')
components/supervisor/component.py
Wednesday, October 31, 12

41. WELCHE PROGRAMME
STARTEN?
[buildout]
parts = supervisor
[supervisor]
recipe = collective.recipe.supervisor
port = {{component.address.listen}}
serverurl = ...
programs =
{%- for program in component.programs %}
{{loop.index0}} prog{{loop.index}}
{{program}} true
{% endfor -%}
components/supervisor/buildout.cfg
Wednesday, October 31, 12

42. RESSOURCEN UND
ABHÄNGIGKEITEN
• provide(key, value)
• require(key, host=None) => [x, y, z]
• require_one(key, host=None) => x
Wednesday, October 31, 12

43. RESSOURCEN UND
ABHÄNGIGKEITEN
• provide ohne require
• require ohne provide
• require_one mit != 1 Element
• Flapping durch Sortieren vermeiden.
• Dependencies werden automatisch gefunden
Wednesday, October 31, 12

44. HOOK KOMPONENTEN
from batou.component import Component
from batou.lib.file import File
class Hello(Component):
def configure(self):
self += File('hello',
source='hello.py', mode=0o755)
self += Program('hello', command='hello')
components/hello/component.py
Wednesday, October 31, 12

45. class Program(HookComponent):
namevar = 'name'
key = 'supervisor:Program'
command = None
program_template = '{name} {command} true'
def configure(self):
super(Program, self).configure()
self.command = os.path.normpath(
os.path.join(self.workdir, self.command))
def format(self, supervisor):
return self.program_template.format(
name=self.name, command=self.command)
Wednesday, October 31, 12

46. HOOK COMPONENTS
class Program(HookComponent):
key = 'supervisor:Program' Kann eigentlich nicht
... hier leben - muss
importierbar sein!
class Supervisor(Component):
def configure(self):
self.programs = self.require(
Program.key, host=self.host)
self += Buildout('supervisor',
python='2.7')
components/supervisor/component.py
Wednesday, October 31, 12

47. WELCHE PROGRAMME
STARTEN
[buildout]
parts = supervisor
[supervisor]
recipe = collective.recipe.supervisor
port = {{component.address.listen}}
serverurl = ...
programs =
{%- for program in component.programs %}
{{loop.index0}} {{program.format}}
{% endfor -%}
components/supervisor/buildout.cfg
Wednesday, October 31, 12

48. SUPERVISOR STARTEN
class Supervisor(Component):
def configure(self):
...
def verify(self):
self.assert_file_is_current(
'var/supervisord.pid',
['.batou.buildout.success'])
self.assert_no_subcomponent_changes()
def update(self):
out, err = self.cmd('bin/supervisorctl pid')
try:
int(out)
except ValueError:
self.cmd('bin/supervisord')
else:
self.cmd('bin/supervisorctl reload')
Wednesday, October 31, 12

49. @REBOOT
class Service(Component):
namevar = 'executable' Nur ein "bag".
pidfile = None
class Supervisor(Component):
def configure(self):
...
self += Service('bin/supervisord',
pidfile='var/supervisord.pid')
Wednesday, October 31, 12

50. PLATTFORMEN
@platform('flyingcircus.io', Service)
class UserInit(Component):
def configure(self):
target = '/var/spool/init.d/{}/{}'.format(
self.environment.service_user,
basename(self.parent.executable))
self += File(target,
source='init.sh',
is_template=True,
mode=0o755,
leading=True)
Wednesday, October 31, 12

51. PLATTFORMEN
[environment]
host_domain = flyingcircus.io
platform = flyingcircus.io
$ bin/batou-local --platform=
production test01 # keine Plattform!
Wednesday, October 31, 12

52. REVIEW
• File taucht häufig auf
• Address
• Jinja2-Templates
• Environment overrides
• Ressourcen + HookComponents
• verify/update-Cycle
• Plattform-Spezifika
Wednesday, October 31, 12

53. TIPPS
• Komponenten unabhängig von Host
• Gemeinsamkeiten in Komponenten bündeln
• verify/update so kurz wie nur möglich
• je abstrakter eine Komponente desto unwarscheinlicher trifft
man verify/update
Wednesday, October 31, 12

54. VORDEFINIERTE
KOMPONENTEN
File VirtualEnv Buildout zip
Secrets Supervisor Download CMMI
git svn tar cron
Wednesday, October 31, 12

55. FAZIT
• modell-getrieben aber pragmatisch
• Konvergenz und Idempotenz
• Der Teufel liegt im Detail. Immer wieder.
Wednesday, October 31, 12

56. AUSSICHT
• Dokumentation
• Bessere Fehlermeldungen
• Besserer Status-Output
• Alternativen zu SSH/HG
Wednesday, October 31, 12

57. http://mobro.co/theuni
Wednesday, October 31, 12

58. FRAGEN?
• http://bitbucket.org/gocept/batou
• http://packages.python.org/batou
• Bitte gebt mir Bierdeckelfeedback!
• http://mobro.co/theuni
Wednesday, October 31, 12