CERN uses Puppet and OpenStack to manage its large HPC infrastructure consisting of 270 hypervisors and 2900 VMs. It aims to expand this cloud infrastructure to 15,000 hypervisors and 150,000-200,000 VMs by 2015, potentially across multiple sites. Puppet modules define configurations for common use cases like OpenStack hypervisors or Hadoop nodes. Foreman is used for lifecycle management and as an external node classifier to tell Puppet masters a node's configuration. Secrets are secured using techniques like Hiera encryption with GPG.

1. A one stop solution
for Puppet and Openstack
Daniel Lobato Garcia
daniel.lobato.garcia@cern,ch
@eLobatoss

3. What is CERN
Between Geneva and the
Jura mountains,
straddling the Swiss-
French border
Mission: learn what is the
universe made of and
how does it work?
3

4. Fundamental
questions in
physics
Why do particles have mass?
What is 96% of the universe
made of?
Why isn’t there anti-matter in
the universe?
What was the state of matter
after the Big Bang?
4

5. 8/12/2013 Document reference 5

6. 8/12/2013 Document reference 6

7. 8/12/2013 Document reference 7

8. 8/12/2013 Document reference 8

9. Current status
• 270 Openstack hypervisors
• 2900 virtual machines
• 300 users
• 14 Puppet masters
• 6 Foreman backend nodes
• Some production services migrating to our
cloud – early birds
9

10. Goals
• Ramp up to 15K hypervisors – 150-200K
vms in 2015
• Multi-site (Hungary)
10

11. 8/12/2013 Document reference 11

12. 8/12/2013 Document reference 12

13. Why?
• Unnecessary homebrew stack of tools
• Shift to cloud standards with minimal
customizations
• High turnover – can’t teach new tools
13

14. Why?
• Symbiotic relationship with the community
14

15. Openstack?
• Modular IaaS free open source project
• APIs ~compatible with those of Amazon
15

16. Openstack Nova
(compute)
Cloud fabric controller
16

17. Openstack
Keystone (Identity)
RBAC
Integrated with LDAP
Multiple auth* methods
17

18. Openstack Glance
(Images)
Discovery, registration,
delivery of images
18

19. Openstack Horizon
(Dashboard)
19

20. Modules
• Puppet definitions for every use case you
can imagine.
• Dynamic environments
• Hadoop node
• Openstack hypervisor
• … you name it
20

21. 21
Workflow..?

22. Modules and Git
• Manifests and hieradata are version
controlled
22

23. 23
Git workflow
Puppet masters

24. 24
Easy cherry pick

25. 25
Git workflow

26. 26
Git workflow
Jens
‘Puppetfiles’
Separate repositories
Makes environments and
creates them on the masters

27. Foreman
• Lifecycle management tool for VMs and
physical servers
• External Node Classifier – tells the puppet
master what a node should look like
27

28. 28

29. 29

30. Power operations & Foreman
8/12/2013 Document reference 30
Foreman Proxy
Physical
boxIPMI
Physical
box
IPMI
Physical
box
IPMI
VMVMVM
Openstack
Nova API

31. Openstack VM creation
8/12/2013 Document reference 31

32. Openstack VM creation
8/12/2013 Document reference 32

33. Openstack VM creation
8/12/2013 Document reference 33

34. Scalability experiences
• Split up services
• Puppet – critical vs non critical
34
12 backend nodes
Batch
4 backend nodes
Interactive

35. Scalability experiences
• Foreman – split into different services
35
ENC
Reports
processing
UI/API
Load balancer
9443 – UI/API
9444 – Reports
9445 – ENC
…

36. Scalability experiences
• Autoscale via alarms (Heat)
• Define situations (i.e: load threshold..)
• Spin up VMs as needed
36

37. Scalability guidelines
37

38. github.com
/
cernops
38

39. 39

41. Secrets provisioning (naïve)
• Use case: provision a db password
41

42. Secrets provisioning (hiera-gpg)
• Use case: provision a db password
42

43. Secrets provisioning (hack)
• Use case: provision a db password
43

44. Secrets provisioning
•Masters
need not
read secrets
44