1. Isn't it Ironic?
Managing a bare metal cloud
Devananda van der Veen
http://github.com/devananda/talks/isnt­it­ironic.html
twitter: @devananda

2. Who am I
Master Engineer at HP
OpenStack Technical
Committee
OpenStack Ironic PTL

3. What we're going to talk about
Virtualization & OpenStack
Ironic's architecture
Configuration choices you need to
make
Operators <­> Developers
Walk through a deploy
Multitenancy
Some other cool stuff

4. Why is virtualization important?

6. "OpenStack is not a virtualization
layer. It is an abstraction layer."
­ Daniel Sabbah, CTO @ IBM

7. Compute Virtualization Drivers
(KVM, Xen, HyperV ...)
flexibility: dev controls their whole
environment
isolation: guests don't share resources
... but noisy neighbors

8. Compute Containerization Drivers
(lxc, docker ...)
specificity: just your app, nothing more
density: far more instances per server
... but shared kernel, memory, and other
resources

9. Compute Physicalization Drivers
(Ironic)
raw performance
non­virtualizable workloads (eg,
GPUs)
... but no hypervisor

10. Ironic is:
An OpenStack service
A Nova driver
also capable of being used stand­
alone

11. Design Goals
Be resilient to hardware and service failures
Predictable and horizontal scale­out
Common API across hardware vendors
Common API for both virtual and physical
resources

12. Key Components
ironic­api: RESTful API service
ironic­conductor: interacts directly with hardware;
asynchronous handling of both requested and periodic
actions.
deploy ramdisk: temporary agent, booted on machines, to
provide remote access to hardware for provisioning and
management.
Nova driver: interface for Nova; enables OpenStack to
provide common abstraction for virtual and physical
machines.
discoverd ramdisk: optional tool for hardware inventory
management

14. Key Technologies
PXE: pre­boot execution environment, allows host to boot
from network
DHCP: dynamic host configuration protocol, used to locate
the NBP on the network, and provide the host OS with IP
address during init
IPMI: intelligent platform management interface, for remote
control of machine power state, boot device, serial console,
etc.
TFTP: trivial file transfer protocol, copies the NBP over the
network
iSCSI: used to remotely attach to HDD and copy the
machine image

15. But you have options...
! PXE: use virtual media channel instead of PXE. Requires
Swift; available in Kilo; limited to "ilo" drivers.
! DHCP: static IP injection is possible, but not really suitable
for larger or dynamic environments
! IPMI: many other out­of­band power management tools
supported
! TFTP: iPXE / HTTP(S) supported, too
! iSCSI: image can be fetched directly by "agent" drivers

16. ... and options ...
Homogeneous hardware?
Easy!
Heterogeneous hardware?
Use nova­scheduler to match flavors to node.properties
Single tenant / small deployment?
Flat network. Maybe use Ironic stand­alone
Service provider for multiple tenants?
Use keystone for auth, nova for quota management, ...
Basically, use OpenStack
Untrusted tenants?
Network isolation is possible (but not trivial) via Neutron
Secure­erase disks, flash firmware between each use

17. Operators <­> Developers

19. Examples

20. Enroll Hardware
$ironicnode-create-dagent_ipmitool
-iipmi_username=admin-iipmi_password=fake-iipmi_address=10.1.2.3
-pcpus=4-pmemory_mb=8192-plocal_gb=500
-enote='spareserver'-nmytest
+--------------+-------------------------------------------------------------+
|Property |Value |
+--------------+-------------------------------------------------------------+
|chassis_uuid|None |
|driver |agent_ipmitool |
|driver_info |{u'ipmi_address':u'10.1.2.3',u'ipmi_username':u'admin', |
| |u'ipmi_password':u'******'} |
|extra |{u'note':u'spareserver'} |
|properties |{u'memory_mb':u'8192',u'local_gb':u'500',u'cpus':u'4'}|
|uuid |7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f |
|name |mytest |
+--------------+-------------------------------------------------------------+
$ironicport-create-n7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f-a00:11:22:00:11:22
+-----------+--------------------------------------+
|Property |Value |
+-----------+--------------------------------------+
|node_uuid|7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f|
|extra |{} |
|uuid |024e52b2-6ae4-483b-a039-d6afae7f6a22|
|address |00:11:22:00:11:22 |
+-----------+--------------------------------------+

21. Validate provided info
$ironicnode-validate7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f
+------------+--------+--------------------------------------------------------------
|Interface |Result|Reason
+------------+--------+--------------------------------------------------------------
|console |False |Missing'ipmi_terminal_port'parameterinnode'sdriver_info.
|deploy |False |Node7a1ce8d0-9679-4d87-8f54-b11f6e8adb8ffailedtovalidate
deployimageinfo.Someparametersweremissing.Missingare:
['driver_info.deploy_kernel','driver_info.deploy_ramdisk','instance_info.image_source']|
|inspect |None |notsupported
|management|True |
|power |True |
+------------+--------+--------------------------------------------------------------

22. (I forgot a few options)
Oops

23. Add or change options
$ironicnode-updatemytestadd
instance_info/image_source=http://192.168.1.1/myimage.qcow2
instance_info/image_checksum=e1d99d6d0ef2144a8d672b0420c547b5
$ironicnode-updatemytestadd
driver_info/deploy_ramdisk=http://192.168.1.1/deploy.initrd
driver_info/deploy_kernel=http://192.168.1.1/deploy.vmlinuz
$ironicnode-updatemytestreplaceextra/note='database'name=db01.example
+------------------------+-------------------------------------------------
|Property |Value
+------------------------+-------------------------------------------------
|extra |{u'note':u'database'}
|name |db01.example

24. Validate info (again)
$ironicnode-validatedb01.example
+------------+--------+---------------------------------------------------------------+
|Interface |Result|Reason |
+------------+--------+---------------------------------------------------------------+
|console |False |Missing'ipmi_terminal_port'parameterinnode'sdriver_info.|
|deploy |True | |
|inspect |None |notsupported |
|management|True | |
|power |True | |
+------------+--------+---------------------------------------------------------------+

25. Show details
$ironicnode-showdb01.example
+------------------------+------------------------------------------------------------
|Property |Value
+------------------------+------------------------------------------------------------
|target_power_state |None
|last_error |
|maintenance_reason |
|provision_state |available
|console_enabled |False
|target_provision_state|None
|maintenance |False
|power_state |poweroff
|driver |agent_ipmitool
|reservation |None
|instance_uuid |None
|driver_internal_info |{}
|chassis_uuid |

26. Maintenance Mode
$ironicnode-set-maintenance--reason'replacingdisks'db01.exampletrue
$ironicnode-showdb01.example
+------------------------+------------------------------------------------------------
|Property |Value
+------------------------+------------------------------------------------------------
|target_power_state |None
|last_error |
|maintenance_reason |replacingdisks
|provision_state |available
|console_enabled |False
|target_provision_state|None
|maintenance |True
|power_state |poweroff
|instance_uuid |None
|driver_internal_info |{}

27. Power Status Loop
$ironicnode-showmy.broken.node
+-----------------+----------------------------------------------------------------------+
|Property |Value |
+-----------------+----------------------------------------------------------------------+
|last_error |Duringsync_power_state,maxretriesexceededfornode |
| |9729f0b2-b270-4d06-aa87-40f2b2cad6ee,nodestateNonedoesnotmatch|
| |expectedstate'off'.UpdatingDBstateto'None'Switchingnodeto |
| |maintenancemode. |
$cat/var/log/upstart/ironic-conductor.log
2015-03-2404:29:19.34926317WARNINGironic.conductor.manager[-]
Duringsync_power_state,couldnotgetpowerstatefornode9729f0b2-b270-4d06-aa87-40f2b2cad6ee.
Error:IPMIcallfailed:powerstatus.

28. Deployment (via Ironic)
$ironicnode-set-provision-statedb01.exampleactive
Theprovisioningoperationcan'tbeperformedonnode
7a1ce8d0-9679-4d87-8f54-b11f6e8adb8fbecauseit'sinmaintenancemode.
$ironicnode-set-maintenancedb01.examplefalse
$ironicnode-set-provision-statedb01.exampleactive
$
#...timegoeson...

29. Deployment (via Ironic)
$ironicnode-showdb01.example
+------------------------+-------------------------------------
|Property |Value
+------------------------+-------------------------------------
|target_power_state |None
|last_error |
|maintenance_reason |None
|provision_state |active
|console_enabled |False
|target_provision_state|None
|maintenance |False
|power_state |poweron
|instance_uuid |None
|driver_internal_info |{}

30. Deployment (via Nova)
$novaboot–flavorbaremetal-imagemyimage-key-namemy_ssh_key...
$tail-f/var/log/upsart/nova-compute.log
...
2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]Freeram(MB):8192
2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]Freedisk(GB):500
2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]FreeVCPUS:4
...
2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]Freeram(MB):0
2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]Freedisk(GB):0
2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]FreeVCPUS:0

31. Two methods for image deployment
Direct from source || Cache on conductor
agent_ipmitool
agent_pyghmi
agent_ilo
pxe_ipmitool
pxe_ipminative
pxe_seamicro
pxe_iboot
pxe_ilo
pxe_snmp
pxe_drac
pxe_irmc
pxe_amt
iscsi_ilo

32. PXE Deploy Process

33. PXE Deploy Process (cont)

34. Agent Deploy Process

35. Agent Deploy Process (cont)

36. (there's a hash ring)
Fault Tolerance
Each node is mapped across the set of conductors which have
enabled that node's driver, using a consistent hash ring.
Starting or stopping a conductor will initiate a rebalance.
Periodic task ensures consistent local state for mapped nodes.
No impact to running instances, but may disrupt reboots (if
nodes netboot)

37. Example
$ironicdriver-list
+---------------------+----------------+
|Supporteddriver(s)|Activehost(s)|
+---------------------+----------------+
|agent_ilo |cdr-A | <<alltheilonodesgohere
|agent_ipmitool |cdr-A,cdr-B | <<ipminodesaresplit50/50
+---------------------+----------------+

38. What happens when I stop cdr­B?
$ironicdriver-list
+---------------------+----------------+
|Supporteddriver(s)|Activehost(s)|
+---------------------+----------------+
|agent_ilo |cdr-A | <<conductor"A"isnowmanaging
|agent_ipmitool |cdr-A | <<100%ofallmynodes
+---------------------+----------------+
Actually, I was just migrating to cdr­C
$ironicdriver-list
+---------------------+----------------+
|Supporteddriver(s)|Activehost(s)|
+---------------------+----------------+
|agent_ilo |cdr-A,cdr-C | <<bothilo&ipminodesare
|agent_ipmitool |cdr-A,cdr-C | <<sharedevenlyacrossconductors
+---------------------+----------------+

39. Is there a command to view the
mapping?
Nope. It's dynamically generated as the cluster changes.

40. Thanks!