LinkedIn emplea cookies para mejorar la funcionalidad y el rendimiento de nuestro sitio web, así como para ofrecer publicidad relevante. Si continúas navegando por ese sitio web, aceptas el uso de cookies. Consulta nuestras Condiciones de uso y nuestra Política de privacidad para más información.
LinkedIn emplea cookies para mejorar la funcionalidad y el rendimiento de nuestro sitio web, así como para ofrecer publicidad relevante. Si continúas navegando por ese sitio web, aceptas el uso de cookies. Consulta nuestra Política de privacidad y nuestras Condiciones de uso para más información.
Compute node HA
a.k.a. “can pets survive in OpenStack?”
Senior Software Engineer, Cloud & High Availability
OpenStack London Meetup, Wednesday 18th
– short update on upstream development
High Availability in
a typical OpenStack cloud today
Typical HA control plane in OpenStack
Control Node 1 Node
Node 1 Node 2
DRBD or shared storage
Node 1 Node 2 Node 3
• Maximises cloud uptime
• Automatic restart of
• Active/Active API services
with load balancing
• DB + MQ either
Under the covers
Node 1 Node 2 Node 3
• Recommended by
official HA guide
• HAProxy distributes service
‒ monitoring and control of nodes
‒ cluster membership /
messaging / quorum /
But what I really want to do is keep my workloads up!
HA only on control plane
Can we simply extend the cluster?
• Corosync requires <= 32 nodes
• But we want lots of compute nodes!
• The obvious workarounds are ugly
‒ Multiple compute clusters
‒ introduces unwanted artificial boundaries
‒ Clusters inside / between guest VM instances
‒ requires cloud users to modify guest images (installing & configuring cluster
‒ cluster stacks are not OS-agnostic
‒ cloud is supposed to make things easier not harder!
pacemaker_remote to the rescue!
• New(-ish) Pacemaker feature
• Allows arbitrary scalability of an existing
• Increases availability of compute nodes
‒ Detects failed compute services
‒ Automatic recovery of compute services where possible
• “Quarantines” failing compute nodes
‒ STONITH (fencing) extends to remote nodes
• Coordinates with control plane
‒ VMs on dead compute nodes are resurrected elsewhere
‒ In nova, this is described as “evacuation”
Public Health Warning
nova evacuate does not really mean evacuation!
Think about earthquakes
Not too late
Too late to
Public Health Warning
• nova evacuate does not do evacuation
• nova evacuate does resurrection
• In Vancouver, nova developers considered a rename
‒ Hasn't happened yet
‒ Due to impact, seems unlikely to happen any time soon
‒ Whenever you see “evacuate” in a nova-related context,
pretend you saw “resurrect”
• NovaCompute / NovaEvacuate custom OCF RAs
‒ used by Red Hat / SUSE / Intel
‒ works with known limitations
‒ PoC to address above limitations
‒ decouples resurrection workflow from Pacemaker
• Masakari (NTT)
‒ similar architecture, different code
‒ monitoring at 3 layers (node, process, hypervisor)
• Approach of AWcloud / ChinaMobile
‒ very different; uses consul / raft / gossip
• Use Mistral to orchestrate resurrection workflow
• Intel currently working on prototype
• Possibly the most promising approach
‒ Mistral considered pretty solid
‒ This is exactly the kind of thing it was designed for
• However, Mistral currently a SPoF … oops
‒ Don't worry, should be fixed in mitaka cycle
• Feasibility of convergence with Masakari will probably
be analysed within next week or two
• openstack-resource-agents project now on
‒ maintained by me
• New #openstack-ha IRC channel on FreeNode
‒ automatic notifications for activity on HA repositories
• New topic category on openstack-dev@ mailing list
Subject: [HA] i can haz pets in my cloud?
• Weekly IRC meetings at Monday 9am UTC
• HA guide currently undergoing a revamp
• Everyone welcome to get involved!
Unpublished Work of SUSE LLC. All Rights Reserved.
This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC.
Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of
their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated,
abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE.
Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
This document is not to be construed as a promise by any participating company to develop, deliver, or market a
product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making
purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document,
and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The
development, release, and timing of features or functionality described for SUSE products remains at the sole
discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at
any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in
this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All
third-party trademarks are the property of their respective owners.