1. Challenges
and
lessons
learned
with
Openstack
deployments
and
MySQL
Sandro
Mazzio+a
Sr
Director
Product
Management
IaaS

2. Who
I
am
o I
am
not
a
DB
expert
or
a
DBA
o I
work
in
Product
Management
since
15+
years
o I
work
at
eNovance
since
2
months
o I
started
with
Openstack
in
July
2011
o I’ve
been
launching
HP
Cloud
Services
DBaaS
offering

3. San Francisco
Montréal
Paris
Bangalore
eNovance
loca>ons
and
figures
People
Customers
Growth in
Revenue
Founded in

4. Value
Engineering
Software
Software Architecture : from centralized apps to distributed services
Software Engineering : from Waterfall to Agile
Infrastructure
IT Infrastructure : from Mainframe to Cloud
Cloud Infrastructure Engineering : from Ops to DevOps
Journey
to
the
cloud
Client / Server
Vertical Scaling
Hardware is key
Centralized
Services / API
Horizontal Scaling
Software is key
Distributed
Expensive Hw
Vertical Scaling
Monolithic
Centralized
Cheap Hw
Horizontal Scaling
Distributed

5. Market
&
opportunity
Large
Enterprises
Software for Private Cloud:
$ 10 bn by 2017
Telcos
xSP
Software for Public Cloud :
$ 8 bn by 2017
Telecom
Equipment Vendors
NFV : $ 2bn by 2017
Not Scalable
Expensive
Not Cloud
Rich Set of Features
No Versioning
Shared Infrastructure
Not Scalable
Expensive
Not Cloud
Containers
Not Scalable
Not Cloud
Migrate from Physical Appliance
To Network Functions Virtualization
eNovance
Cloud Solu/ons
Seamless Build & Delivery
of OpenStack Cloud
Infrastructures

6. Build a Cloud
Deliver your Private or Public
Cloud on Premise with a real
Cloud Approach :
• No Versioning
• Pay as you Grow
• AWS Features Match
• Enterprise Grade SLA
• 24/7 Support
Use a Cloud
Deliver Cloud-Ready
applications on top of existing
Cloud infrastructures :
• Elastic & Scalable
• Continuous Delivery
• Pay as you Grow
• Enterprise Grade SLA
• 24/7 Support
What
we
do
?

7. Openstack
&
eNovance
q Gold
Member
q
Top
Ten
Contributor
q Share
DNA
Open
Cloud,
Open
Source

8. Openstack
is
not
simple

9. Openstack
is
not
cloud
(Subbu
Allamaraju
eBay,
July
25,
2013)
All
key
requirements
for
producRon:
• By
Enterprise
for
Private
Cloud
• By
Service
Providers
for
Public
Cloud
“infra
onboarding,
boostrapping,
remediaRon,
config
management,
patching,
packaging,
upgrades,
high
availability,
monitoring,
metrics,
user
support,
capacity
forecasRng
and
management,
billing
or
chargeback,
reclamaRon,
security,
firewalls,
DNS,
integraRon
with
other
internal
infrastructure
and
tools,
and
on
and
on
and
on”.

10. Openstack
Upstream
does
not
address
HA
Openstack
HA
Guide
h+p://docs.openstack.org/high-­‐
availability-­‐guide/content/
index.html
ApplicaRon
on
top
Openstack
OS
Infra.
Plane:
Openstack
API
Message
Queues
DB
VirtualizaRon
Layer
…

11. eNovance
Openstack
Reference
Architecture
V1.0
Ø Management
cluster
full
highly
available
Ø OpenStack
API,
schedulers
Ø RabbitMQ,
Database,
Cache
Ø Network
services
Ø Compute
nodes
with
KVM
and
Open-­‐
vSwitch
with
GRE
tunnels
as
overlay
Ø Swia
cluster
Ø Ceph
cluster,
used
as
backend
for
Nova,
Glance
and
Cinder

12. eNovance
Openstack
product
• Contains
all
core
projects
of
Openstack
• Supports
HA
in
a
Single
DC
mode
• Based
on
• eDeploy
(eNovance
Bare
Metal
Deployment
Opensource)
• Puppet
as
Soaware
ConfiguraRon
Management

13. What
about
DB
in
our
product?
DB
Requirements:
FuncRonal
&
Performance
requirements:
• 100
%
upRme
• Performance
DegradaRon
is
ok
Business
Requirements:
Opensource
Galera
cluster
for
MySQL

14. eNovance
Galera
for
MySQL
usage:
results
Experience
from
ProducRon
Environment:
1
cluster
of
3
Management
Nodes
• No
Service
InterrupRon
• Performance
DegradaRon
are
OK
• Galera
performed
correctly

15. On
the
topic
of
Mul>
Data
Center
West 1
East
EMEA
ASIA
Goal
is
to
address:
• Disaster
Recovery
• Local
Affinity

16. Not
all
services
are
ready
for
Mul>-­‐DC
• Openstack
has
a
concept
of
AZ,
Host
AggregaRon
=
good
in
a
single
DC
.
Not
good
If
your
DC
are
separated
by
a
WAN…
• Nova
Cells
is
good
for
distribuRon
(but
nobody
else
is
taken
care)…
• MulRple
Region
–
Keystone
in
every
region
or
one
overlay
Galera
3
Galera
SynchronizaRon
over
WAN
Using
Heat
to
orchestrate
MulRple
Region

17. Keystone
Our experience working at large
customers is that:
• Token storage is a nightmare and
• and do not scale well on large scale
deployment
• Key Value Store (memcache, redis,
…) -­‐> MySQL -­‐> memcache (but we
have a bug)… euh…

18. Everything
is
MySQL,
almost…
VS

19. Ceilometer,
we
use
mongoDB
But
Requires
MongoDB
DBA,
Runbook,
skills,
…
Because
we
need
scalibility
and
because
of
the
nature
of
informaRon
to
store

20. Openstack
is
not
cloud
(Subbu
Allamaraju
eBay,
July
25,
2013)
All
key
requirements
for
producRon:
• By
Enterprise
for
Private
Cloud
• By
Service
Providers
for
Public
Cloud
“infra
onboarding,
boostrapping,
remediaRon,
config
management,
patching,
packaging,
upgrades,
high
availability,
monitoring,
metrics,
user
support,
capacity
forecasRng
and
management,
billing
or
chargeback,
reclamaRon,
security,
firewalls,
DNS,
integraRon
with
other
internal
infrastructure
and
tools,
and
on
and
on
and
on”.

21. Why
we
need
update
–
upgrade
–patching…
?
User
CRM
Billing
Portal
seongs
changes
new
release
bug
fixes
driver
firmware
update
IntegraRon
mess

22. What
is
our
vision
Deliver
Openstack
and
app
on
top
of
it
using
Openstack
process
and
tools
ConRnuous
IntegraRon/Delivery

23. Leverage
the
Openstack
process
How
Openstack
handle
• Contribu/on of 400 developers / month?
• Developers from 250 companies?
• 10+ integrated projects?
and
yet
release
a
new
version
• every
6
months
• with
400
new
features
• without
major
inconsistencies?

24. …
and
tools
Everything in
• Process descrip/on and code
• Infrastructure used to handle the process
• Code to deploy the infrastructure
100
Rmes
be+er
than
documentaRon
of
a
manual
process
that
is
never
up
to
date...
Reproducible
environments
• Dynamic Provision test env when
needed by Jenkins
• Allow dev to generate a test env for
debug at will
• Store validated releases and customer
specific env (in Openstack Swi^)

25. Benefits
Improve
Efficiency
• Nothing is local, everything is shared
• Control access rights
• Contribute model
• No need for a full release team
• Factorize hardware needs
Reproducible
Env
• Dynamically provision test
environments when needed by
Jenkins
• Allow developers to generate a test
environment for debugging at will
• Store validated releases and
customer specific environments (in
Swi^)

26. What
we
are
describing
is
Devops…
• Validate
code
changes
and
mulRple
deployments
environments
at
once
• Accommodate
more
devs
working
on
the
same
projects
• Limit
wait
Rmes
⇒
Industrializes
development
and...

27. Conclusion
• Openstack
is
not
easy.
• It
is
ok
at
small
scale
in
a
single
DC
mode.
• eNovance
• has
a
product
to
address
Single
DC
HA
mode
deployment
• is
applying
Openstack
CD/CI
to
Openstack
deployment
itself
• is
working
on
the
MulR
DC
mode

28. Ques>ons?
Sandro Mazzioaa
sandro.mazzioaa@enovance.com
@smazzioaa