SlideShare una empresa de Scribd logo
1 de 81
Descargar para leer sin conexión
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Automated  Deployment  &  Benchmarking
with  Chef,  Cobbler  and  Rally  for  OpenStack
NTT  Communications  Corporation
Yuki  Nishiwaki
Mahito  Ogura
Takeaki  Matsumoto
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
About  Us
1
Mahito  Ogura  m.ogura@ntt.com
Technology  Development
DevOps  Engineer
R&D  for  IaaS,  NoSQL,  DB  as  a  Service,
Hadoop  as  a  service
Contributing  to  Devstack  and  Rally
Yuki  Nishiwaki  yuki.nishiwaki@ntt.com
Technology  Development
Software  Engineer
R&D  for  IaaS
Contributing  to  Devstack  and  Rally
-‐‑‒  11  patches  committed  in  Liberty
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Agenda
2
1.  Company  Overview
2.  Automated  Deployment  &  Benchmarking
–  Deployment
–  Testing
–  Benchmarking
–  Evaluation
3.  Use  Cases
4.  Conclusion
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
About  NTT  Communications
3
OpenStack  in  Production
•  Cloudn  VPC  type  ClosedNW
•  Enterprise  Cloud(coming  soon)
Contribution  to  OpenStack  Community
•  OpenStack  Foundation  Corporate  Sponsor
•  Japan  OpenStack  user  group
•  Bug  Report  /  Bug  Fix  /  Blueprint
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
OpenStack  in  Production
4
Launched  our  first  OpenStack  product  in  October  2013
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
5
OpenStack  deployment  is  not  easy  thing  to  do
•  So  many  components/modules  have  to  be  synced  and  
integrated
•  Environment  needs  to  be  updated  for  bug-‐‑‒fix  and  
enhancement
•  There  is  no  magic  script  to  make  environment  up  and  running  
each  time  we  deploy
OpenStack  Deployment
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Our  Challenge
6
How  can  we  continuously  deploy  OpenStack  
while  consistently  keeping  service  performance?
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
2.  Automated  Deployment  &  Benchmarking
7
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Improvement  cycle
8
Functional  
Test
Benchmark
Output  
Result  
Change  
structure
Deploy  
OpenStack
Evaluate  Result  
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Benchmark
Our  Implementation  (1/2)
9
Functional  
Test
Output  
Result  
Change  
structure
Automate Automate
Hand  work
Deploy  
OpenStackChef RallyTempest
Cobbler
Evaluate  Result   Hand  work
Fluentd Grafana Influxdb
Github
Enterprise
Rally
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Benchmark
Our  Implementation  (2/2)
10
Functional  
Test
Output  
Result  
Change  
structure
Automate Automate
Hand  work
Deploy  
OpenStack
Hand  work
Chef RallyTempest
Cobbler
Evaluate  Result  
Fluentd Grafana Influxdb
Github
Enterprise
Rally
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Deployment
11
Functional  
Test
Output  
Result  
Change  
structure
Automate Automate
Hand  work
Deploy  
OpenStack
Hand  work
Evaluate  Result  
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Automated  deployment  is  divided  into  four  steps
•  Orchestration
•  Configuration
•  Bootstrapping
•  Networking
Deployment  Steps Deployment
12
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Tools  for  Automation
Orchestration	
Configuration	
Bootstrapping	
Networking	
Setting  for  clusters
Setting  for  just  one  server	
Installing  OS  into  physical  servers
Setting  up  network  switch  
13
Chef
Expect
Cobbler
Deployment
Chef
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Infrastructure  as  a  Code
Environmental  information  is  maintained  in  text  format
•  Definition  of  network
•  System  architecture  of  OpenStack
ü  Which  role  does  the  server  have
•  Setting,  Configuration  of  Server  
ü  Chef  Repository
Version  control  is  managed  by  Git
Deployment
14
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Deployment  flow
Git  clone
Update  Chef  
Server
Setting
switch
OS  install
Apply  Chef
(Configuration  
role)
Apply  Chef
(Orchestration  
role)
※Omit  Detail  process,  later  let  us  explain  detail  each  step
Networking	
Bootstrapping	
Configuration	
Orchestration	
Deployment
15
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Deployment  flow
Git  clone
Update  Chef  
Server
Setting
switch
OS  install
Apply  Chef
(Configuration  
role)
Apply  Chef
(Orchestration  
role)
※Omit  Detail  process,  later  let  us  explain  detail  each  step
Networking	
Bootstrapping	
Configuration	
Orchestration	
Deployment
16
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Deployment  flow
Git  clone
Update  Chef  
Server
Setting  
switch
OS  install
Apply  Chef
(Configuration  
role)
Apply  Chef
(Orchestration  
role)
※Omit  Detail  process,  later  let  us  explain  detail  each  step
Networking	
Bootstrapping	
Configuration	
Orchestration	
Deployment
17
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Automate  Networking
•  Auto  VLAN  setup
•  Preconditions
ü  Define  server  port  name
ü  Set  switch  port  description  with  naming  rule  
“<server_̲name>_̲<server_̲port_̲name>”
eth0 eth1 eth2 eth3 eth4 eth5
eth6 eth7
Naming  rule  for  physical  port
-‐‑‒  server_̲1:
    -‐‑‒  eth0:
            -‐‑‒  vlan10:
                  -‐‑‒  native
            -‐‑‒  vlan100
    -‐‑‒  eth1:
            -‐‑‒  vlan10:
                  -‐‑‒  native
Config.yaml
When  I  got  servers,  I  decide  it
Networking	
Deployment
18
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Automate  Networking  flow  (1/4)
19
Service
Chef
Server
1
config.yaml  (image)
-‐‑‒  server_̲1
    -‐‑‒  eth0
            -‐‑‒  vlan10
                  -‐‑‒  native
            -‐‑‒  vlan100
-‐‑‒  eth1
            -‐‑‒  vlan10
Networking	
Deployment
1.  Picks  up  configuration  
including  VLAN  info  
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Automate  Networking  flow  (2/4)
20
Service
Chef
Server
Networking	
Deployment
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
2.  Gets  port  description  
from  network  switch  
2
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Automate  Networking  flow  (3/4)
21
Service
Chef
Server
Server1  eth0(switch1  port  1/15)
                =>  VLAN100,  200
Server1  eth1(switch2  port  1/15)
                                        =>  VLAN100,  200
Networking	
Deployment
3.  Generates  switch  configuration  
based  on  network  layout  and  target  
server  settings  
3
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Automate  Networking  flow  (4/4)
22
Service
Chef
Server
Networking	
Deployment
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
4.  Executes  the  commands    which  
Jenkins  generated
4
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Deployment  flow
Git  clone
Update  Chef  
Server
Setting  
switch
OS  install
Apply  Chef
(Configuration  
role)
Apply  Chef
(Orchestration  
role)
※Omit  Detail  process,  later  let  us  explain  detail  each  step
Networking	
Bootstrapping	
Configuration	
Orchestration	
Deployment
23
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Bootstrapping  flow  (1/3)
24
Service
Chef
Server
Deployment Bootstrap	
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
1.  Run  “chef-‐‑‒client”  via  ssh
2.  Apply  chef
ü Install  cobbler  and  configure
ü Register  machines
ü hostname
ü IP  address
2
1
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Bootstrapping  flow  (2/3)
25
Service
Chef
Server
Deployment Bootstrap	
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
3.  IPMI  power  reset
4.  Start  auto  install  procedure  (PXE)
3
4
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Bootstrapping  flow  (3/3)
26
Service
Chef
Server
Ubuntu
Deployment Bootstrap	
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
5.  Check  ssh  connection  periodically
5
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Deployment  flow
Git  clone
Update  Chef  
Server
Setting  
switch
OS  install
Apply  Chef
(Configuration  
role)
Apply  Chef
(Orchestration  
role)
※Omit  Detail  process,  later  let  us  explain  detail  each  step
Networking	
Bootstrapping	
Configuration	
Orchestration	
Deployment
27
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Configuration,  Orchestration  flow(1/3)
28
Internet
Chef
Server
Ubuntu
Deployment
Rally
Tempest
influxdbCobbler
Configura2on	
Orchestra2on	
1.  Git  clone  the  definition  file  of  
the  system  architecture  of  
OpenStack
{
    “<environment>”:  {
          “servers”:  {
                “server1”:  “backend”,
                “server2”:  “controller”
          },
          “dependencies”:  {
                “backend”:  [],
                “controller”:  [{“backend”:1}]
          }    
    }
}
definition.json
1
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Configuration,  Orchestration  flow(2/3)
29
Internet
Chef
Server
Ubuntu
Deployment
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Configura2on	
Orchestra2on	
3.  Apply  chef
ü backend  configuration  role
ü backend  orchestration  role…
3
2
2.  Run  “chef-‐‑‒client”  via  ssh  in  order  of  
the  definition  file  
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Configuration,  Orchestration  flow(3/3)
30
Internet
Chef
Server
OpenStack
Deployment
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Configura2on	
Orchestra2on	
4
Github  
Enterprise
4.  Remove  orchestration  role
ü divided  role  into  configuration  &  orchestration
ü doesnʼ’t  have  idempotence.
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Deployment  is  ended
Git  clone
Update  Chef  
Server
Setting  
switch
OS  install
Apply  Chef
(Configuration  
role)
Apply  Chef
(Orchestration  
role)
※Omit  Detail  process,  later  let  us  explain  detail  each  step
Networking	
Bootstrapping	
Configuration	
Orchestration	
Deployment
31
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Review  the  Automate  deployment
32
Pros
•  Anyone  can  create  the  same  environment  anytime
Cons
•  Any  single  tool  does  not  satisfy  auto  deployment
ü  We  need  to  use  multiple  tools  in  combination
ü  We  need  wide  knowledge
Deployment
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Functional  Test
33
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Functional  Test
34
Functional  
Test
※  Omit  detail  process  here    
Output  
Result  
Change  
structure
Automate Automate
Hand  work
Deploy  
OpenStack
Hand  work
Evaluate  Result  
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Functional  Test
35
    Tool  for  test
•  We  use  Tempest
    What  is  Tempest?
•  Tempest  is  Official  Integration  Test  Suite
•  Tempest  can  test  any  OpenStack,  all  in  one  or  1000  nodes
Functional  Test
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Functional  Test
36
Tempest  is  used  for  upstream  developing
Community  Workflow  by  http://docs.openstack.org/
Functional  Test
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
What  types  of  test  we  use  
37
    We  use
•  API  Tests
ü  Unit  tests  that  ensure  API  works  properly
•  Scenario  Tests
ü  Series  of  actions  like  actual  users  do
Functional  Test
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
How  to  use
38
We  can  execute  it  in  only  two  steps  
1.  Edit  tempest.conf
2.  Execute  ./run_̲tempest
$  ./run_̲tempest
………
Ran  1124  tests  in  1185.267s
FAILED  (failures=4)
Functional  Test
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Test  flow  (1/4)
39
Functional  Test
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
1.  Create  a  tempest  container  
from  the  image
1
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Test  flow  (2/4)
40
Functional  Test
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
2.  Run  “chef-‐‑‒client”  via  ssh
3.  Apply  chef  recipe,  To  set  up  
tempest.conf
3
2
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Test  flow  (3/4)
41
Functional  Test
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
4.  Run  tempest  against  OpenStack
4
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Test  flow  (4/4)
42
Functional  Test
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
5.  Get  the  results  and  delete  the  
container  which  I  created  right  now
5
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Review  the  Test
43
Pros
•  Generate  configuration  file  automatically  based  on  auto  deployed  
OpenStack
•  Execute  functional  tests  simultaneously  with  different  customized  
Tempest
Cons
•  Doesnʼ’t  cover  system  architecture  
ü  We  proposed  11  patches  for  fixing  tempest
•  Doesnʼ’t  control  test  cases  based  on  API  level
Functional  Test
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Benchmark
44
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Benchmark
45
Functional  
Test
※  Omit  detail  process  here    
Output  
Result  
Change  
structure
Automate Automate
Hand  work
Deploy  
OpenStack
Hand  work
Evaluate  Result  
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Benchmark  
46
    Tool  for  benchmark
•  Rally  (  is  one  of  the  OpenStack  projects  )
    What  is  Rally  ?
•  Released  v0.1.1  in  6  October  2015
•  Benchmarking  tool
ü  Not  only  Benchmark  but    support  functions  of  deploy  and  
verification
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
How  to  use  Rally
47
1.  Create  deployment  
rally@3eba20593250:~∼$  cat  config.json
{
        "admin":  {
                "password":  “p@ssw@rd",
                "tenant_̲name":  ”admin",
                "username":  ”admin"
        },
        "auth_̲url":  "http://lb:5000/v2.0",
        "type":  "ExistingCloud"
}
rally@3eba20593250:~∼$rally  deployment  create  -‐‑‒-‐‑‒filename  config.json  -‐‑‒-‐‑‒name    existing
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
How  to  use  Rally
48
2.  Check  credential
rally@3eba20593250:~∼$  rally  deployment  check
keystone  endpoints  are  valid  and  following  services  are  available:
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
|  services  |  type          |  status        |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
|  cinderv2  |  volumev2  |  Available  |
|  glance      |  image        |  Available  |
|  keystone  |  identity  |  Available  |
|  neutron    |  network    |  Available  |
|  nova          |  compute    |  Available  |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
How  to  use  Rally
49
3.  Create  task  file
rally@3eba20593250:~∼$  cat  benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json
{%  set  flavor_̲name  =  flavor_̲name  or  "m1.tiny"  %}
{
    "NovaServers.boot_̲and_̲delete_̲server":  [
            {
                    "args":  {
                            "flavor":  {
                                    "name":  "{{flavor_̲name}}"
                            },
                            "image":  {
                                    "name":  "^cirros.*$"
                            }
                    }
            }
    ]
}
Task  file  is  expressed  in  json  or  yaml  and  
We  can  use  statement  of  Jinja2.
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
How  to  use  Rally
50
4.  Start  benchmark
rally@3eba20593250:~∼$  rally  task  start  benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json            
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
|                                                                    Response  Times  (sec)                                                                                                                        |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
|  action                                      |  min        |  median|  90%ile|  95%ile  |  max      |  avg      |  success  |  count    |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
|  nova.boot_̲server      |  4.614  |  4.768    |  4.972    |  5.417    |  5.861  |  4.852  |  100.0%    |  10    |
|  nova.delete_̲server  |  2.486  |  2.576    |  2.709    |  2.732    |  2.754  |  2.603  |  100.0%    |  10    |
|  total                                            |  7.148  |  7.369    |  7.658    |  8.018    |  8.377  |  7.455  |  100.0%    |  10    |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
Load  duration:  37.9388210773
Full  duration:  51.1114509106
…
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
How  to  use  Rally
51
5.  Make  result  “Human  Readable”  (Create  HTML  Report)
rally@3eba20593250:~∼$  rally  task  report  -‐‑‒-‐‑‒tasks    5e8815a9-‐‑‒10eb-‐‑‒4d67-‐‑‒86b4-‐‑‒6027f4adc634    
1f71cb68-‐‑‒9c1e-‐‑‒49e3-‐‑‒8af5-‐‑‒6d21dc34c455    -‐‑‒-‐‑‒out  result.html  -‐‑‒-‐‑‒html
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
How  to  use  Rally
52
※1  There  is  blueprint
https://blueprints.launchpad.net/rally/+spec/cli-‐‑‒plot2html-‐‑‒compare-‐‑‒tasks
We  can    see  each  detailed  
result  of  the  benchmark.  
But  we  canʼ’t  compare  each  
result  in  HTML  yet※1
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
How  I  use  Rally
53
Benchmark
Benchmark
Compare
to  the  past
(Summary)
Nothing
to  do
Check
Detail
No
Detect    
unusual
Yes
Rally  task  report
Rally  task  start
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
How  I  use  Rally
54
Benchmark
Benchmark
Compare
to  the  past
(Summary)
Nothing
to  do
Check
Detail
No
Detect    
unusual
Yes
Rally  task  report
Benchmark
Dashboard
We  created  it
Rally  task  start
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Benchmark  Dashboard
55
Benchmark  Dashboard
•  Provides  two  functions
ü  Compare  summarized  result  with  past  result
ü  Provide  link  to  a  html  report  created  by  “rally  task  report”
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Benchmark  Dashboard
56
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Benchmark  flow  (1/4)
57
Benchmark
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
1.  Create  a  container  from  the  
image  which  I  prepared  
1
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Benchmark  flow  (2/4)
58
Benchmark
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
2.  Register  existing  OpenStack
3.  Run  each  benchmark
2
3
Github  
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Internet
Github  
Enterprise
Benchmark  flow  (3/4)
59
Benchmark
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
4.  Upload  report.html  to  Benchmark    
dashboard
5.  Post  summarized  results  to  influxdb
4
5
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Service
Github  
Enterprise
Benchmark  flow  (4/4)
60
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Developer
view  result
We  can  view  the  result  at  the  dashboard
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Review  the  Benchmark
61
Benchmark
Pros
•  We  can  compare  the  result  with  the  past
•  The  process  of  Benchmark  is  fully  automated
Cons
•  We  donʼ’t  support  many  benchmark  scenarios
•  Best  practice  for  benchmark  isnʼ’t  fixed  yet
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Evaluate  Result
62
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Deploy  and  Test  and  Benchmark  cycle
63
Functional  
Test
Output  
Result  
Change  
structure
Automate Automate
Hand  work
Deploy  
OpenStack
Hand  work
Evaluate  Result  
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
When  benchmark  score  is  bad
64
•  Need  to  find  bottlenecks  or  issues
ü  Too  many  host  resources
ü  Too  many  logs
ü  Too  difficult  to  find  bottlenecks  or  issues  ...
•  Develop  analysis  system
ü  To  enable  data  collecting
ü  To  enable  monitoring
ü  To  enable  alert  notify
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Architecture  to  evaluate  the  result
65
OpenStack  logs
Host  resource  data
Storage
Host  resources Log  information
and  more  …
Search  log  data
notify
store
Get  WARNING/ERROR  notifications
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Improvement  cycle
66
Functional  
Test
Output  
Result  
Change  
structure
Automate Automate
Hand  work
Deploy  
OpenStack
Hand  work
Evaluate  Result  
Benchmark
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Use  case
67
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
What  scenario  I  use
68
    Benchmark  Scenario
•  VMTask  scenario  (  which  is  contained  in  rally  at  default  )
ü  Prepare  the  task  files  which  are  different  in  concurrency(30,  50)
nova  boot ssh  login
run  
command  
via  ssh
nova  delete
Use  case
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Why  I  use
69
    Why  I  use  VMTask  scenario
•  The  scenario  is  the  most  common  use  case  
•  We  need  to  know  which  components  will  be  affected  by  
increasing  the  number  of  users
Use  case
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
What  happened
70
    What  happened  when  I  execute  the  benchmark
•  Actually,  at  first  success  rate  isnʼ’t  100%
Use  case
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Analyze  cause  of  failure
71
Check  the  detailed  result  of  the  benchmark
it  Doesnʼ’t  have  
IP  reachability.
Use  case
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Analyze  cause  of  failure
72
Check  the  neutron  log  file
$  less  /var/log/neutron/neutron-‐‑‒server.log
…
TimeoutError:  QueuePool  limit  of  size  5  overflow  20  reached,  
connection  timed  out,  timeout  10
…
Overflow  connection  pool.
Need  to  review  the  configuration  
about  database.
Use  case
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Change  configuration
73
Change  [database]  in  neutron.conf
-‐‑‒  max_̲retries
-‐‑‒  retry_̲interval
-‐‑‒  min_̲pool_̲size
-‐‑‒  max_̲pool_̲size
-‐‑‒  ide_̲timeout
-‐‑‒  max_̲overflow
Use  case
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Check  again
74
$  git  checkout  -‐‑‒b  change_̲db_̲setting_̲in_̲neutron_̲conf
$  git  commit  -‐‑‒am  “foobar”
$  git  push  origin  change_̲db_̲setting_̲in_̲neutron_̲conf
Request  Jenkins  
via  Slack
Use  case
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Check  again
75
This  change  improves  
the  success  rates
Use  case
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Conclusion
76
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
conclusion
77
•  Default  configuration  doesnʼ’t  cover  all  use  cases
•  We  need  to  investigate  the  best  configuration
•  Best  configuration  requires  us  to  follow  the  cycle
ü  deployment,  functional  test,  benchmark,  evaluation  
•  Any  single  tool  cannot  automate  whole  process
ü  multiple  Open  Source  Software  can  automate  whole  process
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
License
78
Jenkins  image  by  Jenkins  project  is  licensed  under  a  Creative  Commons  
Attribution-‐‑‒ShareAlike  3.0  Unported  License.
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
Thanks  for  your  attention
Enterprise  
Cloud
1st  Session
Automate  deployment  &  
benchmark  for  your  OpenStack    
with  Chef,  Cobbler  and  Rally
2nd  Session
NFV  Service  
Federation  across  
Heterogeneous  
Sites
3rd  Session
Enhancement  on  OpenStack  
Networking  for  Carrier  Cloud  
Platform
4th  Session
OpenStack  x  Enterprise  :  Formula  for  Developing  OpenStack-‐‑‒Based  
Cloud  for  Enterprise
DevOps
Network
Enterprise
Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved.
NTT  Com  Booth
P9:Hiten  Hall
Weʼ’re  looking  forward  to  meeting  you  at  
NTT  Com  Booth!

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

OpenStack in 10 minutes with Devstack
OpenStack in 10 minutes with DevstackOpenStack in 10 minutes with Devstack
OpenStack in 10 minutes with Devstack
 
Nova: Openstack Compute-as-a-service
Nova: Openstack Compute-as-a-serviceNova: Openstack Compute-as-a-service
Nova: Openstack Compute-as-a-service
 
Introduction to ZooKeeper - TriHUG May 22, 2012
Introduction to ZooKeeper - TriHUG May 22, 2012Introduction to ZooKeeper - TriHUG May 22, 2012
Introduction to ZooKeeper - TriHUG May 22, 2012
 
Approaching hyperconvergedopenstack
Approaching hyperconvergedopenstackApproaching hyperconvergedopenstack
Approaching hyperconvergedopenstack
 
How logging makes a private cloud a better cloud - OpenStack最新情報セミナー(2016年12月)
How logging makes a private cloud a better cloud - OpenStack最新情報セミナー(2016年12月)How logging makes a private cloud a better cloud - OpenStack最新情報セミナー(2016年12月)
How logging makes a private cloud a better cloud - OpenStack最新情報セミナー(2016年12月)
 
How Many Ohs? (An Integration Guide to Apex & Triple-o)
How Many Ohs? (An Integration Guide to Apex & Triple-o)How Many Ohs? (An Integration Guide to Apex & Triple-o)
How Many Ohs? (An Integration Guide to Apex & Triple-o)
 
An Updated Performance Comparison of Virtual Machines and Linux Containers
An Updated Performance Comparison of Virtual Machines and Linux ContainersAn Updated Performance Comparison of Virtual Machines and Linux Containers
An Updated Performance Comparison of Virtual Machines and Linux Containers
 
Trove Updates - Kilo Edition
Trove Updates - Kilo EditionTrove Updates - Kilo Edition
Trove Updates - Kilo Edition
 
SDN Training - Open daylight installation + example with mininet
SDN Training - Open daylight installation + example with mininetSDN Training - Open daylight installation + example with mininet
SDN Training - Open daylight installation + example with mininet
 
Kubernetes Networking
Kubernetes NetworkingKubernetes Networking
Kubernetes Networking
 
[2018.10.19] Andrew Kong - Tunnel without tunnel (Seminar at OpenStack Korea ...
[2018.10.19] Andrew Kong - Tunnel without tunnel (Seminar at OpenStack Korea ...[2018.10.19] Andrew Kong - Tunnel without tunnel (Seminar at OpenStack Korea ...
[2018.10.19] Andrew Kong - Tunnel without tunnel (Seminar at OpenStack Korea ...
 
How to operate containerized OpenStack
How to operate containerized OpenStackHow to operate containerized OpenStack
How to operate containerized OpenStack
 
[NYC Meetup] Docker at Nuxeo
[NYC Meetup] Docker at Nuxeo[NYC Meetup] Docker at Nuxeo
[NYC Meetup] Docker at Nuxeo
 
Commication Framework in OpenStack
Commication Framework in OpenStackCommication Framework in OpenStack
Commication Framework in OpenStack
 
NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...
NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...
NTTドコモ様 導入事例 OpenStack Summit 2015 Tokyo 講演「After One year of OpenStack Cloud...
 
Docker Networking in OpenStack: What you need to know now
Docker Networking in OpenStack: What you need to know nowDocker Networking in OpenStack: What you need to know now
Docker Networking in OpenStack: What you need to know now
 
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
 
Docker and Maestro for fun, development and profit
Docker and Maestro for fun, development and profitDocker and Maestro for fun, development and profit
Docker and Maestro for fun, development and profit
 
OpenShift v3 Internal networking details
OpenShift v3 Internal networking detailsOpenShift v3 Internal networking details
OpenShift v3 Internal networking details
 
Testing Wi-Fi with OSS Tools
Testing Wi-Fi with OSS ToolsTesting Wi-Fi with OSS Tools
Testing Wi-Fi with OSS Tools
 

Similar a Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack

Foreman-and-Puppet-for-Openstack-Audo-Deployment
Foreman-and-Puppet-for-Openstack-Audo-DeploymentForeman-and-Puppet-for-Openstack-Audo-Deployment
Foreman-and-Puppet-for-Openstack-Audo-Deployment
yating yang
 

Similar a Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack (20)

Autoscaling OpenStack Natively with Heat, Ceilometer and LBaaS
Autoscaling OpenStack Natively with Heat, Ceilometer and LBaaSAutoscaling OpenStack Natively with Heat, Ceilometer and LBaaS
Autoscaling OpenStack Natively with Heat, Ceilometer and LBaaS
 
How do I securely deploy Internet websites in PHP on my IBMi?
How do I securely deploy Internet websites in PHP on my IBMi?How do I securely deploy Internet websites in PHP on my IBMi?
How do I securely deploy Internet websites in PHP on my IBMi?
 
Kirin User Story: Migrating Mission Critical Applications to OpenStack Privat...
Kirin User Story: Migrating Mission Critical Applications to OpenStack Privat...Kirin User Story: Migrating Mission Critical Applications to OpenStack Privat...
Kirin User Story: Migrating Mission Critical Applications to OpenStack Privat...
 
Foreman-and-Puppet-for-Openstack-Audo-Deployment
Foreman-and-Puppet-for-Openstack-Audo-DeploymentForeman-and-Puppet-for-Openstack-Audo-Deployment
Foreman-and-Puppet-for-Openstack-Audo-Deployment
 
Distributed application usecase on docker
Distributed application usecase on dockerDistributed application usecase on docker
Distributed application usecase on docker
 
NTTs Journey with Openstack-final
NTTs Journey with Openstack-finalNTTs Journey with Openstack-final
NTTs Journey with Openstack-final
 
Webinar: Agile Network Deployment
Webinar: Agile Network DeploymentWebinar: Agile Network Deployment
Webinar: Agile Network Deployment
 
Neutron CI Run on Docker
Neutron CI Run on DockerNeutron CI Run on Docker
Neutron CI Run on Docker
 
Netsft2017 day in_life_of_nfv
Netsft2017 day in_life_of_nfvNetsft2017 day in_life_of_nfv
Netsft2017 day in_life_of_nfv
 
OpenStackを利用したEnterprise Cloudを支える技術 - OpenStack最新情報セミナー 2016年5月
OpenStackを利用したEnterprise Cloudを支える技術 - OpenStack最新情報セミナー 2016年5月OpenStackを利用したEnterprise Cloudを支える技術 - OpenStack最新情報セミナー 2016年5月
OpenStackを利用したEnterprise Cloudを支える技術 - OpenStack最新情報セミナー 2016年5月
 
The Switch as a Server - PuppetConf 2014
The Switch as a Server - PuppetConf 2014The Switch as a Server - PuppetConf 2014
The Switch as a Server - PuppetConf 2014
 
Surat MuleSoft Meetup#2 - Anypoint Runtime Fabric
Surat MuleSoft Meetup#2 - Anypoint Runtime FabricSurat MuleSoft Meetup#2 - Anypoint Runtime Fabric
Surat MuleSoft Meetup#2 - Anypoint Runtime Fabric
 
OpenStack with OpenDaylight
OpenStack with OpenDaylightOpenStack with OpenDaylight
OpenStack with OpenDaylight
 
Online spanish meetup #2
Online spanish meetup #2Online spanish meetup #2
Online spanish meetup #2
 
MuleSoft Sizing Guidelines - VirtualMuleys
MuleSoft Sizing Guidelines - VirtualMuleysMuleSoft Sizing Guidelines - VirtualMuleys
MuleSoft Sizing Guidelines - VirtualMuleys
 
OpenStack Sydney summit - OpenStack HA and Testing
OpenStack Sydney summit - OpenStack HA and TestingOpenStack Sydney summit - OpenStack HA and Testing
OpenStack Sydney summit - OpenStack HA and Testing
 
Puppet devops wdec
Puppet devops wdecPuppet devops wdec
Puppet devops wdec
 
Creating Real-Time Data Streaming powered by SQL on Kubernetes - Albert Lewan...
Creating Real-Time Data Streaming powered by SQL on Kubernetes - Albert Lewan...Creating Real-Time Data Streaming powered by SQL on Kubernetes - Albert Lewan...
Creating Real-Time Data Streaming powered by SQL on Kubernetes - Albert Lewan...
 
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...
 
Dev stacklabguide
Dev stacklabguideDev stacklabguide
Dev stacklabguide
 

Más de NTT Communications Technology Development

Más de NTT Communications Technology Development (20)

クラウドを最大限活用するinfrastructure as codeを考えよう
クラウドを最大限活用するinfrastructure as codeを考えようクラウドを最大限活用するinfrastructure as codeを考えよう
クラウドを最大限活用するinfrastructure as codeを考えよう
 
【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介
【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介
【たぶん日本初導入!】Azure Stack Hub with GPUの性能と機能紹介
 
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
 
マルチクラウドでContinuous Deliveryを実現するSpinnakerについて
マルチクラウドでContinuous Deliveryを実現するSpinnakerについて マルチクラウドでContinuous Deliveryを実現するSpinnakerについて
マルチクラウドでContinuous Deliveryを実現するSpinnakerについて
 
Argo CDについて
Argo CDについてArgo CDについて
Argo CDについて
 
SpinnakerとKayentaで 高速・安全なデプロイ!
SpinnakerとKayentaで 高速・安全なデプロイ!SpinnakerとKayentaで 高速・安全なデプロイ!
SpinnakerとKayentaで 高速・安全なデプロイ!
 
100Gbps OpenStack For Providing High-Performance NFV
100Gbps OpenStack For Providing High-Performance NFV100Gbps OpenStack For Providing High-Performance NFV
100Gbps OpenStack For Providing High-Performance NFV
 
Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...
Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...
Can we boost more HPC performance? Integrate IBM POWER servers with GPUs to O...
 
AWS re:Invent2017で見た AWSの強さとは
AWS re:Invent2017で見た AWSの強さとは AWS re:Invent2017で見た AWSの強さとは
AWS re:Invent2017で見た AWSの強さとは
 
分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)分散トレーシング技術について(Open tracingやjaeger)
分散トレーシング技術について(Open tracingやjaeger)
 
Mexico ops meetup発表資料 20170905
Mexico ops meetup発表資料 20170905Mexico ops meetup発表資料 20170905
Mexico ops meetup発表資料 20170905
 
NTT Tech Conference #2 - closing -
NTT Tech Conference #2 - closing -NTT Tech Conference #2 - closing -
NTT Tech Conference #2 - closing -
 
イケてない開発チームがイケてる開発を始めようとする軌跡
イケてない開発チームがイケてる開発を始めようとする軌跡イケてない開発チームがイケてる開発を始めようとする軌跡
イケてない開発チームがイケてる開発を始めようとする軌跡
 
GPU Container as a Service を実現するための最新OSS徹底比較
GPU Container as a Service を実現するための最新OSS徹底比較GPU Container as a Service を実現するための最新OSS徹底比較
GPU Container as a Service を実現するための最新OSS徹底比較
 
SpinnakerとOpenStackの構築
SpinnakerとOpenStackの構築SpinnakerとOpenStackの構築
SpinnakerとOpenStackの構築
 
Troveコミュニティ動向
Troveコミュニティ動向Troveコミュニティ動向
Troveコミュニティ動向
 
Web rtc for iot, edge computing use cases
Web rtc for iot, edge computing use casesWeb rtc for iot, edge computing use cases
Web rtc for iot, edge computing use cases
 
OpenStack Ops Mid-Cycle Meetup & Project Team Gathering出張報告
OpenStack Ops Mid-Cycle Meetup & Project Team Gathering出張報告OpenStack Ops Mid-Cycle Meetup & Project Team Gathering出張報告
OpenStack Ops Mid-Cycle Meetup & Project Team Gathering出張報告
 
NTT Tech Conference #1 Opening Keynote
NTT Tech Conference #1 Opening KeynoteNTT Tech Conference #1 Opening Keynote
NTT Tech Conference #1 Opening Keynote
 
NTT Tech Conference #1 Closing Keynote
NTT Tech Conference #1 Closing KeynoteNTT Tech Conference #1 Closing Keynote
NTT Tech Conference #1 Closing Keynote
 

Último

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Earley Information Science
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 

Último (20)

Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Evaluating the top large language models.pdf
Evaluating the top large language models.pdfEvaluating the top large language models.pdf
Evaluating the top large language models.pdf
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 

Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack

  • 1. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Automated  Deployment  &  Benchmarking with  Chef,  Cobbler  and  Rally  for  OpenStack NTT  Communications  Corporation Yuki  Nishiwaki Mahito  Ogura Takeaki  Matsumoto
  • 2. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. About  Us 1 Mahito  Ogura  m.ogura@ntt.com Technology  Development DevOps  Engineer R&D  for  IaaS,  NoSQL,  DB  as  a  Service, Hadoop  as  a  service Contributing  to  Devstack  and  Rally Yuki  Nishiwaki  yuki.nishiwaki@ntt.com Technology  Development Software  Engineer R&D  for  IaaS Contributing  to  Devstack  and  Rally -‐‑‒  11  patches  committed  in  Liberty
  • 3. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Agenda 2 1.  Company  Overview 2.  Automated  Deployment  &  Benchmarking –  Deployment –  Testing –  Benchmarking –  Evaluation 3.  Use  Cases 4.  Conclusion
  • 4. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. About  NTT  Communications 3 OpenStack  in  Production •  Cloudn  VPC  type  ClosedNW •  Enterprise  Cloud(coming  soon) Contribution  to  OpenStack  Community •  OpenStack  Foundation  Corporate  Sponsor •  Japan  OpenStack  user  group •  Bug  Report  /  Bug  Fix  /  Blueprint
  • 5. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. OpenStack  in  Production 4 Launched  our  first  OpenStack  product  in  October  2013
  • 6. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. 5 OpenStack  deployment  is  not  easy  thing  to  do •  So  many  components/modules  have  to  be  synced  and   integrated •  Environment  needs  to  be  updated  for  bug-‐‑‒fix  and   enhancement •  There  is  no  magic  script  to  make  environment  up  and  running   each  time  we  deploy OpenStack  Deployment
  • 7. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Our  Challenge 6 How  can  we  continuously  deploy  OpenStack   while  consistently  keeping  service  performance?
  • 8. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. 2.  Automated  Deployment  &  Benchmarking 7
  • 9. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Improvement  cycle 8 Functional   Test Benchmark Output   Result   Change   structure Deploy   OpenStack Evaluate  Result  
  • 10. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Benchmark Our  Implementation  (1/2) 9 Functional   Test Output   Result   Change   structure Automate Automate Hand  work Deploy   OpenStackChef RallyTempest Cobbler Evaluate  Result   Hand  work Fluentd Grafana Influxdb Github Enterprise Rally
  • 11. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Benchmark Our  Implementation  (2/2) 10 Functional   Test Output   Result   Change   structure Automate Automate Hand  work Deploy   OpenStack Hand  work Chef RallyTempest Cobbler Evaluate  Result   Fluentd Grafana Influxdb Github Enterprise Rally
  • 12. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Deployment 11 Functional   Test Output   Result   Change   structure Automate Automate Hand  work Deploy   OpenStack Hand  work Evaluate  Result   Benchmark
  • 13. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Automated  deployment  is  divided  into  four  steps •  Orchestration •  Configuration •  Bootstrapping •  Networking Deployment  Steps Deployment 12
  • 14. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Tools  for  Automation Orchestration Configuration Bootstrapping Networking Setting  for  clusters Setting  for  just  one  server Installing  OS  into  physical  servers Setting  up  network  switch   13 Chef Expect Cobbler Deployment Chef
  • 15. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Infrastructure  as  a  Code Environmental  information  is  maintained  in  text  format •  Definition  of  network •  System  architecture  of  OpenStack ü  Which  role  does  the  server  have •  Setting,  Configuration  of  Server   ü  Chef  Repository Version  control  is  managed  by  Git Deployment 14
  • 16. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Deployment  flow Git  clone Update  Chef   Server Setting switch OS  install Apply  Chef (Configuration   role) Apply  Chef (Orchestration   role) ※Omit  Detail  process,  later  let  us  explain  detail  each  step Networking Bootstrapping Configuration Orchestration Deployment 15
  • 17. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Deployment  flow Git  clone Update  Chef   Server Setting switch OS  install Apply  Chef (Configuration   role) Apply  Chef (Orchestration   role) ※Omit  Detail  process,  later  let  us  explain  detail  each  step Networking Bootstrapping Configuration Orchestration Deployment 16
  • 18. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Deployment  flow Git  clone Update  Chef   Server Setting   switch OS  install Apply  Chef (Configuration   role) Apply  Chef (Orchestration   role) ※Omit  Detail  process,  later  let  us  explain  detail  each  step Networking Bootstrapping Configuration Orchestration Deployment 17
  • 19. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Automate  Networking •  Auto  VLAN  setup •  Preconditions ü  Define  server  port  name ü  Set  switch  port  description  with  naming  rule   “<server_̲name>_̲<server_̲port_̲name>” eth0 eth1 eth2 eth3 eth4 eth5 eth6 eth7 Naming  rule  for  physical  port -‐‑‒  server_̲1:    -‐‑‒  eth0:            -‐‑‒  vlan10:                  -‐‑‒  native            -‐‑‒  vlan100    -‐‑‒  eth1:            -‐‑‒  vlan10:                  -‐‑‒  native Config.yaml When  I  got  servers,  I  decide  it Networking Deployment 18
  • 20. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Rally Tempest influxdb Benchmark Dashboard Cobbler Automate  Networking  flow  (1/4) 19 Service Chef Server 1 config.yaml  (image) -‐‑‒  server_̲1    -‐‑‒  eth0            -‐‑‒  vlan10                  -‐‑‒  native            -‐‑‒  vlan100 -‐‑‒  eth1            -‐‑‒  vlan10 Networking Deployment 1.  Picks  up  configuration   including  VLAN  info   Github   Enterprise
  • 21. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Automate  Networking  flow  (2/4) 20 Service Chef Server Networking Deployment Rally Tempest influxdb Benchmark Dashboard Cobbler 2.  Gets  port  description   from  network  switch   2 Github   Enterprise
  • 22. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Rally Tempest influxdb Benchmark Dashboard Cobbler Automate  Networking  flow  (3/4) 21 Service Chef Server Server1  eth0(switch1  port  1/15)                 =>  VLAN100,  200 Server1  eth1(switch2  port  1/15)                                        =>  VLAN100,  200 Networking Deployment 3.  Generates  switch  configuration   based  on  network  layout  and  target   server  settings   3 Github   Enterprise
  • 23. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Automate  Networking  flow  (4/4) 22 Service Chef Server Networking Deployment Rally Tempest influxdb Benchmark Dashboard Cobbler 4.  Executes  the  commands    which   Jenkins  generated 4 Github   Enterprise
  • 24. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Deployment  flow Git  clone Update  Chef   Server Setting   switch OS  install Apply  Chef (Configuration   role) Apply  Chef (Orchestration   role) ※Omit  Detail  process,  later  let  us  explain  detail  each  step Networking Bootstrapping Configuration Orchestration Deployment 23
  • 25. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Bootstrapping  flow  (1/3) 24 Service Chef Server Deployment Bootstrap Rally Tempest influxdb Benchmark Dashboard Cobbler 1.  Run  “chef-‐‑‒client”  via  ssh 2.  Apply  chef ü Install  cobbler  and  configure ü Register  machines ü hostname ü IP  address 2 1 Github   Enterprise
  • 26. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Bootstrapping  flow  (2/3) 25 Service Chef Server Deployment Bootstrap Rally Tempest influxdb Benchmark Dashboard Cobbler 3.  IPMI  power  reset 4.  Start  auto  install  procedure  (PXE) 3 4 Github   Enterprise
  • 27. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Bootstrapping  flow  (3/3) 26 Service Chef Server Ubuntu Deployment Bootstrap Rally Tempest influxdb Benchmark Dashboard Cobbler 5.  Check  ssh  connection  periodically 5 Github   Enterprise
  • 28. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Deployment  flow Git  clone Update  Chef   Server Setting   switch OS  install Apply  Chef (Configuration   role) Apply  Chef (Orchestration   role) ※Omit  Detail  process,  later  let  us  explain  detail  each  step Networking Bootstrapping Configuration Orchestration Deployment 27
  • 29. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Configuration,  Orchestration  flow(1/3) 28 Internet Chef Server Ubuntu Deployment Rally Tempest influxdbCobbler Configura2on Orchestra2on 1.  Git  clone  the  definition  file  of   the  system  architecture  of   OpenStack {    “<environment>”:  {          “servers”:  {                “server1”:  “backend”,                “server2”:  “controller”          },          “dependencies”:  {                “backend”:  [],                “controller”:  [{“backend”:1}]          }        } } definition.json 1 Github   Enterprise
  • 30. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Configuration,  Orchestration  flow(2/3) 29 Internet Chef Server Ubuntu Deployment Rally Tempest influxdb Benchmark Dashboard Cobbler Configura2on Orchestra2on 3.  Apply  chef ü backend  configuration  role ü backend  orchestration  role… 3 2 2.  Run  “chef-‐‑‒client”  via  ssh  in  order  of   the  definition  file   Github   Enterprise
  • 31. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Configuration,  Orchestration  flow(3/3) 30 Internet Chef Server OpenStack Deployment Rally Tempest influxdb Benchmark Dashboard Cobbler Configura2on Orchestra2on 4 Github   Enterprise 4.  Remove  orchestration  role ü divided  role  into  configuration  &  orchestration ü doesnʼ’t  have  idempotence.
  • 32. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Deployment  is  ended Git  clone Update  Chef   Server Setting   switch OS  install Apply  Chef (Configuration   role) Apply  Chef (Orchestration   role) ※Omit  Detail  process,  later  let  us  explain  detail  each  step Networking Bootstrapping Configuration Orchestration Deployment 31
  • 33. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Review  the  Automate  deployment 32 Pros •  Anyone  can  create  the  same  environment  anytime Cons •  Any  single  tool  does  not  satisfy  auto  deployment ü  We  need  to  use  multiple  tools  in  combination ü  We  need  wide  knowledge Deployment
  • 34. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Functional  Test 33
  • 35. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Functional  Test 34 Functional   Test ※  Omit  detail  process  here     Output   Result   Change   structure Automate Automate Hand  work Deploy   OpenStack Hand  work Evaluate  Result   Benchmark
  • 36. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Functional  Test 35    Tool  for  test •  We  use  Tempest    What  is  Tempest? •  Tempest  is  Official  Integration  Test  Suite •  Tempest  can  test  any  OpenStack,  all  in  one  or  1000  nodes Functional  Test
  • 37. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Functional  Test 36 Tempest  is  used  for  upstream  developing Community  Workflow  by  http://docs.openstack.org/ Functional  Test
  • 38. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. What  types  of  test  we  use   37    We  use •  API  Tests ü  Unit  tests  that  ensure  API  works  properly •  Scenario  Tests ü  Series  of  actions  like  actual  users  do Functional  Test
  • 39. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. How  to  use 38 We  can  execute  it  in  only  two  steps   1.  Edit  tempest.conf 2.  Execute  ./run_̲tempest $  ./run_̲tempest ……… Ran  1124  tests  in  1185.267s FAILED  (failures=4) Functional  Test
  • 40. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Test  flow  (1/4) 39 Functional  Test Service Chef Server OpenStack Rally Tempest influxdb Benchmark Dashboard Cobbler 1.  Create  a  tempest  container   from  the  image 1 Github   Enterprise
  • 41. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Test  flow  (2/4) 40 Functional  Test Service Chef Server OpenStack Rally Tempest influxdb Benchmark Dashboard Cobbler 2.  Run  “chef-‐‑‒client”  via  ssh 3.  Apply  chef  recipe,  To  set  up   tempest.conf 3 2 Github   Enterprise
  • 42. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Test  flow  (3/4) 41 Functional  Test Service Chef Server OpenStack Rally Tempest influxdb Benchmark Dashboard Cobbler 4.  Run  tempest  against  OpenStack 4 Github   Enterprise
  • 43. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Test  flow  (4/4) 42 Functional  Test Service Chef Server OpenStack Rally Tempest influxdb Benchmark Dashboard Cobbler 5.  Get  the  results  and  delete  the   container  which  I  created  right  now 5 Github   Enterprise
  • 44. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Review  the  Test 43 Pros •  Generate  configuration  file  automatically  based  on  auto  deployed   OpenStack •  Execute  functional  tests  simultaneously  with  different  customized   Tempest Cons •  Doesnʼ’t  cover  system  architecture   ü  We  proposed  11  patches  for  fixing  tempest •  Doesnʼ’t  control  test  cases  based  on  API  level Functional  Test
  • 45. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Benchmark 44
  • 46. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Benchmark 45 Functional   Test ※  Omit  detail  process  here     Output   Result   Change   structure Automate Automate Hand  work Deploy   OpenStack Hand  work Evaluate  Result   Benchmark
  • 47. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Benchmark   46    Tool  for  benchmark •  Rally  (  is  one  of  the  OpenStack  projects  )    What  is  Rally  ? •  Released  v0.1.1  in  6  October  2015 •  Benchmarking  tool ü  Not  only  Benchmark  but    support  functions  of  deploy  and   verification Benchmark
  • 48. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. How  to  use  Rally 47 1.  Create  deployment   rally@3eba20593250:~∼$  cat  config.json {        "admin":  {                "password":  “p@ssw@rd",                "tenant_̲name":  ”admin",                "username":  ”admin"        },        "auth_̲url":  "http://lb:5000/v2.0",        "type":  "ExistingCloud" } rally@3eba20593250:~∼$rally  deployment  create  -‐‑‒-‐‑‒filename  config.json  -‐‑‒-‐‑‒name    existing Benchmark
  • 49. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. How  to  use  Rally 48 2.  Check  credential rally@3eba20593250:~∼$  rally  deployment  check keystone  endpoints  are  valid  and  following  services  are  available: +-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+ |  services  |  type          |  status        | +-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+ |  cinderv2  |  volumev2  |  Available  | |  glance      |  image        |  Available  | |  keystone  |  identity  |  Available  | |  neutron    |  network    |  Available  | |  nova          |  compute    |  Available  | +-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+ Benchmark
  • 50. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. How  to  use  Rally 49 3.  Create  task  file rally@3eba20593250:~∼$  cat  benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json {%  set  flavor_̲name  =  flavor_̲name  or  "m1.tiny"  %} {    "NovaServers.boot_̲and_̲delete_̲server":  [            {                    "args":  {                            "flavor":  {                                    "name":  "{{flavor_̲name}}"                            },                            "image":  {                                    "name":  "^cirros.*$"                            }                    }            }    ] } Task  file  is  expressed  in  json  or  yaml  and   We  can  use  statement  of  Jinja2. Benchmark
  • 51. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. How  to  use  Rally 50 4.  Start  benchmark rally@3eba20593250:~∼$  rally  task  start  benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json             +-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+ |                                                                    Response  Times  (sec)                                                                                                                        | +-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+ |  action                                      |  min        |  median|  90%ile|  95%ile  |  max      |  avg      |  success  |  count    | +-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+ |  nova.boot_̲server      |  4.614  |  4.768    |  4.972    |  5.417    |  5.861  |  4.852  |  100.0%    |  10    | |  nova.delete_̲server  |  2.486  |  2.576    |  2.709    |  2.732    |  2.754  |  2.603  |  100.0%    |  10    | |  total                                            |  7.148  |  7.369    |  7.658    |  8.018    |  8.377  |  7.455  |  100.0%    |  10    | +-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+ Load  duration:  37.9388210773 Full  duration:  51.1114509106 … Benchmark
  • 52. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. How  to  use  Rally 51 5.  Make  result  “Human  Readable”  (Create  HTML  Report) rally@3eba20593250:~∼$  rally  task  report  -‐‑‒-‐‑‒tasks    5e8815a9-‐‑‒10eb-‐‑‒4d67-‐‑‒86b4-‐‑‒6027f4adc634     1f71cb68-‐‑‒9c1e-‐‑‒49e3-‐‑‒8af5-‐‑‒6d21dc34c455    -‐‑‒-‐‑‒out  result.html  -‐‑‒-‐‑‒html Benchmark
  • 53. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. How  to  use  Rally 52 ※1  There  is  blueprint https://blueprints.launchpad.net/rally/+spec/cli-‐‑‒plot2html-‐‑‒compare-‐‑‒tasks We  can    see  each  detailed   result  of  the  benchmark.   But  we  canʼ’t  compare  each   result  in  HTML  yet※1 Benchmark
  • 54. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. How  I  use  Rally 53 Benchmark Benchmark Compare to  the  past (Summary) Nothing to  do Check Detail No Detect     unusual Yes Rally  task  report Rally  task  start
  • 55. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. How  I  use  Rally 54 Benchmark Benchmark Compare to  the  past (Summary) Nothing to  do Check Detail No Detect     unusual Yes Rally  task  report Benchmark Dashboard We  created  it Rally  task  start
  • 56. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Benchmark  Dashboard 55 Benchmark  Dashboard •  Provides  two  functions ü  Compare  summarized  result  with  past  result ü  Provide  link  to  a  html  report  created  by  “rally  task  report” Benchmark
  • 57. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Benchmark  Dashboard 56 Benchmark
  • 58. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Benchmark  flow  (1/4) 57 Benchmark Service Chef Server OpenStack Rally Tempest influxdb Benchmark Dashboard Cobbler 1.  Create  a  container  from  the   image  which  I  prepared   1 Github   Enterprise
  • 59. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Benchmark  flow  (2/4) 58 Benchmark Service Chef Server OpenStack Rally Tempest influxdb Benchmark Dashboard Cobbler 2.  Register  existing  OpenStack 3.  Run  each  benchmark 2 3 Github   Enterprise
  • 60. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Internet Github   Enterprise Benchmark  flow  (3/4) 59 Benchmark Chef Server OpenStack Rally Tempest influxdb Benchmark Dashboard Cobbler 4.  Upload  report.html  to  Benchmark     dashboard 5.  Post  summarized  results  to  influxdb 4 5
  • 61. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Service Github   Enterprise Benchmark  flow  (4/4) 60 Chef Server OpenStack Rally Tempest influxdb Benchmark Dashboard Cobbler Developer view  result We  can  view  the  result  at  the  dashboard
  • 62. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Review  the  Benchmark 61 Benchmark Pros •  We  can  compare  the  result  with  the  past •  The  process  of  Benchmark  is  fully  automated Cons •  We  donʼ’t  support  many  benchmark  scenarios •  Best  practice  for  benchmark  isnʼ’t  fixed  yet
  • 63. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Evaluate  Result 62
  • 64. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Deploy  and  Test  and  Benchmark  cycle 63 Functional   Test Output   Result   Change   structure Automate Automate Hand  work Deploy   OpenStack Hand  work Evaluate  Result   Benchmark
  • 65. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. When  benchmark  score  is  bad 64 •  Need  to  find  bottlenecks  or  issues ü  Too  many  host  resources ü  Too  many  logs ü  Too  difficult  to  find  bottlenecks  or  issues  ... •  Develop  analysis  system ü  To  enable  data  collecting ü  To  enable  monitoring ü  To  enable  alert  notify
  • 66. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Architecture  to  evaluate  the  result 65 OpenStack  logs Host  resource  data Storage Host  resources Log  information and  more  … Search  log  data notify store Get  WARNING/ERROR  notifications
  • 67. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Improvement  cycle 66 Functional   Test Output   Result   Change   structure Automate Automate Hand  work Deploy   OpenStack Hand  work Evaluate  Result   Benchmark
  • 68. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Use  case 67
  • 69. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. What  scenario  I  use 68    Benchmark  Scenario •  VMTask  scenario  (  which  is  contained  in  rally  at  default  ) ü  Prepare  the  task  files  which  are  different  in  concurrency(30,  50) nova  boot ssh  login run   command   via  ssh nova  delete Use  case
  • 70. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Why  I  use 69    Why  I  use  VMTask  scenario •  The  scenario  is  the  most  common  use  case   •  We  need  to  know  which  components  will  be  affected  by   increasing  the  number  of  users Use  case
  • 71. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. What  happened 70    What  happened  when  I  execute  the  benchmark •  Actually,  at  first  success  rate  isnʼ’t  100% Use  case
  • 72. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Analyze  cause  of  failure 71 Check  the  detailed  result  of  the  benchmark it  Doesnʼ’t  have   IP  reachability. Use  case
  • 73. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Analyze  cause  of  failure 72 Check  the  neutron  log  file $  less  /var/log/neutron/neutron-‐‑‒server.log … TimeoutError:  QueuePool  limit  of  size  5  overflow  20  reached,   connection  timed  out,  timeout  10 … Overflow  connection  pool. Need  to  review  the  configuration   about  database. Use  case
  • 74. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Change  configuration 73 Change  [database]  in  neutron.conf -‐‑‒  max_̲retries -‐‑‒  retry_̲interval -‐‑‒  min_̲pool_̲size -‐‑‒  max_̲pool_̲size -‐‑‒  ide_̲timeout -‐‑‒  max_̲overflow Use  case
  • 75. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Check  again 74 $  git  checkout  -‐‑‒b  change_̲db_̲setting_̲in_̲neutron_̲conf $  git  commit  -‐‑‒am  “foobar” $  git  push  origin  change_̲db_̲setting_̲in_̲neutron_̲conf Request  Jenkins   via  Slack Use  case
  • 76. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Check  again 75 This  change  improves   the  success  rates Use  case
  • 77. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Conclusion 76
  • 78. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. conclusion 77 •  Default  configuration  doesnʼ’t  cover  all  use  cases •  We  need  to  investigate  the  best  configuration •  Best  configuration  requires  us  to  follow  the  cycle ü  deployment,  functional  test,  benchmark,  evaluation   •  Any  single  tool  cannot  automate  whole  process ü  multiple  Open  Source  Software  can  automate  whole  process
  • 79. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. License 78 Jenkins  image  by  Jenkins  project  is  licensed  under  a  Creative  Commons   Attribution-‐‑‒ShareAlike  3.0  Unported  License.
  • 80. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. Thanks  for  your  attention Enterprise   Cloud 1st  Session Automate  deployment  &   benchmark  for  your  OpenStack     with  Chef,  Cobbler  and  Rally 2nd  Session NFV  Service   Federation  across   Heterogeneous   Sites 3rd  Session Enhancement  on  OpenStack   Networking  for  Carrier  Cloud   Platform 4th  Session OpenStack  x  Enterprise  :  Formula  for  Developing  OpenStack-‐‑‒Based   Cloud  for  Enterprise DevOps Network Enterprise
  • 81. Copyright  ©  NTT  Communications  Corporation.  All  rights  reserved. NTT  Com  Booth P9:Hiten  Hall Weʼ’re  looking  forward  to  meeting  you  at   NTT  Com  Booth!