Más contenido relacionado
La actualidad más candente (20)
Similar a Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack (20)
Más de NTT Communications Technology Development (20)
Automated Deployment & Benchmarking with Chef, Cobbler and Rally for OpenStack
- 1. Copyright © NTT Communications Corporation. All rights reserved.
Automated Deployment & Benchmarking
with Chef, Cobbler and Rally for OpenStack
NTT Communications Corporation
Yuki Nishiwaki
Mahito Ogura
Takeaki Matsumoto
- 2. Copyright © NTT Communications Corporation. All rights reserved.
About Us
1
Mahito Ogura m.ogura@ntt.com
Technology Development
DevOps Engineer
R&D for IaaS, NoSQL, DB as a Service,
Hadoop as a service
Contributing to Devstack and Rally
Yuki Nishiwaki yuki.nishiwaki@ntt.com
Technology Development
Software Engineer
R&D for IaaS
Contributing to Devstack and Rally
-‐‑‒ 11 patches committed in Liberty
- 3. Copyright © NTT Communications Corporation. All rights reserved.
Agenda
2
1. Company Overview
2. Automated Deployment & Benchmarking
– Deployment
– Testing
– Benchmarking
– Evaluation
3. Use Cases
4. Conclusion
- 4. Copyright © NTT Communications Corporation. All rights reserved.
About NTT Communications
3
OpenStack in Production
• Cloudn VPC type ClosedNW
• Enterprise Cloud(coming soon)
Contribution to OpenStack Community
• OpenStack Foundation Corporate Sponsor
• Japan OpenStack user group
• Bug Report / Bug Fix / Blueprint
- 5. Copyright © NTT Communications Corporation. All rights reserved.
OpenStack in Production
4
Launched our first OpenStack product in October 2013
- 6. Copyright © NTT Communications Corporation. All rights reserved.
5
OpenStack deployment is not easy thing to do
• So many components/modules have to be synced and
integrated
• Environment needs to be updated for bug-‐‑‒fix and
enhancement
• There is no magic script to make environment up and running
each time we deploy
OpenStack Deployment
- 7. Copyright © NTT Communications Corporation. All rights reserved.
Our Challenge
6
How can we continuously deploy OpenStack
while consistently keeping service performance?
- 8. Copyright © NTT Communications Corporation. All rights reserved.
2. Automated Deployment & Benchmarking
7
- 9. Copyright © NTT Communications Corporation. All rights reserved.
Improvement cycle
8
Functional
Test
Benchmark
Output
Result
Change
structure
Deploy
OpenStack
Evaluate Result
- 10. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark
Our Implementation (1/2)
9
Functional
Test
Output
Result
Change
structure
Automate Automate
Hand work
Deploy
OpenStackChef RallyTempest
Cobbler
Evaluate Result Hand work
Fluentd Grafana Influxdb
Github
Enterprise
Rally
- 11. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark
Our Implementation (2/2)
10
Functional
Test
Output
Result
Change
structure
Automate Automate
Hand work
Deploy
OpenStack
Hand work
Chef RallyTempest
Cobbler
Evaluate Result
Fluentd Grafana Influxdb
Github
Enterprise
Rally
- 12. Copyright © NTT Communications Corporation. All rights reserved.
Deployment
11
Functional
Test
Output
Result
Change
structure
Automate Automate
Hand work
Deploy
OpenStack
Hand work
Evaluate Result
Benchmark
- 13. Copyright © NTT Communications Corporation. All rights reserved.
Automated deployment is divided into four steps
• Orchestration
• Configuration
• Bootstrapping
• Networking
Deployment Steps Deployment
12
- 14. Copyright © NTT Communications Corporation. All rights reserved.
Tools for Automation
Orchestration
Configuration
Bootstrapping
Networking
Setting for clusters
Setting for just one server
Installing OS into physical servers
Setting up network switch
13
Chef
Expect
Cobbler
Deployment
Chef
- 15. Copyright © NTT Communications Corporation. All rights reserved.
Infrastructure as a Code
Environmental information is maintained in text format
• Definition of network
• System architecture of OpenStack
ü Which role does the server have
• Setting, Configuration of Server
ü Chef Repository
Version control is managed by Git
Deployment
14
- 16. Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone
Update Chef
Server
Setting
switch
OS install
Apply Chef
(Configuration
role)
Apply Chef
(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping
Configuration
Orchestration
Deployment
15
- 17. Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone
Update Chef
Server
Setting
switch
OS install
Apply Chef
(Configuration
role)
Apply Chef
(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping
Configuration
Orchestration
Deployment
16
- 18. Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone
Update Chef
Server
Setting
switch
OS install
Apply Chef
(Configuration
role)
Apply Chef
(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping
Configuration
Orchestration
Deployment
17
- 19. Copyright © NTT Communications Corporation. All rights reserved.
Automate Networking
• Auto VLAN setup
• Preconditions
ü Define server port name
ü Set switch port description with naming rule
“<server_̲name>_̲<server_̲port_̲name>”
eth0 eth1 eth2 eth3 eth4 eth5
eth6 eth7
Naming rule for physical port
-‐‑‒ server_̲1:
-‐‑‒ eth0:
-‐‑‒ vlan10:
-‐‑‒ native
-‐‑‒ vlan100
-‐‑‒ eth1:
-‐‑‒ vlan10:
-‐‑‒ native
Config.yaml
When I got servers, I decide it
Networking
Deployment
18
- 20. Copyright © NTT Communications Corporation. All rights reserved.
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Automate Networking flow (1/4)
19
Service
Chef
Server
1
config.yaml (image)
-‐‑‒ server_̲1
-‐‑‒ eth0
-‐‑‒ vlan10
-‐‑‒ native
-‐‑‒ vlan100
-‐‑‒ eth1
-‐‑‒ vlan10
Networking
Deployment
1. Picks up configuration
including VLAN info
Github
Enterprise
- 21. Copyright © NTT Communications Corporation. All rights reserved.
Automate Networking flow (2/4)
20
Service
Chef
Server
Networking
Deployment
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
2. Gets port description
from network switch
2
Github
Enterprise
- 22. Copyright © NTT Communications Corporation. All rights reserved.
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Automate Networking flow (3/4)
21
Service
Chef
Server
Server1 eth0(switch1 port 1/15)
=> VLAN100, 200
Server1 eth1(switch2 port 1/15)
=> VLAN100, 200
Networking
Deployment
3. Generates switch configuration
based on network layout and target
server settings
3
Github
Enterprise
- 23. Copyright © NTT Communications Corporation. All rights reserved.
Automate Networking flow (4/4)
22
Service
Chef
Server
Networking
Deployment
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
4. Executes the commands which
Jenkins generated
4
Github
Enterprise
- 24. Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone
Update Chef
Server
Setting
switch
OS install
Apply Chef
(Configuration
role)
Apply Chef
(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping
Configuration
Orchestration
Deployment
23
- 25. Copyright © NTT Communications Corporation. All rights reserved.
Bootstrapping flow (1/3)
24
Service
Chef
Server
Deployment Bootstrap
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
1. Run “chef-‐‑‒client” via ssh
2. Apply chef
ü Install cobbler and configure
ü Register machines
ü hostname
ü IP address
2
1
Github
Enterprise
- 26. Copyright © NTT Communications Corporation. All rights reserved.
Bootstrapping flow (2/3)
25
Service
Chef
Server
Deployment Bootstrap
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
3. IPMI power reset
4. Start auto install procedure (PXE)
3
4
Github
Enterprise
- 27. Copyright © NTT Communications Corporation. All rights reserved.
Bootstrapping flow (3/3)
26
Service
Chef
Server
Ubuntu
Deployment Bootstrap
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
5. Check ssh connection periodically
5
Github
Enterprise
- 28. Copyright © NTT Communications Corporation. All rights reserved.
Deployment flow
Git clone
Update Chef
Server
Setting
switch
OS install
Apply Chef
(Configuration
role)
Apply Chef
(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping
Configuration
Orchestration
Deployment
27
- 29. Copyright © NTT Communications Corporation. All rights reserved.
Configuration, Orchestration flow(1/3)
28
Internet
Chef
Server
Ubuntu
Deployment
Rally
Tempest
influxdbCobbler
Configura2on
Orchestra2on
1. Git clone the definition file of
the system architecture of
OpenStack
{
“<environment>”: {
“servers”: {
“server1”: “backend”,
“server2”: “controller”
},
“dependencies”: {
“backend”: [],
“controller”: [{“backend”:1}]
}
}
}
definition.json
1
Github
Enterprise
- 30. Copyright © NTT Communications Corporation. All rights reserved.
Configuration, Orchestration flow(2/3)
29
Internet
Chef
Server
Ubuntu
Deployment
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Configura2on
Orchestra2on
3. Apply chef
ü backend configuration role
ü backend orchestration role…
3
2
2. Run “chef-‐‑‒client” via ssh in order of
the definition file
Github
Enterprise
- 31. Copyright © NTT Communications Corporation. All rights reserved.
Configuration, Orchestration flow(3/3)
30
Internet
Chef
Server
OpenStack
Deployment
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Configura2on
Orchestra2on
4
Github
Enterprise
4. Remove orchestration role
ü divided role into configuration & orchestration
ü doesnʼ’t have idempotence.
- 32. Copyright © NTT Communications Corporation. All rights reserved.
Deployment is ended
Git clone
Update Chef
Server
Setting
switch
OS install
Apply Chef
(Configuration
role)
Apply Chef
(Orchestration
role)
※Omit Detail process, later let us explain detail each step
Networking
Bootstrapping
Configuration
Orchestration
Deployment
31
- 33. Copyright © NTT Communications Corporation. All rights reserved.
Review the Automate deployment
32
Pros
• Anyone can create the same environment anytime
Cons
• Any single tool does not satisfy auto deployment
ü We need to use multiple tools in combination
ü We need wide knowledge
Deployment
- 34. Copyright © NTT Communications Corporation. All rights reserved.
Functional Test
33
- 35. Copyright © NTT Communications Corporation. All rights reserved.
Functional Test
34
Functional
Test
※ Omit detail process here
Output
Result
Change
structure
Automate Automate
Hand work
Deploy
OpenStack
Hand work
Evaluate Result
Benchmark
- 36. Copyright © NTT Communications Corporation. All rights reserved.
Functional Test
35
Tool for test
• We use Tempest
What is Tempest?
• Tempest is Official Integration Test Suite
• Tempest can test any OpenStack, all in one or 1000 nodes
Functional Test
- 37. Copyright © NTT Communications Corporation. All rights reserved.
Functional Test
36
Tempest is used for upstream developing
Community Workflow by http://docs.openstack.org/
Functional Test
- 38. Copyright © NTT Communications Corporation. All rights reserved.
What types of test we use
37
We use
• API Tests
ü Unit tests that ensure API works properly
• Scenario Tests
ü Series of actions like actual users do
Functional Test
- 39. Copyright © NTT Communications Corporation. All rights reserved.
How to use
38
We can execute it in only two steps
1. Edit tempest.conf
2. Execute ./run_̲tempest
$ ./run_̲tempest
………
Ran 1124 tests in 1185.267s
FAILED (failures=4)
Functional Test
- 40. Copyright © NTT Communications Corporation. All rights reserved.
Test flow (1/4)
39
Functional Test
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
1. Create a tempest container
from the image
1
Github
Enterprise
- 41. Copyright © NTT Communications Corporation. All rights reserved.
Test flow (2/4)
40
Functional Test
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
2. Run “chef-‐‑‒client” via ssh
3. Apply chef recipe, To set up
tempest.conf
3
2
Github
Enterprise
- 42. Copyright © NTT Communications Corporation. All rights reserved.
Test flow (3/4)
41
Functional Test
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
4. Run tempest against OpenStack
4
Github
Enterprise
- 43. Copyright © NTT Communications Corporation. All rights reserved.
Test flow (4/4)
42
Functional Test
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
5. Get the results and delete the
container which I created right now
5
Github
Enterprise
- 44. Copyright © NTT Communications Corporation. All rights reserved.
Review the Test
43
Pros
• Generate configuration file automatically based on auto deployed
OpenStack
• Execute functional tests simultaneously with different customized
Tempest
Cons
• Doesnʼ’t cover system architecture
ü We proposed 11 patches for fixing tempest
• Doesnʼ’t control test cases based on API level
Functional Test
- 45. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark
44
- 46. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark
45
Functional
Test
※ Omit detail process here
Output
Result
Change
structure
Automate Automate
Hand work
Deploy
OpenStack
Hand work
Evaluate Result
Benchmark
- 47. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark
46
Tool for benchmark
• Rally ( is one of the OpenStack projects )
What is Rally ?
• Released v0.1.1 in 6 October 2015
• Benchmarking tool
ü Not only Benchmark but support functions of deploy and
verification
Benchmark
- 48. Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
47
1. Create deployment
rally@3eba20593250:~∼$ cat config.json
{
"admin": {
"password": “p@ssw@rd",
"tenant_̲name": ”admin",
"username": ”admin"
},
"auth_̲url": "http://lb:5000/v2.0",
"type": "ExistingCloud"
}
rally@3eba20593250:~∼$rally deployment create -‐‑‒-‐‑‒filename config.json -‐‑‒-‐‑‒name existing
Benchmark
- 49. Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
48
2. Check credential
rally@3eba20593250:~∼$ rally deployment check
keystone endpoints are valid and following services are available:
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
| services | type | status |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
| cinderv2 | volumev2 | Available |
| glance | image | Available |
| keystone | identity | Available |
| neutron | network | Available |
| nova | compute | Available |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
Benchmark
- 50. Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
49
3. Create task file
rally@3eba20593250:~∼$ cat benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json
{% set flavor_̲name = flavor_̲name or "m1.tiny" %}
{
"NovaServers.boot_̲and_̲delete_̲server": [
{
"args": {
"flavor": {
"name": "{{flavor_̲name}}"
},
"image": {
"name": "^cirros.*$"
}
}
}
]
}
Task file is expressed in json or yaml and
We can use statement of Jinja2.
Benchmark
- 51. Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
50
4. Start benchmark
rally@3eba20593250:~∼$ rally task start benchmarks/nova/boot-‐‑‒and-‐‑‒delete.json
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
| Response Times (sec) |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
| action | min | median| 90%ile| 95%ile | max | avg | success | count |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
| nova.boot_̲server | 4.614 | 4.768 | 4.972 | 5.417 | 5.861 | 4.852 | 100.0% | 10 |
| nova.delete_̲server | 2.486 | 2.576 | 2.709 | 2.732 | 2.754 | 2.603 | 100.0% | 10 |
| total | 7.148 | 7.369 | 7.658 | 8.018 | 8.377 | 7.455 | 100.0% | 10 |
+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒+
Load duration: 37.9388210773
Full duration: 51.1114509106
…
Benchmark
- 52. Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
51
5. Make result “Human Readable” (Create HTML Report)
rally@3eba20593250:~∼$ rally task report -‐‑‒-‐‑‒tasks 5e8815a9-‐‑‒10eb-‐‑‒4d67-‐‑‒86b4-‐‑‒6027f4adc634
1f71cb68-‐‑‒9c1e-‐‑‒49e3-‐‑‒8af5-‐‑‒6d21dc34c455 -‐‑‒-‐‑‒out result.html -‐‑‒-‐‑‒html
Benchmark
- 53. Copyright © NTT Communications Corporation. All rights reserved.
How to use Rally
52
※1 There is blueprint
https://blueprints.launchpad.net/rally/+spec/cli-‐‑‒plot2html-‐‑‒compare-‐‑‒tasks
We can see each detailed
result of the benchmark.
But we canʼ’t compare each
result in HTML yet※1
Benchmark
- 54. Copyright © NTT Communications Corporation. All rights reserved.
How I use Rally
53
Benchmark
Benchmark
Compare
to the past
(Summary)
Nothing
to do
Check
Detail
No
Detect
unusual
Yes
Rally task report
Rally task start
- 55. Copyright © NTT Communications Corporation. All rights reserved.
How I use Rally
54
Benchmark
Benchmark
Compare
to the past
(Summary)
Nothing
to do
Check
Detail
No
Detect
unusual
Yes
Rally task report
Benchmark
Dashboard
We created it
Rally task start
- 56. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark Dashboard
55
Benchmark Dashboard
• Provides two functions
ü Compare summarized result with past result
ü Provide link to a html report created by “rally task report”
Benchmark
- 57. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark Dashboard
56
Benchmark
- 58. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark flow (1/4)
57
Benchmark
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
1. Create a container from the
image which I prepared
1
Github
Enterprise
- 59. Copyright © NTT Communications Corporation. All rights reserved.
Benchmark flow (2/4)
58
Benchmark
Service
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
2. Register existing OpenStack
3. Run each benchmark
2
3
Github
Enterprise
- 60. Copyright © NTT Communications Corporation. All rights reserved.
Internet
Github
Enterprise
Benchmark flow (3/4)
59
Benchmark
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
4. Upload report.html to Benchmark
dashboard
5. Post summarized results to influxdb
4
5
- 61. Copyright © NTT Communications Corporation. All rights reserved.
Service
Github
Enterprise
Benchmark flow (4/4)
60
Chef
Server
OpenStack
Rally
Tempest
influxdb
Benchmark
Dashboard
Cobbler
Developer
view result
We can view the result at the dashboard
- 62. Copyright © NTT Communications Corporation. All rights reserved.
Review the Benchmark
61
Benchmark
Pros
• We can compare the result with the past
• The process of Benchmark is fully automated
Cons
• We donʼ’t support many benchmark scenarios
• Best practice for benchmark isnʼ’t fixed yet
- 63. Copyright © NTT Communications Corporation. All rights reserved.
Evaluate Result
62
- 64. Copyright © NTT Communications Corporation. All rights reserved.
Deploy and Test and Benchmark cycle
63
Functional
Test
Output
Result
Change
structure
Automate Automate
Hand work
Deploy
OpenStack
Hand work
Evaluate Result
Benchmark
- 65. Copyright © NTT Communications Corporation. All rights reserved.
When benchmark score is bad
64
• Need to find bottlenecks or issues
ü Too many host resources
ü Too many logs
ü Too difficult to find bottlenecks or issues ...
• Develop analysis system
ü To enable data collecting
ü To enable monitoring
ü To enable alert notify
- 66. Copyright © NTT Communications Corporation. All rights reserved.
Architecture to evaluate the result
65
OpenStack logs
Host resource data
Storage
Host resources Log information
and more …
Search log data
notify
store
Get WARNING/ERROR notifications
- 67. Copyright © NTT Communications Corporation. All rights reserved.
Improvement cycle
66
Functional
Test
Output
Result
Change
structure
Automate Automate
Hand work
Deploy
OpenStack
Hand work
Evaluate Result
Benchmark
- 68. Copyright © NTT Communications Corporation. All rights reserved.
Use case
67
- 69. Copyright © NTT Communications Corporation. All rights reserved.
What scenario I use
68
Benchmark Scenario
• VMTask scenario ( which is contained in rally at default )
ü Prepare the task files which are different in concurrency(30, 50)
nova boot ssh login
run
command
via ssh
nova delete
Use case
- 70. Copyright © NTT Communications Corporation. All rights reserved.
Why I use
69
Why I use VMTask scenario
• The scenario is the most common use case
• We need to know which components will be affected by
increasing the number of users
Use case
- 71. Copyright © NTT Communications Corporation. All rights reserved.
What happened
70
What happened when I execute the benchmark
• Actually, at first success rate isnʼ’t 100%
Use case
- 72. Copyright © NTT Communications Corporation. All rights reserved.
Analyze cause of failure
71
Check the detailed result of the benchmark
it Doesnʼ’t have
IP reachability.
Use case
- 73. Copyright © NTT Communications Corporation. All rights reserved.
Analyze cause of failure
72
Check the neutron log file
$ less /var/log/neutron/neutron-‐‑‒server.log
…
TimeoutError: QueuePool limit of size 5 overflow 20 reached,
connection timed out, timeout 10
…
Overflow connection pool.
Need to review the configuration
about database.
Use case
- 74. Copyright © NTT Communications Corporation. All rights reserved.
Change configuration
73
Change [database] in neutron.conf
-‐‑‒ max_̲retries
-‐‑‒ retry_̲interval
-‐‑‒ min_̲pool_̲size
-‐‑‒ max_̲pool_̲size
-‐‑‒ ide_̲timeout
-‐‑‒ max_̲overflow
Use case
- 75. Copyright © NTT Communications Corporation. All rights reserved.
Check again
74
$ git checkout -‐‑‒b change_̲db_̲setting_̲in_̲neutron_̲conf
$ git commit -‐‑‒am “foobar”
$ git push origin change_̲db_̲setting_̲in_̲neutron_̲conf
Request Jenkins
via Slack
Use case
- 76. Copyright © NTT Communications Corporation. All rights reserved.
Check again
75
This change improves
the success rates
Use case
- 77. Copyright © NTT Communications Corporation. All rights reserved.
Conclusion
76
- 78. Copyright © NTT Communications Corporation. All rights reserved.
conclusion
77
• Default configuration doesnʼ’t cover all use cases
• We need to investigate the best configuration
• Best configuration requires us to follow the cycle
ü deployment, functional test, benchmark, evaluation
• Any single tool cannot automate whole process
ü multiple Open Source Software can automate whole process
- 79. Copyright © NTT Communications Corporation. All rights reserved.
License
78
Jenkins image by Jenkins project is licensed under a Creative Commons
Attribution-‐‑‒ShareAlike 3.0 Unported License.
- 80. Copyright © NTT Communications Corporation. All rights reserved.
Thanks for your attention
Enterprise
Cloud
1st Session
Automate deployment &
benchmark for your OpenStack
with Chef, Cobbler and Rally
2nd Session
NFV Service
Federation across
Heterogeneous
Sites
3rd Session
Enhancement on OpenStack
Networking for Carrier Cloud
Platform
4th Session
OpenStack x Enterprise : Formula for Developing OpenStack-‐‑‒Based
Cloud for Enterprise
DevOps
Network
Enterprise
- 81. Copyright © NTT Communications Corporation. All rights reserved.
NTT Com Booth
P9:Hiten Hall
Weʼ’re looking forward to meeting you at
NTT Com Booth!