More Related Content Similar to Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014) (20) More from VMware Tanzu (20) Autoscaling Distributed System with BOSH (Cloud Foundry Summit 2014)1. © 2014 Nippon Telegraph and Telephone Corporation
AUTOSCALING
CLOUD FOUNDRY WITH BOSH
Yudai Iwasaki
NTT Service Innovation Laboratory Group
CF Summit 2014
2. © 2014 Nippon Telegraph and Telephone Corporation
• Core developer of Cloudn PaaS
• Working on Cloud Foundry since 2012
– Nise BOSH, BOSH CloudStack CPI and BOSH AutoScaler
• Twitter: @i_yudai
3. © 2014 Nippon Telegraph and Telephone Corporation
Agenda
• Problem
– System load is not fixed
• Solution
– BOSH AutoScaler
• Getting Started
4. © 2014 Nippon Telegraph and Telephone Corporation
Problem:
System Load is
Not Fixed
5. © 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks
6. © 2014 Nippon Telegraph and Telephone Corporation
Daily, Weekly, Monthly, and Yearly Peaks
7. © 2014 Nippon Telegraph and Telephone Corporation
Manually Scaling?
Always monitor load?
8. © 2014 Nippon Telegraph and Telephone Corporation
Impossible
No one wants to do such a boring job
9. © 2014 Nippon Telegraph and Telephone Corporation
Solution:
BOSH AutoScaler
10. © 2014 Nippon Telegraph and Telephone Corporation
BOSH AutoScaler
• Extension for BOSH
• Flexible scaling policies in deployment
manifest files
• Special support for Cloud Foundry
11. © 2014 Nippon Telegraph and Telephone Corporation
Architecture
Overview
12. © 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler
(with CF Plugin)
Heartbeat
Heartbeat
(subscribe)
process
(BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy
(PUT /deployments)
matching rules against log
periodically
Logging metrics(@buffers)run()
Agent
Agent
Agent
Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component
CF Component
CF Component
Collector
TSDB
Varz
Load policies
(GET /deployments)
13. © 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler
(with CF Plugin)
Heartbeat
Heartbeat
(subscribe)
process
(BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy
(PUT /deployments)
matching rules against log
periodically
Logging metrics(@buffers)run()
Agent
Agent
Agent
Agent
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
CF
CF Component
CF Component
CF Component
Collector
TSDB
Varz
Load policies
(GET /deployments)
14. © 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler
(with CF Plugin)
Heartbeat
(subscribe)
process
(BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy
(PUT /deployments)
matching rules against
log periodically Logging metrics(@buffers)run()
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
Collector
TSDBLoad policies
(GET /deployments)
15. © 2014 Nippon Telegraph and Telephone Corporation
Director
BOSH Client
BOSH Scaler (Listener)
NATS
BOSH NATS Collector
AutoScaler
(with CF Plugin)
Heartbeat
(subscribe)
process
(BOSH::Monitor::Event::Heartbeat)
Event Processor
process (BOSH::Monitor::Event::Base)
Deploy
(PUT /deployments)
matching rules against
log periodically Logging metrics(@buffers)run()
Agent
CF Varz Collector
process (Scaler::CfVarzMetric)
Collector
TSDBLoad policies
(GET /deployments)
16. © 2014 Nippon Telegraph and Telephone Corporation
Policy Definitions
17. © 2014 Nippon Telegraph and Telephone Corporation
Defining policies
• Adding “scale” section
• Defining conditions for
each job
• Various condition classes
to define flexible policies
-‐-‐-‐
name:
cf
....
scale:
jobs:
-‐
name:
router
cooldown:
300
out:
limit:
10
unit:
2
conditions:
-‐
class:
CpuAverage
larger_than:
80
duration:
300
-‐
class:
MemoryAverage
larger_than:
90
duration:
300
in:
limit:
3
conditions:
-‐
class:
CpuAverage
smaller_than:
10
duration:
300
-‐
class:
MemoryAverage
smaller_than:
20
duration:
300
18. © 2014 Nippon Telegraph and Telephone Corporation
Conditions: BOSH Heartbeat
• CpuAverage
– Average CPU percentage for duration
• MemoryAverage
– Average memory percentage for duration
• LoadAverage1
– Latest Load Average in 1 minute
• LoadAverage5
– Latest Load Average in 5 minutes
• LoadAverage15
– Latest Load Average in 15 minutes
19. © 2014 Nippon Telegraph and Telephone Corporation
Conditions: CF Plugin
• CFVarzAverage
– Average Varz value for duration
DEA
available_memory_ratio
available_disk_ratio
Router
total_routes
latency.1m
HM9000
NumberOfRunningInstances
NumberOfCrashedInstances
etcd
SendingRequestRate
ReceivingRequestRate
Loggregator Server
receivedMessageCount
numberOfWebsocketSinks
Cloud Controller
connection_count
threadqueue.num_waiting
24. © 2014 Nippon Telegraph and Telephone Corporation
Getting Started
It’s easy to plug in
25. © 2014 Nippon Telegraph and Telephone Corporation
https://github.com/nttlabs/bosh-scaler
26. © 2014 Nippon Telegraph and Telephone Corporation
-‐-‐-‐
#
BOSH
NATS
settings
nats:
&nats
uri:
mbus://192.168.50.4:21084
user:
nats
pass:
nats
#
BOSH
Director
REST
API
settings
rest:
&rest
endpoint_uri:
https://192.168.50.4:25555
user:
scaler
password:
scaler
disable_verify_certification:
true
collectors:
-‐
class:
BoshNatsCollector
bosh_nats:
*nats
bosh_rest:
*rest
-‐
class:
CfVarzTsdbCollector
port:
4567
listeners:
-‐
class:
BoshScaler
bosh_rest:
*rest
interval:
60
#
seconds
buffer_size:
1000
ui:
enable:
true
port:
8888
Configuration
27. © 2014 Nippon Telegraph and Telephone Corporation
...
properties:
collector:
use_tsdb:
true
deployment_name:
cf
opentsdb:
#
your
AutoScaler
address
address:
192.168.15.139
port:
4567
Cloud Foundry Manifest
28. © 2014 Nippon Telegraph and Telephone Corporation
Optimize
Running Costs with
BOSH AutoScaler