Practical Operation Automation with StackStorm

INTERNET MULTIFEED CO.Copyright ©
Practical Operation Automation with
StackStorm
Shu Sugimoto
Software Development Manager, JPNAP
2018-11-05(Mon)

What you will learn
• Why StackStorm is suitable for automating day to day
operation tasks
• The actual method that helps you implement automation
for your current procedures with StackStorm
• Will not cover
• Southbound implementation to network equipment
• All features of StackStorm
2

Background of “Automation”
• ”Automation” is becoming more and more important
• Business agility
• Time saving
• etc...
• In reality
• “We know that automation is important.”
• “We think now we put more effort into this ever.”
• “But its progress is far less than ideal.”
• Why?
3

Automation is difficult: Why?
• A: Your current operation is NOT computer friendly
• 1. Your procedures are so complicated that you can’t simply
write a shell script that does it
• Which also leads you having many partial scripts,
unmanaged, here and there
• 2. There exists steps that requires human interaction within
your procedure documents like:
• ”Check that the result is sane.”
• “Confirm the output is intended.”
• How can computer tell it’s “sane” or “intended”?
4

Automation is difficult: Why?
• A: Your current operation is NOT computer friendly
• -> “To achieve automation, we first need to rebuild our
whole operation from scratch...”
• => Scope become too huge, impossible to estimate, can’t
set proper goal, brain freeze
• StackStorm might help solving them
5

StackStorm aka st2
• Open source IFTTT-ish middleware/framework
• IF This Then That
6
It’s powerful even “Then That” part alone
https://www.slideshare.net/brocade/eventdriven-automation-devops-way-iot-73581697

How StackStorm fits in
• 1. Powerful Workflow engine
• It’s possible to implement a fairly complex procedure
7

st2 Workflow vs Shell script
8
Shell Script StackStorm Workflow
Image from tweet by StackStorm official Twitter account @Stack_Storm
https://twitter.com/stack_storm/status/684921149898113024

st2 Workflow vs Shell script
9
with-items: branch execution for all items in array
join: wait for all
loop
Super flexible, but easy to code

Workflow components
10
Workflow
Action

Workflow components
11
version: '2.0'
examples.mistral-branching:
description: >
A sample workflow that demonstrates how to use conditions
to determine which path in the workflow to take.
type: direct
input:
- which
tasks:
t1:
action: core.local
input:
cmd: "printf <% $.which %>"
publish:
path: <% task(t1).result.stdout %>
on-success:
- a: <% $.path = 'a' %>
- b: <% $.path = 'b' %>
- c: <% not $.path in list(a, b) %>
a:
action: core.local
input:
cmd: "echo 'Took path A.'"
publish:
stdout: <% task(a).result.stdout %>
b:
action: core.local
input:
cmd: "echo 'Took path B.'"
publish:
stdout: <% task(b).result.stdout %>
c:
action: core.local
input:
Workflow
Action
Action
Action

st2 Workflow
• Consists of Actions
• Defines a flow of your task by connecting Actions
• …in YAML
• Can take inputs (parameters)
• Consumed in workflow
• As an input to child action (mostly)
• Can return an output
• Returns result state
• Success/Failure
• Multiple engines supported
• Mistral v2
12

st2 Action
• Unit in workflow
• The place where actual work is done
• e.g. Creating directories, run `make`, etc
• Can take input/return output
• Returns result
• There are several ways to implement actions
• Write python code -> most popular
• Use built-in runners*
• Super useful built-in runner: `remote-shell-cmd`
13
* Actions are interpreted and run by corresponding runners
e.g. python action -> written in python, run by “python-script” runner

remote-shell-cmd runner
• `remote-shell-cmd`
• Built-in runner
• Takes following parameters as an input
• target hostname
• username
• ssh_key or password
• cwd
• cmd
• Runs cmd in cwd
• on target host as username
• by logging in with ssh
14

Example action backed by remote-shell-cmd
15
---
enabled: true
name: remote1
runner_type: remote-shell-cmd
parameters:
hosts:
default: 192.168.33.10
username:
default: vagrant
password:
default: vagrant
cwd:
default: /vagrant
cmd:
default: |
set -x
pwd
ls -al
df -h
root@9fe86b6dce75:/# st2 run demo.remote1
.
id: 5bdd72e9ecc69005aed541d4
status: succeeded
parameters: None
result:
192.168.33.10:
failed: false
return_code: 0
stderr: '+ pwd
+ ls -al
+ df -h'
stdout: '/vagrant
total 8
drwxr-xr-x 1 vagrant vagrant 128 Nov 3 02:13 .
drwxr-xr-x 23 root root 4096 Nov 1 15:53 ..
drwxr-xr-x 1 vagrant vagrant 128 Nov 2 23:58 .vagrant
-rw-r--r-- 1 vagrant vagrant 165 Nov 3 02:13 Vagrantfile
Filesystem Size Used Avail Use% Mounted on
udev 487M 0 487M 0% /dev
tmpfs 100M 4.4M 96M 5% /run
/dev/mapper/debian--9--vg-root 62G 1.3G 58G 3% /
tmpfs 499M 0 499M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 499M 0 499M 0% /sys/fs/cgroup
/dev/sda1 236M 37M 187M 17% /boot
vagrant 932G 111G 822G 12% /vagrant
tmpfs 100M 0 100M 0% /run/user/1000'
succeeded: true
remote1.yaml (defining custom action)

Example action backed by remote-shell-cmd
16
---
enabled: true
name: remote2
parameters:
hosts:
default: 192.168.33.10
username:
default: vagrant
password:
default: vagrant
cwd:
default: /
cmd:
default: |
set -eux
TMPDIR=$(mktemp -d)
cd $TMPDIR
git clone https://github.com/mtoyoda/sl
cd sl
make
sudo cp sl /usr/local/bin
# cleanup working directory
cd /
rm -Rf $TMPDIR
remote2.yaml
• Written in YAML
• Multiline command accepted
• Shell features accepted
• vars
• comments
• cmd substitution: $()
• etc
• password-less sudo accepted
• pseudo TTY allocation
If you want to run this action for
other host, you can simply do:
$ st2 run demo.remote2 hosts=192.0.2.1
hosts=192.0.2.1,192.0.2.2
It’s even possible to run on
multiple hosts simultaneously
just by:

st2 Workflow features
• Child action can be a workflow
• You can nest workflows in workflows
• No restriction in levels
• Action output can be chained to an input of subsequent
actions
17
A
W
A
A
W
A
A
A
1
2
3
4
5
6
78

Output/Input chaining
18
version: '2.0'
demo.input-output-chaining:
type: direct
tasks:
mktemp:
action: demo.remote-mktemp
publish:
tmpdir: "{{ jsonpath_query(task('mktemp').result, '*.stdout')[0] }}"
on-success:
- build
build:
action: demo.remote-build
input:
cwd: "{{ _.tmpdir }}"
on-success:
- cleanup
cleanup:
action: demo.remote-cleanup
input:
target_path: "{{ _.tmpdir }}"
---
enabled: true
name: remote-mktemp
parameters:
hosts:
default: 192.168.33.10
username:
default: vagrant
password:
default: vagrant
cmd:
default: mktemp -d
---
enabled: true
name: remote-build
parameters:
hosts:
default: 192.168.33.10
username:
default: vagrant
password:
default: vagrant
cmd:
default: |
git clone https://github.com/mtoyoda/sl
cd sl
make
sudo cp sl /usr/local/bin
input-output-chaining.yaml
remote-mktemp.yaml
remote-build.yaml

Other useful features
• Action execution concurrency policy
• You can enforces the number of executions that can run
simultaneously for a specified action
• Either delay/cancel
• Jinja templating in YAML
• Intended for parameter manipulation
• Datastore (st2kv)
• The place that you can store any key-value data
• Encryption support
• Config parameters, transient data that needs to be
shared between workflows
19

• It’s possible to implement a fairly complex procedure
• remote-shell-cmd helps converting existing steps in
procedure document into st2 actions
• Action can encapsulate a set of steps
• e.g.) git clone ~ make ~ make install
• Good isolation makes actions highly reusable
• There are many actions ready for use (Community
packs*)
• https://exchange.stackstorm.org/
• 100+ available
20

• 2. Inquiries feature
• Pause a workflow and wait for human interaction
• “Hey, does this look right?”
• “If so, please return true”
• “if not, please return false”
• Implemented as a built-in action “core.ask”
21

Inquiries
22
Pause here and wait for input
“Would you like to continue? (yes/no)”
Resume the workflow / abort
core.ask
abort!
yes no
Give a response

Inquiries
23
version: '2.0'
demo.inquiry-simple:
type: direct
tasks:
mktemp:
action: demo.remote-mktemp
publish:
tmpdir: "{{ jsonpath_query(task('mktemp').result, '*.stdout')[0] }}"
on-success:
- pause-workflow
pause-workflow:
action: core.ask
on-success:
- build
build:
action: demo.remote-build
input:
cwd: "{{ _.tmpdir }}"
on-success:
- cleanup
cleanup:
action: demo.remote-cleanup
input:
target_path: "{{ _.tmpdir }}"
root@9fe86b6dce75:/# st2 execution get 5bdf1631ecc6900824f95afd
id: 5bdf1631ecc6900824f95afd
action.ref: demo.inquiry-simple
parameters: None
status: paused
result_task: mktemp
result:
192.168.33.10:
failed: false
return_code: 0
stderr: ''
stdout: /tmp/tmp.bFbYga6wDz
succeeded: true
start_timestamp: Sun, 04 Nov 2018 15:54:25 UTC
end_timestamp:
+--------------------------+------------------------+----------------+
| id | status | task |
+--------------------------+------------------------+----------------+
| 5bdf1634ecc6900824f95b00 | succeeded (2s elapsed) | mktemp |
| 5bdf1636ecc6900824f95b02 | pending | pause-workflow |
+--------------------------+------------------------+----------------+
root@9fe86b6dce75:/# st2 inquiry respond 5bdf1636ecc6900824f95b02
continue (boolean): yes
Response accepted for inquiry 5bdf1636ecc6900824f95b02.

Inquiries
24
“What is your favorite editor?”
(vi/vim/emacs/nano)
core.ask
abort!
vi
You can even branch actions based on input value
Oops...
vim emacs nano

• 2. “Inquiries”
• With these features, you can start automating daily
operations without changing any existing processes or
tools
• StackStorm helps you “start small”
25

Our case
• Target: Changing configurations of monitoring servers
(ping/mrtg/etc...) when add/modify/delete-ing IXP
customer
26

300+ lines of diff to check
This example is rather easy
Excerpt of proc doc
300+ lines
“Is intended config added?”

Our case
• Target: Changing configurations of monitoring servers
(ping/mrtg/etc...) when add/modify/delete-ing IXP
customer
• Before
• There is a procedure document for human ops
• Steps summary
• ssh into specific server
• cd to tool dir
• Run `rake`
• Generate configs
• Check diff
• Run `rake deploy`
• Apply configs to servers
28

Workflow strategy
• Replace all steps with custom actions using remote-shell-
cmd runner
• Pause with core.ask when workflow reaches the point that
requires human decision
• Check diff
• (Plus) Send a diff to Slack
• So that operators can check it easily
• Straightforward 
29

New workflow
30
slack
core.ask
deploy
done
abort!
yes no
init
rake
---
name: "server_config_generator_rake"
runner_type: "remote-shell-cmd"
description: "Generate server-config with server-config-generator."
enabled: true
parameters:
scg_env:
type: string
immutable: true
default: "{{ st2kv.system.scg.config.scg_env }}"
env:
type: object
immutable: true
default:
SCG_ENV: "{{ scg_env }}"
cwd:
type: string
default: "{{ st2kv.system.scg.config.scg_directory | trim | d('/usr/local/mfeed/bin/server
cmd:
type: string
immutable: true
default: bash -lc "rake"
hosts:
type: string
immutable: true
default: "{{ st2kv.system.scg.config.scg_hostname }}"
username:
type: string
immutable: true
default: "{{ st2kv.system.scg.config.username | trim | d('mfeed', true) }}"
private_key:
type: string
immutable: true
default: "{{ st2kv.system.scg.config.ssh_key.remote_cmd }}"
sudo:
type: boolean
immutable: true
default: false

New workflow
31
Use `slack.files.upload` action from community
Diff is uploaded as snippet
slack
core.ask
deploy
done
abort!
yes no
init
rake

New workflow
32
“Does this diff look right? (yes/no)”
$ st2 inquiry respond 5bdbe0395c48de01de0f84cd -r
'{"continue": true}'
slack
core.ask
deploy
done
yes no
init
rake
abort!

New workflow
33
slack
core.ask
deploy
done
yes no
init
rake
---
name: "server_config_generator_deploy"
runner_type: "remote-shell-cmd"
description: "Deploy configs to servers"
enabled: true
parameters:
scg_env:
type: string
immutable: true
default: "{{ st2kv.system.scg.config.scg_env }}"
env:
type: object
immutable: true
default:
SCG_ENV: "{{ scg_env }}"
deploy_main:
type: boolean
default: false
description: "Choose a deploy target system. Can choose backup( = false ) or main( = true
cwd:
type: string
default: "{{ st2kv.system.scg.config.scg_directory | trim | d('/usr/local/mfeed/bin/server
cmd:
type: string
immutable: true
default: bash -lc "rake deploy_{% if deploy_main %}main{% else %}backup{% endif %}"
hosts:
type: string
immutable: true
default: "{{ st2kv.system.scg.config.scg_hostname }}"
username:
type: string
immutable: true
default: "{{ st2kv.system.scg.config.username | trim | d('mfeed', true) }}"
private_key:
type: string
immutable: true
default: "{{ st2kv.system.scg.config.ssh_key.remote_cmd }}"
sudo:
type: boolean
immutable: true
default: false
abort!

Findings
• We could implement our workflow in very short time
• Pretty straightforward thanks to `remote-shell-cmd`
and inquiries
• I’m confident that this approach is effective
• Everything is in YAML: Good
• We could apply the exact same methodology for
software development
• git
• Branch > PR > Code review > Merge
• CI/CD
• Staging/Production
• Disposable environment
• Easy to reproduce: just setup everything from git
• no “export/import”
34

Findings
• Development of st2 is active and open
• Fast release cycle: once in 3 months
• They widely accept PR from anyone
• You can find many active members at community Slack
• Direct channel to developers/product manager
• Many contributors who can help you
• Adopting StackStorm will not eliminate the need of
software engineers
• You still need them to achieve sustainable development
35

Conclusion
• With StackStorm, you can “small start” your long journey of
automation
• This can be achieved by its 1. powerful workflow engine,
and 2. inquiries feature
• Once you get there, it will naturally start advancing
• `core.ask` is where you should work on next
36

How to get started
• Building StackStorm environment into your dev machine
• vagrant-st2
• st2-docker
• (oneline installer)
• Tutorials
• Still does not exist a best one...
• https://github.com/StackStorm/st2-
docker/blob/master/docs/tutorial.md
• Official document
• https://docs.stackstorm.com
• For busy people: Skip to ”Actions”, “Workflows”, “Packs”
• Workflow examples
• https://github.com/stackstorm/st2/tree/master/contrib/examples
• Community Slack
• https://stackstorm.com/community-signup
37

StackStorm Tips
• You should use ”orquesta” workflow engine if you start now
• Although all examples in this presentation use mistral
• There are various reasons to this, but the major one is, orquesta is developed
by st2 team by own, mistral not (it’s a part of OpenStack project)
• Can expect much better support and faster bugfix
• Still in beta, but planned to be GA in Nov. 2018
• You should never include any sensitive data like passwords/private_keys in workflows
or actions
• Use st2kv or pack config to split them out
• You should avoid persisting any business data to st2kv
• Keep source of truth in other place
• Keep st2 disposable
• If you require HA deployment, you should check Kubernetes support
38

Practical Operation Automation with StackStorm

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Practical Operation Automation with StackStorm

Similar a Practical Operation Automation with StackStorm (20)

Último

Último (20)

Practical Operation Automation with StackStorm