SaltStack Advanced Concepts

www.vishalbiyani.com
SALTSTACK – ADVANCED CONCEPTS

About Me
Vishal Biyani is a DevOps engineer at large. In his past stints he has
done Java development to operations for enterprise systems.
Current interests around Go, Docker, Mesos and plenty of other
stuff. Scribbles at www.vishalbiyani.com. Other interests around
reading, movies and bit of photography. Latest fascination is to
build a DIY Satellite and launch it.

Quick Poll
• How many know anyone of Puppet, Chef, Ansible or Salt
• How many know Salt?
• WHat problems have you faced in Puppet/Chef/Ansible?

Agenda
• Overview of Basics & some Debugging Techniques
• Peer Communication in Salt
• Events, Orchestration & Reactors
• Mine
• Beacons
• Multi-master & Syndic
• Basic Salt Cloud
You can checkout the project at https://github.com/vishal-biyani/saltstack-cluster - this is a
working Salt server+minion setup and has some of examples in today’s talk. Feel free to
file bugs or send pull requests to enhance.
Remind me to show
some exercises when
you see this balloon if I
forget!

OVERVIEW OF BASICS & SOME
DEBUGGING TECHNIQUES

Quick Basics
• Salt is CM platform based on Python language and YAML data format.
Uses ZeroMQ for scalable and fast communication across large number
of nodes
• Typical use cases you see like any other CM platform:
• Managing OS and platform stack on number of machines
• Remote execution of commands (Remove all data in /temp)
• Managing configuration data
• Orchestration to ensure complex use cases for infrastructure/platform
• Server - Agent based (Called minion-master) though you can use it like
Ansible with salt-ssh module (With some limitations)

Command Line - Remote Execution
•What do you want to target – all
web servers for ex.?
Target
• We are using test module and ping
function - but there are many more
Module+Comman
d
• In our case no argument - but
typically there will be some
arguments
ArgumentsHands
on!

State, Pillar, Grains & Formula
• State files is what defines what needs to be done (logic), the extension is
sls
• The configuration data parameters is what is stored in Pillar (data) –
extension is sls. This data is stored on server and provided to target
machines based on role, environment etc.
• There is another type of data item – grains. This is set on individual target machines,
some values are in built by Salt like os, cpu etc. and others can be custom built.
• State files and Pillar+Grain data together can be used to get to desired
state on target machines.
• Reusable modules/playbooks are called Formula
• Of course there are more components – but this is minimal to get going

Once more
• Pillar data is configuration data that is sent by master to
minions.
• Grains data is what minions produce and provide to
master about themselves.

(Some) Development & Debugging Techniques
• When you are developing a piece of logic for certain target (Let’s say for app-server), go
to that machine and run “sudo salt-call state.highstate” – this allowes better debugging.
Yon always add “–l debug” for much more information.
• “state.show_top” can be used to show what does the top state look like for a specific
node instead of guessing.
• In jinja files to inspect variables you can use “ {{ var_name|pprint }}” – based on type and
value of object, you will know what is going wrong. (I am not python guy – is there a
better way?)
• For watching events coming to master you can use command “salt-run state.event
pretty=True” – this will show what data is coming. What happens to event is purely
visible only in debug logs – so enable that in master/minion config.
Are there more techniques I might be
missing?
Hands
on!

PEER COMMUNICATION IN SALT

Peer Communication 1/2
Use case
• A minion wants to know some details or run some command on other minions
• Communicate between minions for some event coordination
• Get information of other minions for making certain decisions
Before
• If you run publish command - it won't work out of the box, Salt by default does
not enable this:

Peer Communication 2/2
After
• You have to enable peer publishing for specific/all modules and minions to
enable peer communication. Setting change in master config file:
• The output after enabling peer communication for previous command:
Security?
• What if any minions runs destructive commands using 'cmd.run' module on other
minions?
Hands
on!

EVENT, REACTOR AND ORCHESTRATION

Event, Reactor and Orchestration
• Use case
• Events can be sent as part of workflow (Custom events), for example let’s say a
new node is up and needs to be added to load balancer config – so you can
send an event. Internally Salt uses events extensively.
• Reactor is what listens to those events and can invoke another workflow (Call a
SLS file or a formula) etc. You can filter for certain types of events, tags etc.
• Salt internal events can also be listened to and workflows initiated when an
event occurs.
• You can do everything in Reactor that you would do in a normal formula/SLS
call – so fairly complex states can be called as an reaction to certain event.
• You can also pass data with events – which can be retrieved in reactor
execution

SEND AND RECEIVE EVENT
• To send an event through an SLS file:
• To listen to an event and to decide what it should do, you need to add reactor.conf in
/etc/salt/master.d (Or inside master config file):
The tag of the event being sent
You can send data along with event and data can be dynamic as seen from some commands!
This config only tells which event to expect and what action to invoke – like a router sort of. Resides in

Processing The Event
The data is within data – hence you see data.data to get data we just passed
We are calling local state - meaning state of that minion. You can call highstate or other thin
Targeting only certain minions based on data received from event!
Passing event data to the SLS so that it can be used there to make more decusions
Hands
on!

Some learnings from reactor
• Debugging reactor is difficult - enabling debug mode on master as well
minion is your best bet
• Be careful of filters you are applying in event sending/reactor
processing and changing state of your machines. Else events will be
filtered and you will wonder what happened?
• Start with a very basic model of working reactor without any conditions
- test it and then grow it.
• Ask on Salt-Users google group - the community is helpful!
• KISS - Keep It Simple Stupid

SALT MINE
Salina Turda, Romania
(https://en.wikipedia.org/wiki/Salina_Turda) is one of most
visited Salt mine with 2M+ visitors so far!
We are going to talk about a virtual version of the same ;)

Salt Mine
• Mine is like grains – but much more real time. Grains is more or less static data about a
minion. Mine can be used for things which are expected to change over time about a
machine/instance and are provided by machine.
• Mine is also a replacement for peer publishing – remember? (Peer publishing is relatively
slow, because it has to reach all nodes and gather data. So in that sense mine is like a
already built data for certain things you need instead of running publish)
Mine functions are run
every X duration and
data is sent to Master
Master stored only
latest data of every
mine execution
A minion which needs data
about other minions can get
using mine.get with
appropriate filters etc.

Salt mine - working
Schedule at which mine functions shoud be run (Goes on minion)
Mine function definition which goes inside Pillar!
Running mine.get will give you info you need!
Hands
on!

Salt Beacon

Salt Beacons
• Salt beacon can “watch” certain things and fire events (What you can watch is limited
things like service, user login, ps etc.).
• These events can be caught by event bus and can be used to invoke reactor actions.
• Some inbuilt actions are already provided - for example clean up the PID file if the
service goes down (In case someone runs kill -9 or service dies for some reason)
• Beacon config can be done in minion config file or via Pillar (We will use pillar).
• We will only watch event - not write reactor for now (Command to watch events: sudo
salt-run state.event pretty=true )
Hands
on!

Salt Syndic & Salt Multimaster

Salt Syndic, Multimaster
• We won’t cover much code in this and next topic, mostly functional capabilities
• Multi master
• One minion can connect to multiple masters and all of them can be ‘hot’
• On all masters, keys for minions must be accepted separately
• If the setting is set to failover - then minions connect to first master that response
in list
• Syndic:
• Salt syndics are basically masters but responsible for only a part of infrastructure. That way
you can set up multiple syndics to split load.
• Syndic eventually point to a master of master and is in sync
• Both of above topologies can be mixed to achieve a highly scalable, highly available
and fault tolerant infrastructure.

Salt Cloud

Salt Cloud
• Salt cloud enables you provisions systems and bring them under Salt management
immediately.
• Roughly on config side:
• Define a provider (LIke Rackspace, AWS etc.)
• Query provider to get options of region, images etc.
• Create profiles with above information
• Use profiles to create instances and query, destroy etc
• Map files are used to create/destroy machines at scale. Map files are basically
multiple machine information combined into a cluster like definition
• You can define grains etc. when creating instances so you can manage them as per
identity.

Questions/Suggestions?
Thank you for your time and attention

SaltStack Advanced Concepts

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to SaltStack Advanced Concepts

Similar to SaltStack Advanced Concepts (20)

More from Vishal Biyani

More from Vishal Biyani (17)

Recently uploaded

Recently uploaded (20)

SaltStack Advanced Concepts