Using SaltStack to automate enterprise IT operations and support capabilities is not as well documented as the more traditional SaltStack use cases. This session will show how the HP Cloud team runs a secure and reliable SaltStack automation environment by writing Salt states and modules to simplify day-to-day operations and support while extending SaltStack capabilities through dynamic states and modules. The talk will also show how to protect sensitive information and safe guard against user errors.

1. Automating Operations and Support with
Salt Stack
Saurabh Surana

2. Agenda
• OpenStack Trove – Quick Intro
• Salt Environment Overview
• Automation Overview
• What changes for operations with salt stack?
• Salt Modules
• Salt States
• Challenges we are facing

3. OpenStack - Trove
• OpenStack’s Database as a Service Project
• REST API for creating managed database instances
• Provides mysql support out of the box
• Actively being worked on by HP, Rackspace, Mirantis….

4. OpenStack – Trove
TROVE
API
TROVE
TASKMANAGER
TROVE
CONDUCTOR
DBRabbitMQ
MySQL
Instance
MySQL
Instance
OpenStackNova
User1User2

5. Salt Environment Overview
• Separate master setup per deployment
• Multi master setup
• Manage salt-master just like any other minion

6. Salt Environment Overview
• States and Pillar in separate repository
• States are common across all the deployments (Prod
Env)
• Pillar is maintained per environment

7. Salt Environment Overview
• Every database instance that gets created is a minion
• Automated (but secured) way of accepting and deleting
those keys

8. Things to consider for Operations
with Salt Stack

9. How is it different with Salt Stack?
• Access to salt master is needed
• Access to sensitive data (Pillar)
• Access to infrastructure
• Auditing individual user actions
• Auditing Salt Command execution

10. Access to Sensitive data
• Correct ACLs to protect the pillar data
• Consider encrypting your pillar data
• Protecting Salt master configurations and minion keys

11. Access to infrastructure
• Since user is on salt master, he has access to every node
in the infrastructure
• That too, as root, salt minion is running as root
• Protecting what type of actions can be performed by
individual users, by controlling access to modules

12. Some modules worth blocking
• cmd.*
• file.*
• cp.*
• config.*
• pillar.*
• ……
• (There are very few which are to be considered safe)

13. Auditing individual user actions
• Make sure salt commands are executed from their own
user account, this can be achieved with the help of
correct configurations
• Salt master logs every user action
• user name,
• salt command
• salt minion

14. Auditing Salt Command Execution
• Every salt command that was executed, get recorded in
salt job management framework
• Can query the details of a particular job?
• Consider using a returner, simplify serach

15. Controlling salt master through salt
• Controlling salt master like any other salt minion using
salt framework
• We do this by installing a minion on master
• This allows use to treat the master as any other minion in
the system, so we can write states, pillar and modules for
it

16. Salt States

17. Salt States
• Separating states to simplify day to day operations
• Writing states to catch and possiblify mitigate user errors

18. Separating states
• One big state files seems simple and state forward
• Try separating those states into simple group of actions
• This will allow you to run those specific actions when they
are needed and will not affect anything else
• This will be very good addition for operations

19. Separating states
• For example separating following from each other:
– Package Installation
– User creation
– Configuration parameter management
– System level service/package/upgrade management
• Note: Where ever required you need to maintain
necessary dependencies even after separation

20. Separating states
• If package installation is separate, at the time of
upgrading packages, users will be unaffected
• If user management is separate state file then, at the time
of password rotation, nothing for packages or system
level services will be affected

21. Writing states to catch user errors
• Writing state with necessary checks to help catch user
errors
• E.g while writing states for salt-master, make sure that its
getting executed on the master only. May be add a check
on grains[id]

22. Salt Modules
• Extending salt modules
• Writing modules to catch user errors

23. Extending Salt Modules
• Consider extending salt modules:
• If existing salt module is possible to cause harms if
misused
• If for certain action user needs to run multiple salt
commands
• If completing an action, user will need values from pillar
data

24. Thanks
• Demo Scripts/files @
https://github.com/saurabhsurana/salt-stack-demo