See how Kamil Cholewiński talks about Practical automation in Tech Talk episode 10 Visit pilot.co — World’s best engineering and design talent on demand. YouTube: https://youtu.be/x0eQ7x7xN8o

1. Practical automation

2. Automation?
For whom?
For DevOps and IT...
...and for anyone who can code
...and for mere mortals as well!
How?
With Ansible, Docker, Hubot...
...and with whatever else you see fit

3. Why automate?
Humans love being lazy
Do the extra work now
Be lazy forever after
Most things are already automated
Except for those you really need
Engineer: a person that takes 3 hours to turn a 2-hour task into a 1-hour task
Reap benefits until heat death of the universe
(Or until things break mysteriously...)
"If it's worth doing it twice, it's worth automating it"
Except when it's not
Often, attempting automating things can get you nowhere and waste time

4. So when is it a good idea to automate?
Anything repetitive
Create / delete user accounts, rotate passwords...
Upgrade vulnerable software packages...
Tweak a config file...
...on ALL the servers!
...all. the. time...
Anything daunting
Go through all email groups and delete those that have no members
Migrate data from an old service to a new one
Any low-hanging fruit
Things that are not hard, not too boring; still easy to script
Good ROI, easy to get into the habit of automating

5. What if something can't be automated?
Delegate it!
Maybe there's someone more competent?
Maybe it should be a person's decision, not a machine's?
Maybe you can replace yourself with a push-button, and give the button to the guy in
charge?
Think of a better process - talk to your fellow humans

6. Automation: For devops
(Where devops = developers and sysops working together.)

7. Problem: Managing server configuration
My story:
Developer suddenly turned sysadmin; inherited entire company's web hosting &
server infrastructure; ca 30-40 servers
Nothing was documented, everything was done by hand
Started an inventory in a spreadsheet; quickly realized it doesn't scale, and most
importantly: it's not machine-readable

8. First attempt: Ansible
Server / configuration management tool
Installed Ansible, re-created the inventory as .ini file, now I can ping ALL
hosts with one command:
ansible -m ping all
Yes, this is the way to go!
git commit -m "Hello, world"
We kept the repository ("The Cookbook") private
Lots of valuable, reusable code...
...But many parts tightly integrated with our infrastructure
2 years, 1800+ commits, 300+ pull requests, and 8 contributors later...

9. How do we use Ansible? What can it do for
you?
Need a new server or two...
ansible-playbook cloud-ec2.yml -e project=kittens -e env=live -e count=2
Provision the base system? Simple:
ansible-playbook provision-base-system.yml --limit tag_project_kittens
Deploy project "kittens" to staging environment:
ansible-playbook deploy-kittens.yml -e env=stage
Update our DNS zones:
ansible-playbook managed-dns.yml
Audit all servers for access; remove revoked public keys, etc:
ansible-playbook managed-cleanup.yml

10. How does a typical playbook look like?
---
- hosts: managed
vars:
docker_opts: ""
handlers:
- name: Restart Docker daemon
service: name=docker state=restarted
- name: Refresh apt cache
apt: update_cache=yes
- name: Reload systemd daemon
shell: /bin/systemctl daemon-reload
tasks:
- name: Add docker apt key
apt_key:
keyserver: pgp.mit.edu
id: "58118E89F3A912897C070ADBF76221572C52609D"
tags: install
- name: Install apt-transport-https
apt: name=apt-transport-https
tags: install
- name: Add docker apt sources
copy:
content: |
deb https://apt.dockerproject.org/repo {{
ansible_distribution.lower()
}}-{{
ansible_distribution_release
}} main
dest: /etc/apt/sources.list.d/docker.list
notify: Refresh apt cache
tags: install

11. - name: Flush handlers
meta: flush_handlers
- name: Create /etc/apt/preferences
file:
dest: /etc/apt/preferences
state: directory
tags: install
- name: Pin Docker version
copy:
dest: /etc/apt/preferences/docker
content: |
Package: docker-engine
Pin: version 1.12.*
Pin-Priority: 1100
tags: install
- name: Install Docker
apt: name=docker-engine
notify: Restart Docker daemon
tags: install
- name: Install Docker Python goodies
pip: name={{ item.name }} version={{ item.version }} executable=pip2.7
environment:
PATH: /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
with_items:
- { name: "docker-py", version: "1.9.0" }
- { name: "docker-compose", version: "1.8.0" }
tags: install
- name: Create docker group
group:
name: docker
state: present
notify: Restart Docker daemon
tags: install
- name: (systemd) Tweak docker daemon flags
when: docker_opts
copy:
dest: /etc/systemd/system/docker.service
content: |

12. content: |
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// {{ docker_opts }}
n
notify:
- Restart Docker daemon
- Reload systemd daemon
tags: install
- name: Flush handlers
meta: flush_handlers

13. Problem: Is Ansible good or bad?
Better than doing it by hand? Definitely!
Better than curl | sudo bash? Yes!!!
Better than shell scripts? Worse?
Better than any of the million other tools? Maybe...
Better than hand-rolling something? Not so sure now...

14. Problem: Is your automation efficient?
Now we can spin up & setup servers with a few simple commands. But it was
slow... Base+nginx+SSL+Python+app often took ~2h. We hated it and we
avoided it.
Solution? Create a base image
First, we tried Packer (http://packer.io/) - but it had annoying bugs; was very difficult
to debug
Second attempt: plain Ansible - worked much better
Maybe there are better tools to do these things?
We haven't even tried any of the other popular solutions: Chef, Puppet,
Fabric, Atlas/Terraform, etc.

15. Some take-aways
No automation is the worst.
Slow automation is no better than no automation.
Buggy automation is even worse!
Solutions can sometimes create more problems.
Research your tools, pick stuff that works for your needs!

16. Problem: So how exactly do you deploy these
"kittens"?
Our company has very dynamic and growing needs
Remember rule #1: automate the repetitive, boring, daunting stuff
Sysop team spent days writing elaborate Ansible playbooks to set up servers
for projects
But wait! Rule #2: can't automate something? Delegate it!
First attempt: let developers write Ansible playbooks...
Ansible has a pretty steep learning curve and some serious limitations
Difficulties micro-managing access
Difficulties achieving parity between local dev, vagrant, CI, stage, live...
Result: playbooks full of ugly hacks
Second attempt: Docker

17. Docker: some seriously magical stuff
happened
Docker puts apps into portable "containers"
Many difficult problems become easy
Developers are (mostly) in charge of their Dockerfiles
Developers no longer blocked by sysadmins
Builds are 100% automated and 100% reproducible
Much easier to review, audit, understand
Output is a fixed image; if it runs well in dev, stage, it will (probably) run well
in prod
Our deploy playbooks for Ansible are basically copy-paste & tweak; work
underway to unify them
We've created a set of base images that make it easy to follow our internal
conventions - people stick to conventions more often

18. Docker: but not everything is so rosy!
Another tool to learn for the (always busy) developers.
Difference between "build time" and "run time" greatly emphasized, for the
better and for the worse.
# XXX broken?
RUN service nginx start
RUN curl localhost
Many difficult problems stay difficult, or become even harder.
Not a silver bullet: we've dockerized most of our apps, but Ansible still
manages the raw computing infrastructure.
(You can pry Ansible from my cold, dead hands!)
(But I'm still writing my own configuration management tool, as a side
project.)

19. More conclusions?
Choose tools that empower your team-mates!
Different tools for different types of tasks

20. Automation: For IT staff
Often burdened with stuff that's not always easily automated:
Purchases, finance
Onboarding, helpdesk, troubleshooting
Setup, wiring, "hands" in the server room
Often burdened with daunting, scriptable tasks, but no time to develop the
automation
At our company, sysadmins and "IT guys" often work together closely

21. Some classical problems
Have hundreds of groups on G Suite. Review and delete empty ones.
Review group membership; create list of users from outside the domain.
Update everyone's email signatures.
Ensure there are no rogue .mp3's on anyone's Google Drive.
Create a project on JIRA with all the standard workflows, templates, etc.
...all of that, regularly.

22. Solutions?
Write some simple scripts!
Our in-house tool: https://bitbucket.org/u9/gadmin
E.g. to delete empty groups on G Suite:
$ gadmin groups count example.com
developers@example.com 11
foo@example.com 0
older-group@example.com 0
some-group@example.com 4
trash@example.com 0
$ gadmin groups count example.com | awk '$2=="0" {print $1}'
| xargs -r -n1 gadmin groups delete
Very much WIP, it may eat your laundry ;)

23. Solutions?
Write some more sophisticated scripts!
We have in-house tools for:
Device tracking for testers
JIRA project setup
Processing invoices
URL shortening
And more...

24. Solutions?
Use a third-party service!
We use BetterCloud - https://www.bettercloud.com/
Automates stuff like email signatures, Drive sharing policies, user
provisioning, and a lot more...

25. Automation: For developers: Hubot
Chatbot; built internally at Github, open-sourced: https://hubot.github.com/
Plenty of community-maintained plugins
Easy to roll your own: JS, Coffee...
We use it internally for a whole bunch of small stuff:
Standup alarms
hubot: @here it's 11:00 AM, standup time! https://hangups.example.com/kittens-
standup
Trigger a deployment
me: hubot deploy kittens 0.2.17 to dev
hubot: OK! Running:
hubot:
ansible-playbook deploy-kittens.yml -e version=0.2.17 -e env=dev

26. Automation: For developers: Hubot
...and for fulfilling our random gif needs
me: hubot pug me
hubot: https://media.tumblr.com/tumblr_lisv04akTl1qb08qmo1_500.jpg

27. Automation: For developers:make
Old, venerable tool - 1976, probably older than you
Surprisingly versatile
(If your problem can be expressed as producing an output file from a
bunch of input files!)
Build your project (obviously)
Maintain .gitignore via https://www.gitignore.io/
Maintain your scripts & dotfiles across machines (make + git)
These slides were made with make ;)
%.html: %.md
@pandoc --self-contained -f markdown -t slidy $< -o $@
make practical-automation.html

28. Automation: For everyone!
Creating sophisticated automation requires coding skills, understanding of
the problem domain, commitment to maintenance...
But not all tools need to be THIS sophisticated!
If enough people have the same problem...

29. Automation: For mere mortals: GUI tools,
Automation-as-a-Service...
Plenty of options, domain-specific or more general:
AutoHotKey - https://www.autohotkey.com/ - for Windows users
BetterCloud - https://www.bettercloud.com/ - G Suite and more
If This Then That - https://ifttt.com/
Zapier - https://zapier.com/
Various Slack bots; off-the-shelf Hubot plugins

30. Questions? Comments?
Any tools YOU would recommend?
Success stories?
Hate mail?
I will answer everything :)
cat /dev/audience
| while read question
do
echo 42 > /dev/audience
done
harry666t@gmail.com
kamil@rollc.at

31. Thanks!