In our On Premise hosting environment we still run a lot of applications on traditional stacks without using containers. In order to run them in a secured way we created a mature patch automation. Thanks to ansible, rundeck, icinga and a bunch of other opensource tools we are able to update and reboot most of our systems without our customers noticing. We do that throughout the day on a regular base using rundeck, or even on short notice if another “heartbleed” occurs.

1. AUTOMATED PATCH MANAGEMENT
WITH ANSIBLE AND RUNDECK
Schwarz IT KG - @crsp & @shakalandy

2. ABOUT US
2
Andreas Lehr
@shakalandy
Rico Spiesberger
@crsp
“Hosting and Domain Services” department
- lidl.de
- lidl-reisen.de/.at/.ch/...
- lidl-shop.nl/.be/.cz/.pl/...
- mobile app backend (30 countries)

3. ABOUT SCHWARZ IT
➔ Central IT of the Schwarz Group (Lidl, Kaufland, PreZero,
GreenCycle,...)
➔ ~ 3000 employees
➔ HQ in Weinsberg/Heilbronn - Location in Berlin
➔ We have Jobs - https://jobs.schwarz
3

4. WHAT’S WRONG HERE?
4
WTF?

5. AUTOMATED PATCHING
5
WHY WE’VE DONE IT
HOW WE’VE DONE IT
LIVE-DEMO (sort of)

6. WHY AUTOMATED PATCHING?
6
manual patching takes too
much valuable time

7. WHY AUTOMATED PATCHING?
7
Make security and auditors
happy

8. WHY AUTOMATED PATCHING?
8
Have a mature and reliable
process

9. HOW WE’VE DONE IT
c 9
Ansible and Rundeck
• 1 week cycle for DEV/TEST/QA
• 4 week cycle for PROD
• Emergency stuff can be patched
without prior information
• Target: no manual process, but fully
automated

10. PATCHING WORKFLOW
10
Set Monitoring Downtime

11. PATCHING WORKFLOW
11
Create VMWare Snapshot

12. PATCHING WORKFLOW
12
Send Notifications

13. PATCHING WORKFLOW
13
Host Preparation

14. PATCHING WORKFLOW
14
finally: upgrade time

15. PATCHING WORKFLOW
15
reboot if needs-restarting

16. PATCHING WORKFLOW
c 16
● remove old kernels
● patching date > /etc/last_patching (Monitoring, motd, ansible
CMDB)
● activate Loadbalancer health checks
● clean up (yum clean up, etc)
● update “patchlist” documentation (For auditors and POs)
● remove downtime
● remove snapshot (3 days later)
after reboot tasks

17. IMPEDIMENTS AND RECOMMENDATIONS ON AUTOMATED PATCHING
17
have fixed timeslots

18. IMPEDIMENTS AND RECOMMENDATIONS ON AUTOMATED PATCHING
18
Delete Snapshots
automatically

19. IMPEDIMENTS AND RECOMMENDATIONS ON AUTOMATED PATCHING
19
Rebooting HW servers
takes some time…..

20. IMPEDIMENTS AND RECOMMENDATIONS ON AUTOMATED PATCHING
20
preload packages

21. IMPEDIMENTS AND RECOMMENDATIONS ON AUTOMATED PATCHING
21
have enough space in
/var/yum and /tmp

22. IMPEDIMENTS AND RECOMMENDATIONS ON AUTOMATED PATCHING
22
parallel patching:
ansible forks=20+ and strategy: free

23. LIVE-DEMO!?!?

24. QUESTIONS?

25. Thanks. Don’t forget - https://jobs.schwarz
25