This document provides an overview of using systemd to manage services effectively. It discusses defining service behavior and types, handling timeouts and failures, tightening security, and automating monitoring and management. The key steps outlined are to define expected behavior, plan for the unexpected, tighten security early, and automate monitoring. Various systemd directives and options are explained for tasks like controlling resources, dependencies, reloading, and failure handling.
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...
Effective service and resource management with systemd
1. Effective Service + Resource
Management with systemd
Adventures running millions of systemd services for
2. About Me and Pantheon
● Production users
of systemd since 2011
● Millions of units in
deployment across hundreds
of servers
● Committer since 2012
● Focus has been on journal
logging, control group
scalability, and general
systemd scalability
3. The Basic Steps
1 Define expected behavior and control
2 Plan for the unexpected
3 Tighten security
4 Manage, monitor, and automate
5. Type=simple (the default)
systemctl start foo.service systemctl stop foo.service
ExecStart=/usr/bin/foo
/etc/systemd/system/foo.service
Considered started for dependencies
Considered stopped for dependencies
[Service]
ExecStart=/usr/bin/foo
# systemctl daemon-reload
8. Type=notify
systemctl start foo.service systemctl stop foo.service
ExecStart...
[Service]
Type=notify
ExecStart=/usr/bin/foo
TimeoutStartSec=30
NotifyAccess=all ⬅maybe
/etc/systemd/system/foo.service
Called from daemon:
systemd-notify --ready
Best of
All
Types
16. WantedBy=
Implicit in late bootup:
systemctl start multi-user.target
[Service]
ExecStart=/usr/bin/foo
[Install]
WantedBy=multi-user.target
/etc/systemd/system/foo.service
Use Me
# systemctl enable foo.service
Added to transaction by wants:
systemctl start foo.service
multi-user.target completes startup
Operations in systemd happen in transactions, which are ordered sets of jobs.
…the successor to runlevels
17. Other Dependencies
Inclusion
These dependencies will add more units to a
transaction. There is no effect on ordering.
● Requires=bar.service
○ If foo.service is starting, starting bar.service
will also happen. A failure to start bar.service
will cause the entire transaction to fail.
○ Inverse of RequiredBy=
● Wants=bar.service
○ A weak form of Requires=. If bar.service fails
to start, the transaction will still succeed.
○ Inverse of WantedBy=
● Also=bar.service
○ When foo.service is enabled to start by
default, bar.service will also be enabled.
Ordering
These dependencies will order units in the
transaction. They will not add specified units if
they are not already in the transaction.
● Before=bar.service
○ If bar.service is in the same transaction, bar.
service will not begin starting until foo.
service is finished starting.
● After=bar.service
○ If bar.service is in the same transaction, foo.
service will not begin starting until bar.
service is finished starting.
[Unit]
Requires=bar.service
After=bar.service
...
/etc/systemd/system/foo.service
19. Control Groups Options for Resources
Absolute Limits
● MemoryLimit=
○ Caution: Certain limits cause further
allocation for a group to use swap, impacting
system performance.
● TasksMax=
○ Maximum combined processes and threads,
including kernel threads.
● BlockIOReadBandwidth=
○ Limits reading block I/O to the specified
bytes
per second.
● BlockIOWriteBandwidth=
○ Limits writing block I/O to the specified
bytes
Relative Controls and More
● CPUShares=
○ When under contention, CPU is allocated by
the kernel proportionally using the number
for this service versus the combined shares of
all others.
● BlockIOWeight=
○ When under contention, block I/O is
allocated by the kernel proportionally using
the number for this service versus the
combined weights of all others.
● nftables for network traffic
○ Not configured in systemd, but nftables can
leverage systemd’s control groups for traffic
shaping and other rules.
20. Using Traditional ulimit/rlimit Options
● CPU
○ LimitCPU=
○ LimitNPROC=
○ LimitRTPRIO=
○ LimitRTTIME=
○ LimitNICE=
● Disk
○ LimitCORE=
● Memory
○ LimitDATA=
○ LimitFSIZE=
○ LimitSTACK=
○ LimitMSGQUEUE=
○ LimitAS=
○ LimitRSS=
○ LimitMEMLOCK=
● Other
○ LimitSIGPENDING=
○ LimitNOFILE=
○ LimitLOCKS=
22. Directives for Detecting and Responding to Failure
Detecting Failure
● SuccessExitStatus=
○ Whitelist of exit codes and signals to indicate a
normal exit. Defaults to zero and the usual process
signals for healthy processes.
● RestartPreventExitStatus=
○ Blacklist of exit codes and signals to not trigger
restarts. Useful to restart on most failures but not
unrecoverable ones like a bad configuration.
● RestartForceExitStatus=
○ The opposite of the previous option.
● StartLimitInterval= and StartLimitBurst=
○ Thresholds at which attempted failure recovery
becomes a stickier failure.
Responding to Failure
● Restart=
○ Allows many options, but on-failure is
probably best for most cases.
● FailureAction=
○ Supports options like rebooting or shutting
down the system on service failure.
● StartLimitAction=
○ Same as FailureAction= but triggered when
StartLimit… thresholds get hit.
● systemctl reset-failed
○ Resets status units marked as failed.
23. Built-In Service Monitoring with Watchdog
Services
● WatchdogSec=
○ Configures the maximum interval for the
healthy service to ping systemd.
● $WATCHDOG_USEC and $WATCHDOG_PID
○ Environmental variables set for a service that
is expected to provide systemd with
watchdog pings.
● systemd-notify WATCHDOG=1
○ CLI; the most basic way for a service to send
systemd a watchdog ping.
● sd_notify(0, “WATCHDOG=1”);
○ A better way that requires linking to a
systemd library.
Overall System
● RuntimeWatchdogSec=
○ Configures the maximum interval for
systemd to ping the hardware watchdog
service (if it exists). If the hardware fails to
receive an expected ping, it will reboot the
system.
● ShutdownWatchdogSec=
○ Bounds the time the watchdog hardware is
willing to wait for a clean shutdown for the
triggered reboot.
27. Monitor at the Box Level
Plug a systemctl call into your monitoring tool:
# systemctl --state=failed --all
0 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
29. Pantheon is a Chef Shop
template '/etc/systemd/system/foo.service' do
mode '0644'
source 'foo.service.erb'
end
service 'foo.service' do
provider Chef::Provider::Service::Systemd
supports :status => true, :restart => true, :reload => true
action [ :enable, :start ]
end
30. Questions? Follow Ups?
Reach out to me @DavidStrauss.
Want to get more hands-on? We’re hiring!
pantheon.io/careers