Managing Services with systemd

At a glance#

• Purpose: Start, stop, inspect and troubleshoot services with systemd, and write unit files for custom applications.
• Applies to: RHEL 7+, Ubuntu 16.04+, and any modern systemd distribution.
• Risk: Low for inspection; Medium when enabling or masking services on production hosts.
• Time: 15 minutes for routine work; 45 minutes to write and test a new unit.

Overview#

systemd is the init system and service manager on every current Linux distribution. It starts services at boot, restarts them when they fail, and captures their output into the journal.

The commands below are the ones actually used day to day, followed by how to write a unit file for an application that does not ship with one.

Everyday commands#

# state of a service
sudo systemctl status nginx

# start / stop / restart
sudo systemctl start nginx
sudo systemctl stop nginx
sudo systemctl restart nginx

# reload configuration without dropping connections
sudo systemctl reload nginx

# start automatically at boot
sudo systemctl enable nginx

# do not start at boot
sudo systemctl disable nginx

# enable and start in one step
sudo systemctl enable --now nginx

Note: restart stops and starts, dropping active connections. reload re-reads configuration in place and is what you want for nginx, Apache and most daemons. If a service does not support reload, systemd falls back to a restart — check with systemctl show nginx -p CanReload.

Enabled and started are independent. A service can be running but not enabled, in which case it disappears after the next reboot. This catches people out after maintenance:

systemctl is-active nginx     # running right now?
systemctl is-enabled nginx    # will it start at boot?

Finding services#

# all running services
systemctl list-units --type=service --state=running

# everything that failed
systemctl --failed

# all installed unit files and whether they are enabled
systemctl list-unit-files --type=service

# search by name
systemctl list-units --type=service | grep -i mysql

# what a service depends on
systemctl list-dependencies nginx

systemctl --failed is the first command to run on a server that is behaving oddly. It shows everything that tried to start and could not.

Reading logs#

systemd captures service output into the journal:

# logs for one service
sudo journalctl -u nginx

# follow live
sudo journalctl -u nginx -f

# last 50 lines
sudo journalctl -u nginx -n 50

# since a time
sudo journalctl -u nginx --since "10 minutes ago"
sudo journalctl -u nginx --since "2026-08-11 09:00" --until "2026-08-11 10:00"

# only errors
sudo journalctl -u nginx -p err

# this boot only
sudo journalctl -u nginx -b

Full detail is in Log Analysis with journalctl and rsyslog.

Writing a unit file#

For an application without a packaged service — a Go binary, a Python daemon, an internal tool.

1. Create the unit#

sudo nano /etc/systemd/system/myapp.service

[Unit]
Description=Internal reporting API
Documentation=https://wiki.internal/myapp
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5s
StandardOutput=journal
StandardError=journal

# basic hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true

[Install]
WantedBy=multi-user.target

Key directives:

Warning: ExecStart must be an absolute path, and the command must stay in the foreground for Type=simple. A binary that forks and exits will be treated as crashed and restarted repeatedly.

2. Create the service account#

sudo useradd -r -s /sbin/nologin -d /opt/myapp myapp
sudo chown -R myapp:myapp /opt/myapp

-r creates a system account with no home directory and no login.

3. Load and start#

sudo systemctl daemon-reload
sudo systemctl enable --now myapp
sudo systemctl status myapp

Warning: daemon-reload is required after creating or editing any unit file. Without it systemd keeps using the cached version and your changes appear to do nothing.

Modifying a packaged unit#

Never edit files in /lib/systemd/system/ or /usr/lib/systemd/system/ — a package update overwrites them. Use an override:

sudo systemctl edit nginx

This opens an empty drop-in at /etc/systemd/system/nginx.service.d/override.conf. Add only what you are changing:

[Service]
Restart=always
RestartSec=10s
LimitNOFILE=65535

sudo systemctl daemon-reload
sudo systemctl restart nginx

Check what is actually in effect:

systemctl cat nginx
systemctl show nginx -p Restart -p LimitNOFILE

systemctl cat prints the original unit followed by every override, which is the quickest way to understand a service whose behaviour does not match its packaged unit.

Timers#

systemd timers are an alternative to cron with better logging and dependency handling.

sudo nano /etc/systemd/system/backup.service

[Unit]
Description=Nightly backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

sudo nano /etc/systemd/system/backup.timer

[Unit]
Description=Run nightly backup at 02:00

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
systemctl list-timers

Persistent=true runs a missed job at the next boot — useful on machines that are not always on. Cron has no equivalent.

Masking#

Masking makes a service impossible to start, even as a dependency of something else:

sudo systemctl mask apache2
sudo systemctl unmask apache2

Use it when a package keeps starting a service you do not want. disable only stops it starting at boot; another unit can still pull it in.

Verification#

# running and enabled
systemctl is-active myapp
systemctl is-enabled myapp

# no failures on the host
systemctl --failed

# effective configuration including overrides
systemctl cat myapp

# it actually survives a reboot
sudo reboot
# then
systemctl status myapp

The reboot test matters. A service that runs now but is not enabled looks perfectly healthy until the next unplanned restart.

Troubleshooting#

Related#

• Log Analysis with journalctl and rsyslog
• Scheduling Tasks with Cron
• Linux Performance Troubleshooting: CPU, Memory, Disk and Network
• Windows Services and Task Scheduler