Stackvora.tech

systemd Cheatsheet: Manage Linux Services Like a Pro

systemd has become the de facto init system and service manager for most modern Linux distributions, replacing older systems like SysVinit and Upstart.…

systemd has become the de facto init system and service manager for most modern Linux distributions, replacing older systems like SysVinit and Upstart. Its capabilities extend far beyond simple service orchestration, encompassing logging, device management, and more. This guide focuses on common systemctl commands and unit file configurations essential for daily administration tasks, targeting versions 240 and later, which are prevalent in contemporary distributions such as RHEL 8/9, CentOS 8, Ubuntu 20.04+, and Debian 11+.

Understanding systemd Unit Types

systemd manages various types of "units," each defining a specific resource or action. While service units are the most common, understanding others can be beneficial.

  • .service: Manages processes and daemons. This is the most frequently used unit type.
  • .socket: Listens on a socket (e.g., TCP, UDP, IPC) and activates services when traffic arrives.
  • .device: Manages kernel-provided devices, often used for hardware-dependent actions.
  • .mount: Controls mount points in the filesystem hierarchy.
  • .automount: Provides on-demand mounting for filesystems.
  • .target: Groups other units together, similar to runlevels in SysVinit (e.g., multi-user.target, graphical.target).
  • .timer: Activates other units (e.g., services) based on timers, acting like an enhanced cron.

Core systemctl Commands for Service Management

The systemctl command is your primary interface for interacting with systemd. Here are the essential operations:

Service Status and Control

To check the status of a service, start, stop, or restart it:

# Check the current status of the Nginx web server
systemctl status nginx.service

# Start a service
systemctl start apache2.service

# Stop a running service
systemctl stop postgresql.service

# Restart a service (stops and then starts)
systemctl restart sshd.service

# Reload a service's configuration without fully restarting (if supported by the service)
systemctl reload httpd.service

# Reload or restart a service if configuration changed
systemctl reload-or-restart uwsgi.service

Note that .service can often be omitted for service units, e.g., systemctl status nginx is equivalent to systemctl status nginx.service.

Enabling and Disabling Services

Enabling a service ensures it starts automatically at boot. Disabling prevents this.

# Enable a service to start at boot
systemctl enable jenkins.service

# Disable a service from starting at boot
systemctl disable firewalld.service

# Enable and start a service immediately
systemctl enable --now caddy.service

# Disable and stop a service immediately
systemctl disable --now cockpit.service

# Check if a service is enabled
systemctl is-enabled docker.service

# Mask a service: Prevents it from being started manually or by other services.
# This creates a symlink to /dev/null, making it impossible to activate.
systemctl mask autofs.service

# Unmask a service to restore its normal behavior
systemctl unmask autofs.service

Listing Units and Their States

To get an overview of all units or filtered lists:

# List all loaded units (active or inactive, running or not)
systemctl list-units

# List all units (including those not yet loaded)
systemctl list-units --all

# List only running (active and loaded) services
systemctl list-units --type=service --state=running

# List all failed units for troubleshooting
systemctl list-units --failed

# List all available unit files (whether enabled or not)
systemctl list-unit-files

Useful filters for list-units include --type= (e.g., service, socket, target) and --state= (e.g., active, inactive, failed, loaded).

Journalctl: The systemd Log Viewer

systemd consolidates logging through the journal. journalctl is the command-line utility to access these logs.

# View all journal entries
journalctl

# View logs for a specific service
journalctl -u nginx.service

# View logs for a specific service and follow new entries in real-time
journalctl -u apache2.service -f

# View logs for a specific service since a certain time
journalctl -u postgresql.service --since "2 hours ago"
journalctl -u sshd.service --since "2023-01-15 10:00:00"

# View logs with human-readable timestamps
journalctl -u docker.service -o short-monotonic

# View kernel messages (equivalent to dmesg)
journalctl -k

# View journal entries for the current boot only
journalctl -b

# Clear old journal entries (e.g., keep only the last 7 days or 1GB)
# Use with caution! This removes logs permanently.
journalctl --vacuum-time=7d
journalctl --vacuum-size=1G

By default, journalctl uses pager (less), allowing searching (/), moving (arrow keys), and exiting (q).

Creating and Managing Custom Service Units

For custom applications or non-packaged software, you'll need to create your own .service unit files.

Unit File Location and Structure

Unit files are typically located in:

  • /etc/systemd/system/: For custom unit files or overriding vendor units. This location takes precedence.
  • /run/systemd/system/: For runtime-generated unit files.
  • /usr/lib/systemd/system/: For unit files provided by installed packages (vendor units).

When creating a new unit, always place it in /etc/systemd/system/. After creating or modifying a unit file, you must reload the systemd daemon:

systemctl daemon-reload

Example: A Simple Web Application Service Unit

Let's create a service for a hypothetical Python web application. Assume the application is /usr/local/bin/mywebapp/app.py and uses a virtual environment at /opt/mywebapp/venv.

/etc/systemd/system/mywebapp.service:

[Unit]
Description=My Awesome Python Web Application
Documentation=https://docs.mywebapp.example.com
After=network.target

[Service]
# User and Group under which the service will run
User=mywebappuser
Group=mywebappgroup

# Working directory for the service
WorkingDirectory=/usr/local/bin/mywebapp

# Define environment variables if needed
Environment="FLASK_APP=app.py"
Environment="PATH=/opt/mywebapp/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# The command to execute when the service starts
# Type=simple is default, for services that fork, use Type=forking.
# Here, we use exec gunicorn for proper signal handling.
ExecStart=/opt/mywebapp/venv/bin/gunicorn -w 4 -b 0.0.0.0:8000 app:app

# How systemd should handle the service exiting
# on-failure: Restart if the service process exits with a non-zero status
# always: Always restart, regardless of exit status
# on-abnormal: Restart if exits with signal or abnormal exit code
# on-success: Restart only if exits with zero exit code (rare)
Restart=on-failure

# Time to wait before restarting (e.g., 5 seconds)
RestartSec=5

# Limit CPU usage, memory, etc.
# CPUQuota=100%
# MemoryLimit=512M

# Standard output and error handling
StandardOutput=journal
StandardError=journal

[Install]
# This section defines when the service should be started automatically.
# multi-user.target is generally used for server environments.
WantedBy=multi-user.target

After creating the file:

# Reload systemd to recognize the new unit file
sudo systemctl daemon-reload

# Enable the service to start at boot and start it now
sudo systemctl enable --now mywebapp.service

# Check its status
systemctl status mywebapp.service

# View its logs
journalctl -u mywebapp.service -f

Key Unit File Directives Explained

A unit file is structured into sections, each containing key-value directives:

[Unit] Section:

  • Description=: Human-readable description of the unit.
  • Documentation=: URL to documentation.
  • After=, Before=: Specifies ordering dependencies. After=network.target is common for network-dependent services.
  • Requires=, Wants=: Specifies stronger/weaker dependencies. Requires= means the unit won't start if the dependency fails; Wants= is less strict.

[Service] Section (for .service units):

  • Type=: Defines the process startup type (e.g., simple, forking, oneshot, notify, dbus). simple is default for processes that run in the foreground. forking is for traditional daemons that fork a child process and exit.
  • ExecStart=: The command executed to start the service.
  • ExecStop=: Optional command to stop the service. If omitted, systemd sends SIGTERM.
  • ExecReload=: Optional command to reload configuration.
  • User=, Group=: User and group under which the service process runs.
  • WorkingDirectory=: Sets the current working directory for ExecStart.
  • Environment=, EnvironmentFile=: Sets environment variables.
  • Restart=: Defines when the service should be restarted (no, on-success, on-failure, always, etc.).
  • RestartSec=: Delay before restarting.
  • TimeoutStartSec=, TimeoutStopSec=: Maximum time to wait for start/stop operations.
  • StandardOutput=, StandardError=: Where stdout/stderr should be directed (inherit, null, journal, file:/path).

[Install] Section:

  • WantedBy=, RequiredBy=: Specifies the target(s) that should enable this unit. WantedBy=multi-user.target is typical for most server services.

Common Pitfalls

  • Forgetting systemctl daemon-reload: After modifying a unit file, systemd needs to reload its configuration. Without this, changes won't take effect.
  • Incorrect ExecStart path or permissions: Ensure the command specified in ExecStart is fully qualified (e.g., /usr/bin/python3 instead of just python3) and has execute permissions for the specified User.
  • Type=forking vs. Type=simple: Many modern applications run in the foreground (Type=simple). If your application daemonizes itself (forks and exits), use Type=forking, otherwise systemd might incorrectly assume the service failed.
  • Missing dependencies (After=, Wants=): A service failing to start because the network isn't up, or a database isn't ready, is common. Use After=network.target or After=postgresql.service accordingly.
  • Logs not appearing: Check StandardOutput= and StandardError= directives. If they are set to inherit, they might go to /dev/null or a default location. Setting them to journal is usually best practice.
  • Overriding vendor units incorrectly: Instead of directly editing files in /usr/lib/systemd/system/ (which can be overwritten by package updates), create an override file (e.g., /etc/systemd/system/nginx.service.d/override.conf) or copy the unit to /etc/systemd/system/ and modify it there.

Back to the knowledge base · Ask the AI assistant