Stackvora.tech

systemd Survival Guide: Units, Targets and Journals

systemd has become the de facto init system and service manager for most modern Linux distributions, including RHEL/CentOS 7+, Ubuntu 15.04+, Debian 8+,…

systemd has become the de facto init system and service manager for most modern Linux distributions, including RHEL/CentOS 7+, Ubuntu 15.04+, Debian 8+, and Fedora. Its primary role is to initialize the system, manage services, and log system events efficiently. Understanding its core components—units, targets, and the journal—is critical for effective system administration and troubleshooting.

This guide delves into the practical aspects of systemd, providing specific examples and commands to manage services, inspect system state, and diagnose issues on servers running systemd.

Understanding systemd Units

At the heart of systemd are "units." A unit is a configuration file that defines how systemd should manage a resource. These files typically reside in /etc/systemd/system/ (for custom configurations), /run/systemd/system/ (for runtime-generated units), or /usr/lib/systemd/system/ (for units installed by packages).

There are several types of units, each identified by its file extension:

  • .service: Manages daemons and applications. This is the most common type.
  • .socket: Describes IPC or network sockets for socket-activated services.
  • .device: Represents kernel-recognized devices.
  • .mount, .automount: Controls mount points and automatic mounts.
  • .swap: Manages swap partitions or files.
  • .target: Groups other units and defines synchronization points in the boot process.
  • .path: Activates services when a specific file system path changes.
  • .timer: Schedules events, similar to cron jobs.
  • .slice: Groups processes for resource management (e.g., cgroups).
  • .scope: Similar to slice units, but for externally created processes.

Service Unit Files (.service)

Service units are the most frequently interacted-with unit type. They define how to start, stop, reload, and monitor a specific service. A typical service unit file is divided into three main sections:

  • [Unit]: Contains generic information about the unit, its description, and dependencies.
    • Description=: A human-readable description of the service.
    • Documentation=: URI pointing to documentation for this service.
    • After=, Before=: Specifies ordering dependencies. E.g., After=network.target means this service should start only after network interfaces are up.
    • Requires=, Wants=: Specifies hard or soft dependencies. Requires means if the dependency fails, this unit also fails. Wants is a weaker dependency.
  • [Service]: Defines the behavior of the service itself.
    • Type=: Defines the process start-up type (e.g., simple, forking, oneshot, dbus, notify, idle). simple is default for most apps that don't fork. forking is for services that fork a child process and the parent exits.
    • ExecStart=: The command to execute to start the service.
    • ExecStop=: The command to execute to stop the service gracefully.
    • ExecReload=: The command to execute to reload the service's configuration.
    • Restart=: Defines when the service should be restarted (e.g., no, on-success, on-failure, on-abnormal, on-watchdog, always). on-failure is a common choice.
    • User=, Group=: Specifies the user and group under which the service should run.
    • WorkingDirectory=: Sets the working directory for the executed commands.
    • Environment=, EnvironmentFile=: Sets environment variables for the service.
    • TimeoutStartSec=, TimeoutStopSec=: Configures timeouts for start and stop operations.
  • [Install]: Contains installation information, primarily used by systemctl enable.
    • WantedBy=, RequiredBy=: Specifies which target(s) should pull this unit into its dependency tree when enabled. WantedBy=multi-user.target is common for server applications.

Here's a custom service unit example for a Node.js application:

# /etc/systemd/system/myapp.service
[Unit]
Description=My Node.js Application
Documentation=https://example.com/myapp-docs
After=network.target

[Service]
User=myappuser
Group=myappuser
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/node /opt/myapp/server.js
# For older Node versions where /usr/bin/node might be a symlink to /usr/bin/nodejs,
# adjust ExecStart accordingly, e.g., ExecStart=/usr/bin/nodejs /opt/myapp/server.js
Restart=on-failure
RestartSec=5s # Wait 5 seconds before restarting
StandardOutput=journal
StandardError=journal
Environment=NODE_ENV=production
LimitNOFILE=65536 # Increase open file limit for high-concurrency apps

[Install]
WantedBy=multi-user.target

After creating or modifying a unit file, always run sudo systemctl daemon-reload to inform systemd about the changes. Then, you can enable and start your service:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service

The --now flag automatically starts the service after enabling it. Without --now, the service would only start on the next boot.

Managing Services with systemctl

systemctl is the primary command-line utility for controlling systemd. Here are essential commands:

  • systemctl start <unit>: Starts a unit (e.g., nginx.service).
  • systemctl stop <unit>: Stops a unit.
  • systemctl restart <unit>: Restarts a unit.
  • systemctl reload <unit>: Reloads configuration for a unit (if ExecReload is defined).
  • systemctl enable <unit>: Enables a unit to start automatically at boot. This creates a symlink in /etc/systemd/system/<target>.wants/.
  • systemctl disable <unit>: Disables a unit from starting automatically.
  • systemctl status <unit>: Shows the current status, recent logs, and PID of a unit.
  • systemctl is-active <unit>: Checks if a unit is currently active.
  • systemctl is-enabled <unit>: Checks if a unit is configured to start on boot.
  • systemctl list-units --type=service: Lists all active service units. Add --all to show inactive ones.
  • systemctl list-unit-files --type=service: Lists all installed service unit files and their enable status.
# Check the status of NGINX
systemctl status nginx.service

# Restart NGINX
systemctl restart nginx.service

# Enable NGINX to start on boot and start it now
systemctl enable --now nginx.service

# Reload systemd daemon to pick up new unit files or changes
systemctl daemon-reload

# Get details about a specific unit file location and content
systemctl cat nginx.service

Working with systemd Targets

Targets are special unit files (.target) that group other systemd units and serve as synchronization points during system boot or operation. They are analogous to runlevels in older init systems but are more flexible because a system can be in multiple targets simultaneously.

Common targets:

  • multi-user.target: Represents a system with all services running, ready for multiple users, but without a graphical interface. Similar to runlevel 3.
  • graphical.target: Includes multi-user.target and adds units required for a graphical login. Similar to runlevel 5.
  • rescue.target: Provides a rescue shell, used for emergency system repairs.
  • emergency.target: A more minimal environment than rescue.target, only mounting the root filesystem read-only.
  • reboot.target, poweroff.target, halt.target: Units for system shutdown and reboot.

You can inspect the default target and switch between targets:

  • systemctl get-default: Shows the default target (e.g., multi-user.target).
  • systemctl set-default <target>: Sets a new default target.
  • systemctl isolate <target>: Switches to a specific target. This stops units not required by the new target and starts those that are. Use with caution on production systems.
# Check current default target
systemctl get-default

# Set graphical.target as default (e.g., for a workstation)
# Not recommended for servers
# systemctl set-default graphical.target

# Boot into multi-user.target (common for servers)
systemctl set-default multi-user.target

Leveraging the systemd Journal

journald is the component of systemd responsible for collecting and managing log data. It centralizes messages from the kernel, initrd, services, and applications into a structured, indexed binary format. This offers significant advantages over traditional plain-text logs, such as faster searching and filtering.

The primary command to interact with the journal is journalctl.

  • journalctl: Displays all collected log messages.
  • journalctl -f: Shows the latest journal entries and "follows" new messages in real-time (like tail -f).
  • journalctl -u <unit>: Displays log messages for a specific unit (e.g., nginx.service).
  • journalctl -u <unit> -f: Follows logs for a specific unit.
  • journalctl --since "YYYY-MM-DD HH:MM:SS": Filters logs by a specific start time.
  • journalctl --until "YYYY-MM-DD HH:MM:SS": Filters logs by a specific end time.
  • journalctl --since "1 hour ago": Filters logs from a relative time.
  • journalctl -p <priority>: Filters by message priority (e.g., emerg, alert, crit, err, warning, notice, info, debug). For example, -p err shows errors and above.
  • journalctl _PID=<PID>: Filters logs by process ID.
  • journalctl -o json: Outputs logs in JSON format for easier parsing.
  • journalctl --disk-usage: Shows the current disk usage of the journal.
  • journalctl --vacuum-size=500M: Reduces journal size to 500MB.
  • journalctl --vacuum-time=7d: Removes journal entries older than 7 days.
# View logs for NGINX in real-time
journalctl -u nginx.service -f

# Show all error messages from the last hour
journalctl --since "1 hour ago" -p err

# Display logs related to a specific process ID (replace 12345 with actual PID)
journalctl _PID=12345

# Show kernel messages from the current boot
journalctl -k

# Show system messages from a previous boot (use -b -1 for last boot)
journalctl -b

By default, journald stores logs in /run/log/journal/ (volatile, lost on reboot). For persistent storage across reboots, ensure Storage=persistent is set in /etc/systemd/journald.conf, which will then store logs in /var/log/journal/.

Common Pitfalls and Troubleshooting

  • Forgetting daemon-reload: After modifying any .service, .target, or other unit files, systemd needs to reload its configuration: sudo systemctl daemon-reload. Without this, your changes won't take effect.
  • Incorrect Paths in ExecStart: Ensure all paths in ExecStart (and other Exec* commands) are absolute. Relative paths can lead to unexpected failures.
  • Permissions Issues: The User= and Group= directives are crucial. If your service fails to start, check if the specified user has necessary permissions to access files, directories, and ports.
  • Service Type Mismatch: If your service forks into a background process (like older Apache versions), using Type=simple will cause systemd to incorrectly think the service exited. For such services, use Type=forking and potentially PIDFile=.
  • Dependency Loops: Carefully review After=, Before=, Requires=, and Wants= directives. Incorrect configurations can lead to services not starting or boot failures. Use systemd-analyze plot > boot.svg to visualize boot dependencies if facing complex issues.
  • Journal Persistence: If you lose logs after a reboot, confirm that Storage=persistent is enabled in /etc/systemd/journald.conf and that /var/log/journal/ exists and has correct permissions (root:systemd-journal).
  • Resource Limits: For high-concurrency applications, adjust resource limits like LimitNOFILE (number of open files) or LimitNPROC (number of processes) in the [Service] section to prevent "Too many open files" errors.

Back to the knowledge base · Ask the AI assistant