Read systemd Logs With journalctl

journalctl is the primary utility for querying and displaying logs collected by systemd-journald , the logging service within the systemd ecosystem.…

journalctl is the primary utility for querying and displaying logs collected by systemd-journald, the logging service within the systemd ecosystem. Unlike traditional syslog, systemd-journald stores logs in a structured, binary format, offering enhanced querying capabilities, reliable timestamps, and metadata preservation. This article delves into the practical aspects of using journalctl to efficiently navigate, filter, and export system logs on Linux systems utilizing systemd.

Understanding systemd-journald Persistence

By default, systemd-journald logs are often stored in volatile memory (/run/log/journal) and are thus lost across reboots. For most production environments, persistent logging is critical. To enable persistence, you must create a directory for the journal files on disk:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald.service

After this, systemd-journald will store its journal files under /var/log/journal/<machine-id>/. The <machine-id> is a unique identifier for the system, ensuring logs are properly segregated if the /var/log/journal directory is shared (e.g., in a container environment or shared storage setup, though this is uncommon for journald). You can verify the storage location with:

journalctl --disk-usage

This command will show the total disk space consumed by the journal files and their storage path. For finer control over journal persistence, such as limiting disk usage or enabling compression, edit /etc/systemd/journald.conf. Relevant parameters include:

• Storage=persistent: Ensures logs are saved to /var/log/journal. Other options are auto (default, persistent if /var/log/journal exists), volatile, and none.
• SystemMaxUse=500M: Sets a maximum disk usage for the journal, e.g., 500 MB.
• SystemKeepFree=10%: Ensures that at least 10% of the disk space remains free.
• Compress=yes: Compresses older journal files to save disk space.

After modifying journald.conf, restart the service for changes to take effect: sudo systemctl restart systemd-journald.service.

Basic Log Inspection and Following

The simplest way to view all collected logs is journalctl without any arguments. This displays all logs from the oldest entry to the newest, paginated by default using a pager like less.

journalctl

To view logs in real-time, similar to tail -f, use the -f or --follow flag:

journalctl -f

This is particularly useful for monitoring system activity or application output as it happens.

Filtering by systemd Unit

One of the most powerful features of journalctl is its ability to filter logs by specific systemd units. This is invaluable for debugging service-related issues.

journalctl -u nginx.service
journalctl -u sshd.service -f

The first command shows all logs for the nginx service. The second command follows logs for the sshd service in real-time. You can omit the .service suffix for most common unit types (e.g., nginx instead of nginx.service).

To view logs for multiple units, simply specify them:

journalctl -u nginx -u php-fpm

Filtering by Time

journalctl offers flexible time-based filtering. The --since and --until flags accept various formats:

• Relative times: "1 hour ago", "20 minutes ago", "yesterday", "now"
• Absolute times: "2023-10-27 10:30:00", "2023-10-27 10:30"
• Specific dates: "2023-10-27"

# Show logs from the last hour
journalctl --since "1 hour ago"

# Show logs from yesterday until 5 minutes ago
journalctl --since "yesterday" --until "5 minutes ago"

# Show logs for a specific time window for the SSH daemon
journalctl -u sshd --since "2023-10-27 08:00" --until "2023-10-27 09:00"

Output Formatting

By default, journalctl outputs human-readable text. However, for scripting or integration with other tools, different output formats are available via the -o or --output flag:

# View logs in JSON format for the Nginx service
journalctl -u nginx -o json

# View verbose logs for the last 10 lines
journalctl -n 10 -o verbose

The -n or --lines flag specifies the number of lines to display, similar to tail -n.

Advanced Filtering

Beyond basic unit and time filtering, journalctl supports more granular selection criteria.

Filtering by Priority/Severity

Logs have different priority levels (e.g., error, warning, info, debug). Use the -p or --priority flag:

# Show all error (priority 3) and critical (priority 2) messages
journalctl -p err

# Show logs with priority level warning (4) or higher (lower number = higher priority)
journalctl -p warning

# Priority levels:
# 0: emerg (system is unusable)
# 1: alert (action must be taken immediately)
# 2: crit (critical conditions)
# 3: err (error conditions)
# 4: warning (warning conditions)
# 5: notice (normal but significant condition)
# 6: info (informational messages)
# 7: debug (debug-level messages)

Filtering by Fields

systemd-journald collects numerous structured fields with each log entry. You can filter by these fields directly. Common fields include:

• _PID: Process ID
• _UID: User ID
• _COMM: Executable name
• _SYSTEMD_UNIT: systemd unit name (same as -u filter)
• _HOSTNAME: Hostname of the logging machine
• MESSAGE: The actual log message content

# Show messages from a specific PID
journalctl _PID=1234

# Show logs from the 'bash' executable
journalctl _COMM=bash

# Show kernel logs (boot messages)
journalctl -k

# Show messages related to network interfaces
journalctl -g "eth0|enp" # -g is for grep-like matching on MESSAGE field

For a complete list of fields available in an entry, use journalctl -o verbose.

Exporting Logs to Syslog or Files

While journalctl is powerful, some centralized logging systems (SIEMs) or legacy tools might still expect classic syslog format. systemd-journald can forward logs to a traditional syslog daemon (e.g., rsyslog or syslog-ng) if one is running. This is usually configured by default on most distributions, where rsyslog is set up to read from /run/systemd/journal/syslog.

To manually export logs, you can pipe journalctl output:

# Export all logs for Nginx to a file in default short format
journalctl -u nginx > nginx_logs.txt

# Export all logs from today in JSON format to a compressed file
journalctl --since "today" -o json | gzip > today_logs.json.gz

# Export binary journal data for later analysis on another system
journalctl -u httpd --since "1 week ago" -o export > httpd_journal_export.journal

To analyze an exported binary journal file on another machine:

journalctl --file=httpd_journal_export.journal

Common Pitfalls and Troubleshooting

• Volatile Logs Lost on Reboot: If your logs disappear after a reboot, systemd-journald is likely configured for volatile storage. Create /var/log/journal and restart systemd-journald.service as described above.
• Disk Space Consumption: Persistent journals can consume significant disk space. Regularly monitor journalctl --disk-usage and configure SystemMaxUse and SystemKeepFree in /etc/systemd/journald.conf to manage this.
• Time Zone Issues: journalctl by default displays timestamps in the local system's timezone. This can be confusing when analyzing logs from systems in different timezones. Use -u or --utc to display timestamps in UTC.
• Missing Application Logs: If an application's logs are not appearing in journalctl, ensure it's properly configured to send output to stdout or stderr, especially if it's run as a systemd service. systemd captures these streams automatically. If the application writes to its own log files, journalctl will not see them unless a mechanism like systemd-journal-remote or a log forwarder is used.
• Permissions: Regular users can typically read journal logs. If a user cannot access logs, check permissions on /var/log/journal. Group systemd-journal often has read access. Adding a user to this group (sudo usermod -aG systemd-journal <username>) might be necessary, followed by re-logging for changes to take effect.