Create a Drop-In Override Without Editing Vendor Units

Systemd unit files, particularly those provided by distribution packages, often require customization to align with specific operational needs or security…

Systemd unit files, particularly those provided by distribution packages, often require customization to align with specific operational needs or security policies. Directly modifying these vendor-supplied files (typically found in /usr/lib/systemd/system/ or /lib/systemd/system/) is a practice fraught with peril, as package updates will invariably overwrite your changes. Systemd offers a robust mechanism for safely overriding unit parameters without touching the original files: drop-in unit directories.

This article details the creation and management of drop-in overrides, focusing on practical examples and the underlying systemd logic. We will explore how to modify service behaviors like resource limits, environment variables, and execution parameters, ensuring these customizations persist across package upgrades and system reconfigurations.

Understanding Systemd Unit Load Order

Before diving into drop-ins, it's crucial to understand how systemd resolves conflicting unit definitions. Systemd loads unit files from several directories, processing them in a specific order:

• /etc/systemd/system/: User-defined units and local overrides. This is the primary location for administrators to place custom units or drop-ins.
• /run/systemd/system/: Runtime-generated units. These are typically created by scripts or other services and are transient.
• /usr/local/lib/systemd/system/: Units installed by local administrators, often from source builds.
• /usr/lib/systemd/system/ or /lib/systemd/system/: Vendor-supplied units from distribution packages.

When multiple unit files define the same parameter, the one loaded last takes precedence. Drop-in files leverage this hierarchy by being placed in a specially named subdirectory within /etc/systemd/system/, ensuring their directives override those in the original vendor unit.

Creating a Drop-In Override with systemctl edit

The simplest and recommended method for creating drop-in overrides is using systemctl edit. This command automatically handles the creation of the necessary directory structure and opens an editor for the override file.

Consider a scenario where we need to increase the maximum number of open files (LimitNOFILE) for the Nginx web server, which by default might be set lower than required for high-traffic environments. The default Nginx unit file (e.g., nginx.service) from your distribution might reside at /usr/lib/systemd/system/nginx.service.

To override a specific parameter like LimitNOFILE, execute:

sudo systemctl edit nginx.service

This command will open your default text editor (e.g., vi, nano) with an empty file. The path to this file will be something like /etc/systemd/system/nginx.service.d/override.conf. Add only the directives you wish to change. For our Nginx example, we'd add:

[Service]
LimitNOFILE=65535
LimitNPROC=2048

Save and close the file. systemctl edit will automatically notify systemd to reload its configuration. If you create the file manually, you must run sudo systemctl daemon-reload.

What happens internally?

When you use systemctl edit nginx.service, systemd performs the following actions:

1. It identifies the main unit file (e.g., nginx.service).
2. It creates a directory named nginx.service.d within /etc/systemd/system/ if it doesn't exist.
3. It then opens a file named override.conf (or a temporary name) inside this directory for editing.
4. Upon saving, the contents of override.conf are merged with the original unit file. Directives in override.conf take precedence.
5. If systemctl edit was used, it automatically triggers systemctl daemon-reload.

After modifying, apply the changes:

sudo systemctl restart nginx.service

You can verify the active configuration using systemctl status nginx.service or by inspecting the full configuration with systemctl show nginx.service and searching for the overridden parameter (e.g., grep LimitNOFILE).

sudo systemctl show nginx.service | grep -E 'LimitNOFILE|LimitNPROC'
LimitNOFILE=65535
LimitNPROC=2048

Common Override Scenarios

Modifying Environment Variables

To set or append environment variables for a service, you can use Environment= or EnvironmentFile= within the [Service] section. For instance, to add a specific timezone or a custom library path for a Java application:

sudo systemctl edit myjavapp.service

Add the following:

[Service]
Environment="TZ=America/New_York"
Environment="LD_LIBRARY_PATH=/opt/myjavapp/lib"

Note that Environment= directives are cumulative by default. If you specify multiple Environment= lines, they all contribute to the service's environment. To clear existing environment variables before adding new ones, use Environment= without any values, then define new ones.

Changing Execution Commands or Arguments

While often discouraged for major changes (as it might break package expectations), you can override ExecStart=, ExecStop=, and other execution commands. To completely replace an ExecStart command, you must first clear the existing one:

sudo systemctl edit myapp.service

Add the following:

[Service]
ExecStart=
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/custom.conf --port 8081

The empty ExecStart= line clears any previously defined ExecStart commands, ensuring only your new command is used.

Adjusting Restart Policies

For critical services, you might want a more aggressive restart policy than the default. For example, to restart a service automatically if it fails:

sudo systemctl edit database.service

Add the following:

[Service]
Restart=on-failure
RestartSec=5s

This tells systemd to restart the service if it exits with a non-zero status, and to wait 5 seconds before attempting a restart.

Disabling Vendor-Supplied Directives

Some directives can be disabled by assigning an empty value. This is particularly useful for overriding parameters that might not have a simple "off" switch. For example, to disable a specific Requires= or Wants= dependency that a vendor unit might impose (though this can have unintended side effects and should be used with caution):

sudo systemctl edit myapp.service

Add the following:

[Unit]
Requires=
Requires=another-essential-service.service

In this example, if the original unit had Requires=some-optional-dependency.service, the empty Requires= would clear it, and then Requires=another-essential-service.service would set the new dependency. This clearing mechanism applies to many array-like directives.

Inspecting Drop-In Files

You can directly view the content of your drop-in file:

cat /etc/systemd/system/nginx.service.d/override.conf

To see the combined, effective configuration of a service, including all overrides, use:

sudo systemctl cat nginx.service

This command will display the original unit file followed by the contents of any drop-in files, in the order they are applied, giving you a complete picture of the service's configuration.

Removing a Drop-In Override

To remove a drop-in override, simply delete the override file or the entire drop-in directory:

sudo rm /etc/systemd/system/nginx.service.d/override.conf

Or, to remove the entire directory and all its drop-ins:

sudo rm -r /etc/systemd/system/nginx.service.d/

After deletion, you must reload the systemd daemon and restart the service for the changes to take effect:

sudo systemctl daemon-reload
sudo systemctl restart nginx.service

The service will then revert to its original vendor-supplied configuration for the parameters that were previously overridden.

Common Pitfalls

• Incorrect Section: Ensure your override directives are placed under the correct section (e.g., [Service], [Unit], [Install]). Directives in the wrong section are ignored.
• Syntax Errors: Typos in directive names or values will prevent systemd from parsing the unit correctly. Always verify with systemctl status <unit> for parsing errors.
• Forgetting daemon-reload: If you manually create or modify drop-in files, you must run sudo systemctl daemon-reload for systemd to recognize the changes. systemctl edit handles this automatically.
• Not Restarting the Service: Reloading the daemon only tells systemd about the new configuration. The running service instance still uses the old configuration until it is restarted (systemctl restart <unit>) or reloaded (systemctl reload <unit>, if supported by the service).
• Overriding Incorrectly: Some directives are cumulative (e.g., Environment=), while others are replaced (e.g., LimitNOFILE=). For directives that are replaced, ensure you don't accidentally remove an essential setting by providing an empty value unless intended (e.g., ExecStart= followed by a new ExecStart=). Consult the systemd.unit and systemd.service man pages for specific directive behaviors.
• Multiple Drop-Ins: While override.conf is common, you can have multiple drop-in files (e.g., 00-limits.conf, 10-env.conf) in the .d directory. They are processed in lexical order. Be mindful of potential conflicts if the same directive is defined in multiple drop-ins within the same directory.