Pin Package Versions and Avoid Surprise Upgrades

Uncontrolled package upgrades can introduce breaking changes, downtime, and compatibility issues across your infrastructure. Pinning package versions or…

Uncontrolled package upgrades can introduce breaking changes, downtime, and compatibility issues across your infrastructure. Pinning package versions or preventing automatic updates for critical components is a fundamental practice for maintaining stability, especially in production environments. This article details common strategies for managing package versions across Debian/Ubuntu, RHEL/Rocky Linux, and Python ecosystems, emphasizing predictable deployments and simplified rollback procedures.

Understanding the Need for Version Pinning

Modern operating systems and application stacks rely on thousands of packages, many with interdependent relationships. While automatic updates can deliver security patches and new features, they often come with risks:

• Breaking Changes: A new package version might introduce API changes, remove deprecated functions, or alter default behavior, disrupting applications.
• Dependency Conflicts: Upgrading one package might pull in new versions of its dependencies that are incompatible with other installed applications.
• Regression Bugs: New versions, even bug fixes, can inadvertently introduce new bugs or performance regressions.
• Unpredictable Behavior: Production environments demand consistent behavior. Uncontrolled updates make it difficult to reproduce issues or guarantee application stability.
• Compliance: Some regulatory requirements necessitate strict control over software versions deployed.

Version pinning helps mitigate these risks by allowing explicit control over which package versions are installed and when updates occur. This strategy is critical for systems like database servers, load balancers, specific application runtimes, or custom-compiled software dependencies.

Debian/Ubuntu: Managing Packages with apt and dpkg

Debian and Ubuntu systems primarily use apt and dpkg for package management. There are several ways to pin or hold packages.

Using apt-mark hold

The simplest method to prevent a package from being upgraded or removed is to mark it as "held".

sudo apt-mark hold nginx

To verify which packages are on hold:

apt-mark showhold

When you run sudo apt upgrade or sudo apt dist-upgrade, held packages will be explicitly skipped. To unhold a package:

sudo apt-mark unhold nginx

This method prevents unintended upgrades but doesn't guarantee a specific version if the package isn't already installed, or if its dependencies require an upgrade that breaks the held package.

Pinning with APT Preferences (/etc/apt/preferences.d/)

For more granular control, especially when you need to specify a particular version from a repository or prevent an upgrade beyond a certain version, APT preferences are used. Preferences are defined in files under /etc/apt/preferences.d/ (or directly in /etc/apt/preferences, though separate files are recommended for organization).

To pin nginx to version 1.22.1-1~jammy, for example, create a file like /etc/apt/preferences.d/nginx.pref:

Package: nginx
Pin: version 1.22.1-1~jammy
Pin-Priority: 1000

The Pin-Priority value is crucial:

• Pin-Priority > 1000: Forces the installation of a version, even if it means downgrading or violating dependencies (use with extreme caution).
• Pin-Priority 990-1000: Forces the installation of a specific version from any source, overriding the default source priority.
• Pin-Priority 500 (default): Allows installation from any source, preferring newer versions.
• Pin-Priority 1-99: Prevents installation from this source unless no other version is available.

After creating or modifying a preference file, update APT's package cache:

sudo apt update

You can then simulate an upgrade to see the effect:

apt policy nginx

This will show you which version is currently installed and which candidate version would be chosen based on your pins and repository priorities.

RHEL/Rocky Linux: dnf and yum Strategies

RHEL-based distributions (including CentOS, Fedora, AlmaLinux, Rocky Linux) use dnf (or legacy yum) for package management.

Excluding Packages in dnf.conf

The simplest way to prevent specific packages from being updated is to add them to the exclude directive in /etc/dnf/dnf.conf or in a repository-specific .repo file (e.g., /etc/yum.repos.d/epel.repo).

# /etc/dnf/dnf.conf
[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False
exclude=nginx* mariadb-server*

The exclude directive accepts wildcards (*). Any package matching the patterns listed will be ignored during dnf update, dnf upgrade, or dnf install operations. This is a broad stroke and prevents any update to these packages.

Using dnf versionlock (Recommended)

For more precise control, the dnf versionlock plugin is the recommended method. It allows you to specify exact package versions to lock, preventing any further updates to those packages.

First, ensure the plugin is installed:

sudo dnf install 'dnf-command(versionlock)'

To lock a package to its currently installed version:

sudo dnf versionlock add nginx

This will add an entry to /etc/dnf/plugins/versionlock.list specifying the exact NGINX version. For example:

# This file is for versionlock plugin.
# It uses .spec-style versionlock (epoch:name-version-release.arch)
0:nginx-1.22.1-1.el9.x86_64

You can also specify a particular version to lock:

sudo dnf versionlock add 0:mariadb-server-10.5.16-1.el9.x86_64

To view locked packages:

dnf versionlock list

To remove a lock:

sudo dnf versionlock delete nginx

The dnf versionlock method is generally preferred over exclude because it's more explicit and easier to manage with specific versions rather than broad package name exclusions.

Python: Managing Dependencies with pip and Virtual Environments

For Python applications, dependency management is critical to reproducibility. Relying solely on global packages or implicit latest versions can lead to "it works on my machine" issues.

Virtual Environments

Always use virtual environments (venv or conda) for Python projects. This isolates project dependencies from the system-wide Python installation and other projects.

python3 -m venv my_project_venv
source my_project_venv/bin/activate

Pinning with requirements.txt

After installing your project's dependencies within a virtual environment, explicitly pin their versions using pip freeze:

pip install django==4.2.6 psycopg2-binary==2.9.7
pip freeze > requirements.txt

The requirements.txt file should then contain exact versions:

Django==4.2.6
psycopg2-binary==2.9.7

When deploying or setting up a new environment, install dependencies using this file:

pip install -r requirements.txt

This ensures that the exact same versions are installed everywhere.

Advanced Pinning with pip-tools or Poetry/Rye

For complex Python projects, tools like pip-tools, Poetry, or Rye offer more sophisticated dependency resolution and pinning:

• pip-tools: Allows you to define high-level dependencies in requirements.in (e.g., django~=4.2) and then compiles a fully pinned requirements.txt (including transitive dependencies) using pip-compile.
• Poetry/Rye: These tools manage dependencies via pyproject.toml, resolve compatible versions, and generate lock files (poetry.lock, requirements.lock) that precisely pin all transitive dependencies. They also handle virtual environment creation and package publishing.

Example pyproject.toml for Poetry:

[tool.poetry]
name = "my-app"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.9"
django = "~4.2.0"
gunicorn = "^20.1.0"
psycopg2-binary = "^2.9.7"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Running poetry install generates poetry.lock with all exact versions.

Integrating with Configuration Management

Regardless of the operating system or application stack, any version pinning strategy should be integrated with your configuration management system (e.g., Ansible, SaltStack, Puppet, Chef).

Instead of manually running apt-mark hold or dnf versionlock add on each server, define these states in your configuration management playbooks or states. This ensures that the desired package versions are maintained across server rebuilds, auto-scaling events, or environment provisioning.

Ansible Example (Debian/Ubuntu):

- name: Hold nginx package
  ansible.builtin.dpkg_selections:
    name: nginx
    selection: hold

- name: Install specific Python packages
  ansible.builtin.pip:
    name: "{{ item.name }}=={{ item.version }}"
    virtualenv: /opt/my_app/venv
    state: present
  loop:
    - { name: 'django', version: '4.2.6' }
    - { name: 'psycopg2-binary', version: '2.9.7' }

SaltStack Example (RHEL/Rocky Linux):

nginx_package_lock:
  pkg.versionlock:
    - name: nginx

mariadb_package_lock:
  pkg.versionlock:
    - name: mariadb-server
    - version: 10.5.16-1.el9.x86_64

python_app_dependencies:
  pip.installed:
    - pkgs:
        - django==4.2.6
        - gunicorn==20.1.0
    - bin_env: /opt/my_app/venv/bin/pip

This approach transforms package pinning from a manual, error-prone task into an automated, auditable, and reproducible process.

Common Pitfalls and Troubleshooting

• Forgetting to Update Pins: Pinned packages won't receive security updates. Regularly review and plan updates for pinned components, even if it means manual intervention.
• Dependency Hell: Pinning too many packages, especially transitive dependencies, can lead to complex dependency conflicts when you eventually try to upgrade. Aim to pin critical direct dependencies.
• Repository Changes: If a repository removes an old package version, your pins might break. Always use reliable, stable repositories or mirror them internally.
• Testing Upgrades: Before applying any package version changes to production, thoroughly test them in a staging environment that mirrors production as closely as possible.
• Lack of Documentation: Ensure that all package pinning decisions are clearly documented, including the reasons for pinning and the expected lifecycle for updates.
• apt-mark hold vs. APT Pinning: apt-mark hold is simple but less flexible. APT pinning (with Pin-Priority) allows for more nuanced control, such as preferring a specific version from a non-default repository while still allowing other packages to update normally.