Python venvs and dependency hygiene

Managing Python project dependencies effectively is crucial for reproducible builds, consistent deployments, and avoiding "it works on my machine"…

Managing Python project dependencies effectively is crucial for reproducible builds, consistent deployments, and avoiding "it works on my machine" scenarios. Virtual environments (venvs) provide isolation, preventing conflicts between project requirements, while robust dependency management tools ensure that exact package versions are tracked and installed. This article details best practices for Python dependency hygiene, focusing on the synergy between virtual environments and advanced dependency management tools.

The core principle is simple: each Python project should operate within its own isolated environment, with all its dependencies, direct and transitive, explicitly defined and version-locked. This guarantees that installations are deterministic across development, testing, and production environments.

Understanding Python Virtual Environments (venvs)

A virtual environment is a self-contained directory tree that contains a Python interpreter, alongside its own site-packages directories, where installed third-party libraries are stored. This isolation prevents different projects from interfering with each other's dependencies.

Creating and Activating venvs

Python's built-in venv module is the standard way to create virtual environments. For example, to create a venv named .venv in your project root:

python3 -m venv .venv

The .venv directory is a common convention and is typically added to .gitignore to avoid committing transient files.

To activate the venv:

• Linux/macOS: source .venv/bin/activate
• Windows (Command Prompt): .venv\Scripts\activate.bat
• Windows (PowerShell): .venv\Scripts\Activate.ps1

Once activated, your shell prompt will usually indicate the active venv (e.g., (.venv) user@host:~/my_project$). All pip install commands will then install packages into this isolated environment.

To deactivate:

deactivate

Tools for venv Management

While python -m venv is fundamental, several tools enhance venv workflow:

• Poetry: (Covered later) Integrates venv management directly into its dependency workflow.
• Pipenv: Also manages venvs internally, abstracting their direct manipulation.
• pyenv: Manages multiple Python versions, and often integrates with virtualenv or venv creation, allowing you to easily switch between Python 3.9 and 3.10 for different projects.
• virtualenv package: A third-party alternative to venv, offering more features and compatibility with older Python versions. Often used implicitly by tools like Pipenv.

Dependency Declaration: requirements.txt vs. pyproject.toml

The traditional method for declaring direct project dependencies is requirements.txt. However, modern Python projects increasingly adopt pyproject.toml for a more structured and comprehensive approach.

requirements.txt (Direct Dependencies)

A simple text file listing direct dependencies, often with version specifiers:

Django==4.2.7
requests>=2.28.0,<3.0.0
gunicorn

To install these, you'd use:

pip install -r requirements.txt

This approach works, but requirements.txt alone doesn't typically list transitive dependencies (dependencies of your dependencies) or their exact pinned versions. This is where lock files become essential.

pyproject.toml (Modern Standard)

PEP 518 introduced pyproject.toml for build system configuration, and PEP 621 extended it for project metadata and dependencies. This is now the recommended way to declare project information and dependencies, often managed by tools like Poetry or PDM.

# pyproject.toml example
[project]
name = "my_project"
version = "0.1.0"
description = "A short description of my project."
dependencies = [
    "Django~=4.2.0",
    "requests>=2.28,<3.0",
    "gunicorn",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "flake8",
]

[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

Tools like Poetry read this file to manage dependencies and build processes.

Dependency Locking for Reproducibility

Declaring dependencies is one thing; ensuring that the exact same versions of all packages (direct and transitive) are installed every time is another. This requires a "lock file."

The Problem with Unlocked Dependencies

Consider requirements.txt with requests>=2.28.0. Today, pip install -r requirements.txt might install requests==2.28.1 and its dependency urllib3==1.26.15. Next month, a new release requests==2.29.0 might come out, potentially with a breaking change in a transitive dependency like urllib3==2.0.0. Your build could break, or tests could fail, despite your direct dependency specification remaining unchanged. This is known as "dependency drift."

Solution: Lock Files

A lock file pins all dependencies, direct and transitive, to their exact versions and often includes hashes for integrity verification. This makes installations fully deterministic.

Common lock file formats:

• requirements.txt (pip freeze style): Generated by pip freeze > requirements.txt. This can serve as a lock file, but it's a manual process and doesn't explicitly distinguish direct from transitive dependencies.
• requirements.in / requirements.txt (pip-tools): Uses an .in file for direct dependencies and generates a pinned .txt file.
• poetry.lock (Poetry): A comprehensive lock file generated and managed by Poetry.
• Pipfile.lock (Pipenv): Generated and managed by Pipenv.
• uv.lock (uv): Generated by the ultra-fast uv package manager.

Advanced Dependency Management with pip-tools, Poetry, and uv

pip-tools: The Minimalist Approach

pip-tools (pip-compile and pip-sync) provides a clean way to manage lock files using standard requirements.txt format. It requires pip-tools to be installed:

pip install pip-tools

Workflow:

1. Create requirements.in listing only your direct dependencies:

   # requirements.in
   Django~=4.2.0
   requests>=2.28,<3.0
   gunicorn
   # Optional: -r base.in for shared dependencies

2. Compile the lock file:

   pip-compile requirements.in

   This generates requirements.txt with all pinned direct and transitive dependencies and their hashes. Example output snippet:

   #
   # This file is autogenerated by pip-compile --index-url https://pypi.org/simple/
   # To update, run:
   #
   #    pip-compile requirements.in
   #
   Django==4.2.7 \
       --hash=sha256:d8c0b...
   asgiref==3.7.2 \
       --hash=sha256:4f8e6...
   requests==2.31.0 \
       --hash=sha256:4c219...
       # via -r requirements.in
   urllib3==2.0.7 \
       --hash=sha256:e766e...
       # via requests

3. Install/sync dependencies from the lock file:

   pip-sync requirements.txt

   This command intelligently installs missing packages, updates outdated ones, and uninstalls unneeded ones to match requirements.txt exactly.
4. Commit both requirements.in and requirements.txt to version control.

pip-tools is excellent for its simplicity and reliance on standard pip formats.

Poetry: Comprehensive Project Management

Poetry goes beyond just dependency locking; it's a full-fledged Python project manager that handles package installation, dependency resolution, virtual environment management, packaging, and publishing. It uses pyproject.toml.

Installation:

curl -sSL https://install.python-poetry.org | python3 -

Workflow:

1. Initialize a new project (or integrate with an existing one):

   poetry new my_project
   cd my_project
   # or for an existing project:
   # poetry init

2. Add dependencies:

   poetry add Django@~4.2 requests@^2.28 gunicorn

   This updates pyproject.toml and generates/updates poetry.lock.
3. Add development dependencies:

   poetry add --group dev pytest flake8

4. Install all dependencies (including creating a venv if none is active):

   poetry install

   This installs packages from poetry.lock. If poetry.lock doesn't exist, it resolves dependencies from pyproject.toml and creates the lock file.
5. Run commands within the project's venv:

   poetry run python manage.py runserver

6. Update dependencies (to newer allowed versions):

   poetry update

   This updates poetry.lock to the latest compatible versions.
7. Commit pyproject.toml and poetry.lock to version control.

Poetry's integrated approach simplifies many common development tasks.

uv: The Speed Demon

uv is a new, extremely fast Python package installer and resolver, written in Rust. It aims to be a drop-in replacement for pip and pip-tools commands, significantly accelerating dependency operations. It can work with requirements.txt, requirements.in, and pyproject.toml.

Installation:

pip install uv  # or via `curl -LsSf https://astral.sh/uv/install.sh | sh`

Workflow (using pyproject.toml for declaration and uv.lock for locking):

1. Define dependencies in pyproject.toml as shown in the earlier example.
2. Sync/install dependencies and generate a lock file (or update an existing one):

   uv pip install -e . --with dev --lock-file

   This command reads pyproject.toml, resolves all dependencies (including dev optional dependencies), and generates or updates uv.lock. The -e . ensures local project components are also considered.
3. Install from the lock file:

   uv pip install --locked

   This ensures only exact versions from uv.lock are installed. If uv.lock is missing, it will error out, enforcing the lock file usage.
4. Commit pyproject.toml and uv.lock.

uv also supports requirements.in/requirements.txt compilation:

# Similar to pip-compile
uv pip compile requirements.in -o requirements.txt

# Similar to pip-sync
uv pip sync requirements.txt

uv's primary advantage is its speed, making it highly attractive for CI/CD pipelines and large projects.

Choosing the Right Tool

Common Pitfalls

• Forgetting to activate venv: Packages get installed globally, leading to conflicts. Always check your prompt or use which python and which pip.
• Not committing lock files: This defeats the purpose of deterministic builds. Always commit requirements.txt (from pip-tools), poetry.lock, or uv.lock.
• Manually editing lock files: These are generated files. Edit your primary dependency declaration file (e.g., requirements.in, pyproject.toml) and regenerate the lock file.
• Ignoring .venv: While .venv should be in .gitignore, ensuring it's actually ignored is important to prevent accidental commits of large, binary virtual environments.
• Outdated dependencies in lock file: Regularly run pip-compile, poetry update, or uv pip install --with dev --lock-file to update your lock file with security patches and compatible feature releases.
• Using pip install -r requirements.txt on a pip-tools generated file: While it works, it bypasses the uninstall functionality of pip-sync, potentially leaving unused packages in the venv. Use pip-sync instead.