Cache Dependencies Properly in GitHub Actions

Efficient dependency caching is crucial for optimizing GitHub Actions workflow execution times and reducing build costs. Properly configured caches…

Efficient dependency caching is crucial for optimizing GitHub Actions workflow execution times and reducing build costs. Properly configured caches prevent redundant downloads and installations of project dependencies on every run, leading to significantly faster feedback cycles. This article details best practices for implementing robust caching strategies across common ecosystems, focusing on keying mechanisms, fallback strategies, and verification.

Understanding `actions/cache` Fundamentals

The actions/cache action is GitHub's native solution for caching files and directories. It operates on a simple principle: you provide a key to identify the cache entry and a path to the files/directories to cache. If a cache entry with the exact key exists, the path is restored. Otherwise, the step executes, and upon successful completion of the job, the path is saved with the provided key.

Keying Strategies: The Core of Effective Caching

The cache key is paramount. It must uniquely represent the state of your dependencies. The most reliable method is to derive the key from a hash of your project's lock file or manifest file. This ensures that the cache is invalidated and rebuilt only when dependencies genuinely change.

Node.js/npm: package-lock.json or npm-shrinkwrap.json
Yarn: yarn.lock
pnpm: pnpm-lock.yaml
Python/pip: requirements.txt or poetry.lock
Composer/PHP: composer.lock
Maven/Java: pom.xml (combined with Maven's local repository structure)
Gradle/Java: build.gradle/build.gradle.kts and gradle/wrapper/gradle-wrapper.properties

The hashFiles function is invaluable here, creating an MD5 hash of the specified file(s). For example, a pnpm-based Node.js project would use:

key: ${{ runner.os }}-pnpm-${{ hashFiles('**/pnpm-lock.yaml') }}

Prepending runner.os to the key is a best practice, as dependencies compiled on Linux (ubuntu-latest) are often incompatible with those compiled on macOS (macos-latest) or Windows (windows-latest). Similarly, include any tool versions that affect dependency resolution, e.g., Node.js version, Python version.

Fallback Keys with `restore-keys`

While an exact key match is ideal, sometimes you need a fallback. restore-keys allow you to specify an ordered list of keys to try if the primary key doesn't yield an exact match. This is particularly useful for achieving partial cache hits when the lock file changes, but many dependencies remain the same.

Consider the pnpm example again. If pnpm-lock.yaml changes, the primary key will miss. However, we might still want to restore a cache based on a previous lock file hash or even just the OS and pnpm version, then allow pnpm to update only the changed dependencies.

- name: Cache pnpm modules
  uses: actions/cache@v3
  with:
    path: ~/.pnpm-store
    key: ${{ runner.os }}-pnpm-${{ hashFiles('**/pnpm-lock.yaml') }}
    restore-keys: |
      ${{ runner.os }}-pnpm-
      ${{ runner.os }}-

In this example:

It first attempts an exact match with the current pnpm-lock.yaml hash.
If that fails, it tries any cache starting with ${{ runner.os }}-pnpm-, effectively ignoring the lock file hash and picking the most recent pnpm cache for the OS.
If that also fails, it tries any cache starting with ${{ runner.os }}-, providing a very generic fallback.

This strategy maximizes the chances of a cache hit, even if it's a partial one, which can still save significant time.

Common Dependency Cache Paths

The path parameter specifies which directory or files to cache. These are typically the global package cache directories for your chosen ecosystem.

Node.js Ecosystem

npm: ~/.npm (or ${{ github.workspace }}/.npm if configured locally)
Yarn: ~/.cache/yarn (Yarn v1) or ${{ github.workspace }}/.yarn/cache (Yarn v2+/Berry)
pnpm: ~/.pnpm-store (or a custom path defined by pnpm config get store-dir)

# Example for pnpm
- name: Setup pnpm
  uses: pnpm/action-setup@v2
  with:
    version: 8
    run_install: false # Don't install here, let cache handle it
- name: Get pnpm store directory
  shell: bash
  run: |
    echo "PNPM_STORE_PATH=$(pnpm store path)" >> $GITHUB_ENV
- name: Cache pnpm modules
  uses: actions/cache@v3
  with:
    path: ${{ env.PNPM_STORE_PATH }}
    key: ${{ runner.os }}-pnpm-${{ hashFiles('**/pnpm-lock.yaml') }}
    restore-keys: |
      ${{ runner.os }}-pnpm-
- name: Install dependencies
  run: pnpm install --frozen-lockfile

Python Ecosystem

pip: ~/.cache/pip
Poetry: ~/.cache/pypoetry/virtualenvs (for virtual environment packages)

# Example for pip
- name: Set up Python
  uses: actions/setup-python@v4
  with:
    python-version: '3.10'
- name: Cache pip dependencies
  uses: actions/cache@v3
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}-${{ hashFiles('**/pyproject.toml') }}
    restore-keys: |
      ${{ runner.os }}-pip-
- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt

Note the use of multiple hashFiles in the key for Python. If your project uses requirements.txt but also has a pyproject.toml with build system requirements, hashing both ensures cache invalidation when either changes.

Java Ecosystem

Maven: ~/.m2/repository
Gradle: ~/.gradle/caches

# Example for Maven
- name: Set up JDK 17
  uses: actions/setup-java@v3
  with:
    distribution: 'temurin'
    java-version: '17'
- name: Cache Maven dependencies
  uses: actions/cache@v3
  with:
    path: ~/.m2/repository
    key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
    restore-keys: |
      ${{ runner.os }}-maven-
- name: Build with Maven
  run: mvn -B package --file pom.xml

For Gradle, consider caching both ~/.gradle/caches and ~/.gradle/wrapper. The wrapper cache stores the Gradle distribution itself, which can also be substantial.

# Example for Gradle
- name: Cache Gradle dependencies
  uses: actions/cache@v3
  with:
    path: |
      ~/.gradle/caches
      ~/.gradle/wrapper
    key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
    restore-keys: |
      ${{ runner.os }}-gradle-

Verifying Cache Behavior

After implementing caching, always verify its effectiveness by examining the workflow logs. The actions/cache step will output a message indicating whether a cache hit occurred:

Cache restored successfully: An exact match was found for the primary key.
Cache restored from key ${{ key }}: A match was found using one of the restore-keys. This is a partial hit.
Cache not found for input keys: [key, restore-keys...]: No cache entry was found. The job will proceed, and a new cache will be saved if the job completes successfully.

It's beneficial to run a workflow after a change to your lock file (e.g., adding a new dependency) to observe a cache miss and then a subsequent run to confirm a cache hit for the new lock file hash. Cold cache scenarios (where no cache is hit) should become rare in a stable repository with frequent commits.

Considerations and Trade-offs

Cache Size: GitHub imposes a cache size limit (currently 10GB per repository, up to 2GB per cache entry). Large dependency trees can exceed this. If you hit limits, consider caching only essential parts or splitting caches.
Cache Expiration: GitHub automatically evicts caches that haven't been accessed in 7 days or when the total repository cache size exceeds the limit. You cannot manually expire caches.
Cache Invalidation: While lock file hashing is robust, remember to include other factors in your key if they influence dependency resolution (e.g., Node.js version, Python version, JVM version). Changing these without updating the key will lead to stale caches.
Monorepos: In monorepos, use more specific hashFiles paths, e.g., packages/my-app/pnpm-lock.yaml, to avoid invalidating the cache for one application when another's dependencies change.
When to cache: Cache after dependency installation. Don't cache source code or build artifacts unless specifically intended and sized appropriately.

Common Pitfalls

Incorrect Paths: Specifying the wrong directory for path (e.g., caching node_modules directly instead of the global cache). This leads to no benefit or errors. Always cache the tool's official global cache directory.
Stale Keys: Forgetting to include critical version numbers or other environmental factors in the key, leading to incompatible restored caches. For example, not including node-version in the key when using setup-node.
Overly Generic Keys: Using a key like ${{ runner.os }}-cache offers minimal benefit as it doesn't invalidate when dependencies change. Always hash a lock file.
Not installing dependencies after restore: A cache restore only copies files. You still need to run the dependency install command (e.g., npm install, pip install -r requirements.txt). However, with a hit, these commands will run significantly faster as most dependencies are already present in the cache.
Caching node_modules directly for Node.js: While possible, it's generally less efficient than caching the global npm/yarn/pnpm cache because node_modules often contains OS-specific binaries and can become very large. The global cache stores archives that are more easily shared and less prone to platform-specific issues.