Stackvora.tech

Docker BuildKit: cache mounts, secrets and inline caching

Efficient Docker image builds are critical for rapid development cycles and streamlined CI/CD pipelines. While the core principles of Dockerfile…

Efficient Docker image builds are critical for rapid development cycles and streamlined CI/CD pipelines. While the core principles of Dockerfile optimization (e.g., ordering layers by change frequency) remain relevant, modern Docker engines, specifically those leveraging BuildKit, offer advanced capabilities to dramatically reduce build times and improve security. This article delves into three powerful BuildKit features: cache mounts for package managers, secure secret injection, and remote layer caching, illustrating how to harness them effectively.

Enabling BuildKit

BuildKit is the default builder for Docker Engine 23.0 and later versions. However, in many CI/CD environments or when using older Docker daemon versions, it's prudent to explicitly enable it. This can be done by setting the DOCKER_BUILDKIT environment variable or by using docker buildx.

  • Environment Variable: For single-stage builds or when using the standard docker build command, set DOCKER_BUILDKIT=1. This makes BuildKit the backend for the default builder. 
    export DOCKER_BUILDKIT=1
docker build -t myapp:latest .
  • Docker Buildx: For more advanced scenarios, especially multi-platform builds, remote caching, or when managing multiple builders, docker buildx is the preferred interface. It uses BuildKit exclusively. 
    docker buildx create --name mybuilder --use
docker buildx build --platform linux/amd64,linux/arm64 -t myapp:latest .

    Using buildx create --use ensures subsequent buildx build commands use this new builder instance. For CI, it's common to name the builder for clarity, e.g., docker buildx create --name ci-builder --use.

Cache Mounts for Package Managers

One of the most significant contributors to long build times, especially for applications with many dependencies (Node.js, Python, Go, Java), is the re-downloading and re-installation of packages in each build. BuildKit's --mount=type=cache feature addresses this by allowing specific directories to persist across builds, acting as a build-time cache without polluting the final image layers.

Mechanism and Syntax

The cache mount type provides a persistent, isolated directory for build steps. It's automatically cleaned up by BuildKit when no longer needed or explicitly pruned. The syntax is --mount=type=cache,target=/path/in/container[,id=mycache][,sharing=locked|shared|private].

  • target: The absolute path inside the container where the cache directory will be mounted.
  • id: (Optional) A unique identifier for the cache. Useful when multiple build steps need to share the same cache or when you want to explicitly manage it.
  • sharing: (Optional) Controls how the cache is shared between concurrent builds (default is shared). locked prevents concurrent writes, private gives each build its own copy.

Practical Examples

Consider a Node.js application:

# syntax=docker/dockerfile:1.7
FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json ./
# Mount /root/.npm to cache npm packages
RUN --mount=type=cache,target=/root/.npm \
    npm ci --prefer-offline --no-audit
COPY . .
RUN npm run build

In this example, /root/.npm is the default cache directory for npm. The first time this build runs, npm ci downloads dependencies into /root/.npm. Subsequent builds, as long as package.json or package-lock.json haven't changed, will find dependencies in the cache, significantly speeding up the npm ci step. Similar patterns apply to other ecosystems:

  • Python (pip): --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
  • Go (mod): --mount=type=cache,target=/go/pkg/mod go mod download
  • Maven (Java): --mount=type=cache,target=/root/.m2 mvn dependency:resolve

The syntax=docker/dockerfile:1.7 line at the top is crucial. It tells BuildKit to use a specific Dockerfile frontend that supports these advanced features. Always specify a recent version (e.g., 1.4 or higher for cache mounts).

Secure Secret Injection with BuildKit

Handling sensitive information like API keys, private repository credentials, or database passwords during build time without embedding them into the final image layers is a common security challenge. BuildKit's --mount=type=secret provides a robust solution.

The Problem with Traditional Approaches

Historically, developers might have resorted to:

  1. Hardcoding secrets (terrible security practice).
  2. Passing secrets as build arguments (--build-arg), which are visible in image history via docker history and potentially accessible through docker inspect metadata.
  3. Baking secrets into an intermediate layer and then attempting to remove them, which often fails to fully erase the secret from the layer history.
BuildKit's secret mounts solve this by making secrets available only within the scope of a specific RUN command, exposed as temporary files.

Syntax and Usage

The syntax for defining a secret in the Dockerfile is --mount=type=secret,id=my_secret_id[,target=/path/in/container]. During the docker buildx build command, you then provide the secret content:

# syntax=docker/dockerfile:1.7
FROM alpine/git:2.40.1
WORKDIR /app
# Mount a secret named 'gh_token' to /run/secrets/gh_token
RUN --mount=type=secret,id=gh_token \
    git clone https://username:$(cat /run/secrets/gh_token)@github.com/private/repo.git .

And when building:

# Create a file containing the secret
echo "ghp_yourpersonalaccesstoken" > ./.github_token

# Build, providing the secret from the local file
docker buildx build \
  --secret id=gh_token,src=./.github_token \
  -t myapp:latest .

Alternatively, you can provide the secret directly as a string, though file-based input is often preferred for security best practices (e.g., sourcing from a secret management tool):

# Using --secret id=gh_token,env=GH_TOKEN to pass an environment variable
# Note: This variant is for cases where the secret is already in an env var.
# For file contents, src is generally safer.
export GITHUB_PAT="ghp_yourpersonalaccesstoken"
docker buildx build \
  --secret id=gh_token,env=GITHUB_PAT \
  -t myapp:latest .

The secret is exposed as a temporary file at /run/secrets/my_secret_id (or the specified target path) within the build step. It is never written to any intermediate layer or the final image. After the RUN command finishes, the secret is gone.

Note: BuildKit secrets are not the same as Docker Swarm secrets. BuildKit secrets are build-time-only and specifically designed for preventing secret leakage into image layers.

Remote Caching with BuildKit

For CI/CD pipelines, where new build agents are frequently provisioned or caches are aggressively cleaned, local BuildKit caches are insufficient. Remote caching allows build layers to be pushed to and pulled from a remote registry or other storage backends, significantly speeding up builds across different machines or fresh environments.

How it Works

BuildKit can export and import build cache metadata and content to various backends:

  • type=registry: Pushes cache manifests and layers to a Docker-compatible image registry.
  • type=local: Saves/loads cache to/from a local directory.
  • type=inline: Embeds cache metadata directly into the image manifest (less effective for large caches or non-trivial image changes).

For CI, type=registry is the most common and effective.

Registry Cache Configuration

To use a remote registry for caching, you need two main flags with docker buildx build:

  • --cache-from type=registry,ref=your_registry/your_image:cache_tag: Instructs BuildKit to pull cache metadata and layers from the specified registry reference before starting the build.
  • --cache-to type=registry,ref=your_registry/your_image:cache_tag[,mode=max|min][,compression=...]: Instructs BuildKit to push cache metadata and layers to the specified registry reference after a successful build.
    • mode=max (default): Pushes all cacheable layers. This is generally preferred for CI as it maximizes cache reuse for future builds.
    • mode=min: Pushes only layers that are part of the resulting image. Useful if you want to optimize storage but might lead to less cache hits.

Example CI Configuration (GitHub Actions)

Let's consider a CI pipeline building an application image and pushing the cache to GitHub Container Registry (GHCR).

name: CI Build and Push

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write # Required to push to GHCR
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push Docker image with remote cache
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: ghcr.io/${{ github.repository }}:latest
          cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache
          cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max

In this setup:

  • ghcr.io/${{ github.repository }}:buildcache serves as a dedicated cache image. This image itself isn't meant for deployment; its sole purpose is to store BuildKit's cache layers.
  • cache-from pulls any existing cache.
  • cache-to,mode=max pushes all cacheable build steps back to the registry, ensuring that subsequent CI runs, even on fresh runners, benefit from previous builds' work.
This approach drastically reduces build times, especially for projects with stable dependencies or complex multi-stage builds.

Alternative Cache Backends

While registry caching is common, BuildKit also supports:

# Local cache (useful for development, can be cleared with `docker builder prune`)
docker buildx build --cache-from type=local,src=./build_cache --cache-to type=local,dest=./build_cache .

# Inline cache (embeds cache metadata into the final image's manifest)
# Less effective for CI or frequent changes due to pulling full image for cache.
docker buildx build --cache-to type=inline .

Common Pitfalls and Troubleshooting

  • Forgetting # syntax=docker/dockerfile:1.x: Many BuildKit features require a specific Dockerfile frontend version. Without it, Docker falls back to an older parser, and advanced features will fail with syntax errors or be ignored. Always include this line at the very top.
  • Cache Invalidation Issues:
    • Incorrect Dockerfile layer order: Placing frequently changing files (e.g., application source code) before dependency installation steps will invalidate the cache for package installs on every change.
    • Wildcard COPY . .: If you use COPY . . too early, changes to any file in your build context will invalidate all subsequent cache layers. Be selective: COPY package.json ., then RUN npm install, then COPY . ..
    • Secrets changing: While secrets themselves don't invalidate layers, operations dependent on them might produce different outputs, breaking cache.
  • Registry Authentication for Remote Cache: Ensure your build environment (e.g., CI runner) is correctly authenticated to push and pull from the target registry (docker login or equivalent).
  • Debugging Cache Misses: Use docker buildx build --progress=plain . to get detailed output on each build step, including whether it was a cache hit or miss and its duration. This helps pinpoint slow steps. Steps marked [cached] were served from the cache.
  • Builder Management: If you're using docker buildx, remember to manage your builders. Periodically prune old build caches with docker builder prune to free up disk space. If you encounter issues, try resetting with docker buildx rm mybuilder && docker buildx create --name mybuilder --use.
  • buildx vs. build: While DOCKER_BUILDKIT=1 docker build enables BuildKit features for the default builder, docker buildx build provides a more robust and feature-rich interface for managing multiple builders, multi-platform builds, and advanced caching. For CI/CD, buildx is generally recommended.

Back to the knowledge base · Ask the AI assistant