Stackvora.tech

GitHub Actions: 8 patterns for fast pipelines

Optimizing GitHub Actions pipeline execution speed is crucial for developer productivity and efficient resource utilization. Slow pipelines lead to…

Optimizing GitHub Actions pipeline execution speed is crucial for developer productivity and efficient resource utilization. Slow pipelines lead to delayed feedback, increased CI/CD costs, and a frustrating development experience. This article explores actionable patterns and features within GitHub Actions to significantly accelerate your CI/CD workflows, focusing on concrete implementations and their impact.

We'll delve into strategies ranging from intelligent caching to advanced parallelization and resource management, applicable to various project types and deployment targets.

1. Strategic Caching with actions/cache

The actions/cache action is fundamental for speeding up workflows by reusing build outputs, dependencies, or compiled artifacts from previous runs. Effective caching can drastically reduce setup and build times, especially for projects with extensive dependency trees.

Key Considerations for Caching

  • Cache Key Granularity: The cache key determines when a cache hit occurs. It should be specific enough to avoid stale caches but general enough to provide frequent hits. Common practice is to include a hash of dependency files (e.g., package-lock.json, Gemfile.lock, pom.xml) and the runner's OS.
  • Cache Path: Specify the exact directory to cache. For Node.js, this is often ~/.npm or node_modules. For Python, ~/.cache/pip. For Maven, ~/.m2.
  • Cache Restoration Keys (restore-keys): Use restore-keys to attempt restoring from less specific keys if a direct match for the primary key isn't found. This provides a fallback mechanism, increasing cache hit rates.

Example: Node.js Dependency Caching

This example caches node_modules based on the package-lock.json file.


name: Node.js CI

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Cache Node.js modules
      uses: actions/cache@v4
      with:
        path: ~/.npm # or node_modules for local caching, but ~/.npm is generally better
        key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
        restore-keys: |
          ${{ runner.os }}-node-
    - name: Use Node.js 20.x
      uses: actions/setup-node@v4
      with:
        node-version: 20.x
    - name: Install dependencies
      run: npm ci # 'ci' is faster for CI environments than 'install'
    - name: Run tests
      run: npm test

2. Parallel Execution with Matrices

Matrices allow you to run multiple variations of a job concurrently, making them ideal for testing against different environments, Node.js versions, Python versions, or running test suites in parallel shards. This dramatically reduces the total wall-clock time for a workflow.

Applying Matrices for Parallel Sharding

For large test suites, sharding tests across multiple parallel jobs can provide significant speedups. This often requires a test runner that supports sharding (e.g., Jest with --shard, RSpec with a custom sharder, or dynamic allocation with a tool like Buildkite's Test Analytics).

Example: Testing Multiple Node.js Versions & Parallel Test Shards

This workflow tests against Node.js 18 and 20, and simultaneously shards a test suite into 3 parts.


name: Parallel Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18.x, 20.x]
        shard-index: [0, 1, 2] # For 3 parallel shards
        shard-total: [3]      # Total number of shards
    steps:
    - uses: actions/checkout@v4
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v4
      with:
        node-version: ${{ matrix.node-version }}
    - name: Install dependencies
      run: npm ci
    - name: Run tests (Shard ${{ matrix.shard-index + 1 }}/${{ matrix.shard-total }})
      # Assuming your test runner supports sharding, e.g., Jest with a custom script
      run: npm run test -- --shard=${{ matrix.shard-index }}/${{ matrix.shard-total }}
      env:
        NODE_OPTIONS: --max_old_space_size=4096 # Increase memory for large tests if needed

3. Reusable Workflows for Modularity and Consistency

Reusable workflows (introduced in GitHub Actions v2.3.0) promote "Don't Repeat Yourself" (DRY) principles by allowing you to define a workflow once and call it from multiple other workflows. This reduces maintenance overhead, ensures consistency, and can simplify complex pipeline structures.

Benefits of Reusable Workflows

  • Reduced Duplication: Centralize common steps (e.g., build, deploy to staging) into a single definition.
  • Easier Maintenance: Changes to a shared process only need to be made in one place.
  • Improved Readability: Top-level workflows become simpler, delegating details to reusable components.
  • Enforced Best Practices: Standardize common security or compliance checks.

Example: Reusable Build Workflow

.github/workflows/build-component.yml (Reusable workflow):


name: Build Component

on:
  workflow_call:
    inputs:
      component-name:
        required: true
        type: string
      node-version:
        required: true
        type: string
    outputs:
      build-artifact-path:
        description: "Path to the built artifact"
        value: ${{ jobs.build.outputs.artifact-path }}

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      artifact-path: ${{ steps.build-step.outputs.artifact-path }}
    steps:
    - uses: actions/checkout@v4
    - name: Setup Node.js ${{ inputs.node-version }}
      uses: actions/setup-node@v4
      with:
        node-version: ${{ inputs.node-version }}
    - name: Install dependencies for ${{ inputs.component-name }}
      run: npm ci --prefix ${{ inputs.component-name }}
    - name: Build ${{ inputs.component-name }}
      id: build-step
      run: |
        npm run build --prefix ${{ inputs.component-name }}
        echo "artifact-path=./${{ inputs.component-name }}/dist" >> "$GITHUB_OUTPUT"
    - name: Upload artifact
      uses: actions/upload-artifact@v4
      with:
        name: ${{ inputs.component-name }}-build-output
        path: ./${{ inputs.component-name }}/dist

.github/workflows/main-app.yml (Calling workflow):


name: Main App CI

on: [push]

jobs:
  build-frontend:
    uses: ./.github/workflows/build-component.yml
    with:
      component-name: frontend
      node-version: 20.x
    secrets: inherit # If reusable workflow needs repo secrets

  deploy:
    needs: build-frontend
    runs-on: ubuntu-latest
    steps:
    - name: Download frontend artifact
      uses: actions/download-artifact@v4
      with:
        name: frontend-build-output
        path: ./dist
    - name: Deploy to staging
      run: echo "Deploying frontend from $(ls ./dist)"

4. Managing Concurrency with concurrency

The concurrency keyword allows you to group workflows or jobs and ensures that only a specified number of runs in that group can execute at any given time. This is vital for managing shared resources (e.g., deployment environments, test databases) and preventing race conditions or overloading systems.

Key Features of Concurrency Groups

  • group: A string that identifies the concurrency group. Multiple workflows or jobs can share the same group name.
  • cancel-in-progress (optional): If set to true, any in-progress job or workflow in the same concurrency group will be canceled when a new run is queued. This is particularly useful for environments where only the latest push matters (e.g., a "deploy to staging" workflow).

Example: Staging Deployment Concurrency

This ensures only one "Deploy to Staging" workflow can run at a time, and new pushes cancel previous ones.


name: Deploy to Staging

on:
  push:
    branches:
      - main

concurrency:
  group: deploy-staging-env
  cancel-in-progress: true

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: staging # Associate with a protected environment
    steps:
    - uses: actions/checkout@v4
    - name: Deploy application
      run: |
        echo "Deploying to staging environment..."
        sleep 60 # Simulate deployment time
        echo "Deployment to staging complete!"

5. Keyless Cloud Authentication with OIDC

OpenID Connect (OIDC) provides a secure, keyless way for GitHub Actions workflows to authenticate to cloud providers (AWS, Azure, GCP). Instead of long-lived secrets, OIDC issues short-lived, verifiable tokens, significantly reducing the attack surface and simplifying secret management.

OIDC Workflow

  1. GitHub Actions workflow requests an OIDC token from GitHub's OIDC provider.
  2. The cloud provider (e.g., AWS IAM, Azure AD, GCP IAM) is configured with a trust policy that validates tokens issued by GitHub's OIDC provider.
  3. The cloud provider exchanges the valid OIDC token for temporary cloud credentials.
  4. The workflow uses these temporary credentials to interact with cloud resources.

Example: Authenticating to AWS with OIDC

Requires prior setup in AWS IAM for a Role that trusts token.actions.githubusercontent.com and maps to the GitHub repository and workflow.


name: Deploy to AWS S3

on: [push]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write # Required for OIDC authentication
      contents: read  # To checkout repository contents
    steps:
    - uses: actions/checkout@v4
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsS3DeployRole # Replace with your IAM Role ARN
        aws-region: us-east-1
    - name: Sync S3 Bucket
      run: aws s3 sync . s3://your-s3-bucket-name --delete

Note: The IAM Role's trust policy for arn:aws:iam::123456789012:role/GitHubActionsS3DeployRole must look something like:


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:your-org/your-repo:ref:refs/heads/main"
        }
      }
    }
  ]
}

For stricter control, you can define the sub condition to match specific branches, environments, or even workflow names.

6. Path Filtering with on.push.paths and on.pull_request.paths

Path filtering allows you to trigger workflows only when specific files or directories are changed. This is highly effective for monorepos or projects where different components have independent CI/CD requirements, preventing unnecessary workflow runs and saving compute minutes.

Example: Component-Specific CI

This workflow only runs when changes occur within the frontend/ directory.


name: Frontend CI

on:
  push:
    paths:
      - 'frontend/**'
      - '.github/workflows/frontend-ci.yml' # Always trigger if workflow config changes
  pull_request:
    paths:
      - 'frontend/**'
      - '.github/workflows/frontend-ci.yml'

jobs:
  build-and-test-frontend:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Install and test frontend
      run: |
        cd frontend
        npm ci
        npm test

7. Self-Hosted Runners for Specialized Environments or Cost Optimization

While GitHub-hosted runners are convenient, self-hosted runners offer flexibility for specific scenarios:

  • Specialized Hardware: Require GPUs, specific CPU architectures (ARM), or large amounts of RAM/storage.
  • Private Network Access: Need to access resources within a private network without public exposure.
  • Pre-installed Dependencies: Reduce setup time by having a runner with all required software pre-installed.
  • Cost Optimization: For very high usage, self-hosting can sometimes be more cost-effective, though it incurs maintenance overhead.

To register a self-hosted runner, navigate to your repository/organization settings -> Actions -> Runners -> New self-hosted runner. Follow the on-screen instructions to download, configure, and run the runner agent. Then, specify runs-on: self-hosted in your workflow.

8. Optimized Docker Builds with BuildKit

When your workflows involve building Docker images, leveraging BuildKit features can provide substantial speed improvements. BuildKit offers parallel build steps, improved caching, and remote cache backends.

Integrating BuildKit

The docker/build-push-action action (available from GitHub Actions team) supports BuildKit and makes it easy to integrate. To enable BuildKit, you often just need to set the DOCKER_BUILDKIT environment variable to 1, or the action handles it for you.

Example: Docker Build with Cache


name: Docker Image Build

on:
  push:
    branches: [ "main" ]

jobs:
  build-docker-image:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3 # Required for advanced BuildKit features

      - name: Log in to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_TOKEN }}

      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: yourusername/your-app:latest
          cache-from: type=gha # Cache layers using GitHub Actions cache
          cache-to: type=gha,mode=max

Using cache-from: type=gha and cache-to: type=gha leverages GitHub Actions' own caching mechanism for Docker layers, ensuring consistent and fast builds across runs.

Common Pitfalls

  • Over-caching or Stale Caches: If cache keys are too broad or not invalidated correctly, you might restore outdated dependencies or build artifacts, leading to unexpected errors. Always include a hash of relevant dependency files in your cache key.
  • Insufficient Cache Cleanup: GitHub Actions caches are not infinite. Periodically review and delete unused caches via the UI or API to prevent hitting storage limits.
  • Uncontrolled Parallelism: While matrices are powerful, running too many jobs in parallel can exhaust runner limits or overload downstream services (e.g., test databases). Use concurrency to manage shared resources.
  • Verbose Logging: Excessive logging can slow down job execution and make troubleshooting harder. Focus on logging only essential information.
  • Not Using npm ci: For Node.js projects, always prefer npm ci over npm install in CI/CD environments as it's faster and ensures a clean, reproducible installation based on package-lock.json.
  • Ignoring path Filters: In monorepos, neglecting path filters leads to unnecessary runs, wasting time and resources.
  • Hardcoding Cloud Credentials: Avoid storing AWS access keys directly as GitHub Secrets. Always prioritize OIDC for secure, keyless authentication.

Back to the knowledge base · Ask the AI assistant