Run Faster Tests by Splitting and Parallelizing

Inefficient test suites are a significant bottleneck in modern CI/CD pipelines, directly impacting developer productivity and deployment frequency.…

Inefficient test suites are a significant bottleneck in modern CI/CD pipelines, directly impacting developer productivity and deployment frequency. Long-running tests delay feedback, increase context switching, and can lead to less frequent code integration. This article details practical strategies for splitting and parallelizing test execution, focusing on GitHub Actions, Jest, and Pytest, to drastically reduce CI build times.

The goal isn't just faster CI runs, but more agile development cycles. Aim for individual CI jobs to complete in under 5 minutes; while CI minutes have a cost, the cumulative impact of developer waiting time is often far greater.

Understanding Test Splitting and Parallelization

Test splitting refers to dividing your entire test suite into smaller, independent subsets. Parallelization then involves executing these subsets concurrently across multiple CI agents or processes. This approach leverages available compute resources more effectively, turning a single, monolithic test run into several smaller, simultaneous ones.

There are generally two primary strategies for splitting tests:

• Static Sharding: Pre-defining groups of tests that run together. This is simpler to implement but can lead to imbalanced shards if test execution times vary significantly.
• Dynamic Sharding: Distributing tests based on historical execution times or real-time availability. This is more complex but offers better load balancing and can adapt to changing test suites.

Parallelizing with GitHub Actions Matrix Strategy

GitHub Actions' strategy.matrix is an excellent tool for defining variations of a job. It allows you to run the same job with different input parameters, making it ideal for parallelizing test shards. Each combination defined in the matrix will run as a separate job in parallel.

Basic Matrix Sharding for Tests

To implement sharding, you'll typically define a matrix variable, say shard, and specify the total number of shards. Your test runner then uses these values to determine which subset of tests to execute.

name: CI Pipeline

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false # Allows other shards to complete even if one fails
      matrix:
        shard: [1, 2, 3, 4] # Define 4 shards for parallel execution

    env:
      TOTAL_SHARDS: 4 # Total number of shards, passed to test runner

    steps:
    - uses: actions/checkout@v4
    - name: Setup Node.js
      uses: actions/setup-node@v4
      with:
        node-version: '20'
        cache: 'npm'
    - run: npm ci

    - name: Run Shard ${{ matrix.shard }}
      run: npm test -- --shard=${{ matrix.shard }}/${{ env.TOTAL_SHARDS }}
      # The '--' passes arguments to the underlying test command (npm test)
      # For direct Jest: jest --shard=${{ matrix.shard }}/${{ env.TOTAL_SHARDS }}

In this example, four separate jobs will run concurrently, each handling one quarter of the test suite. The shard variable from the matrix and the TOTAL_SHARDS environment variable are passed to the test command.

Optimizing Shard Count

Determining the optimal number of shards requires experimentation. Too few shards leave CPU idle, too many can incur overhead from setup/teardown and context switching. A good starting point is 2-4x the number of CPU cores available on your CI runner, or empirically, the number that brings your longest shard under the 5-minute target.

Jest Test Sharding

Jest, a popular JavaScript testing framework, has built-in support for sharding as of version 27. Jest's sharding mechanism is dynamic, meaning it attempts to distribute test files based on execution time and file count to balance the load across shards.

Implementing Jest Sharding

As seen in the GitHub Actions example, Jest directly accepts a --shard argument:

jest --shard=CURRENT_SHARD_INDEX/TOTAL_SHARDS

Where CURRENT_SHARD_INDEX is 1-based (e.g., 1, 2, 3, 4) and TOTAL_SHARDS is the total number of shards. Jest uses a hashing algorithm and test file metadata to decide which files belong to each shard.

Example with npm script

If you're using npm test, ensure your script passes through the arguments correctly:

// package.json
{
  "name": "my-project",
  "version": "1.0.0",
  "scripts": {
    "test": "jest"
  },
  ...
}

Then, in your CI:

npm test -- --shard=${{ matrix.shard }}/${{ env.TOTAL_SHARDS }}

The -- separates npm arguments from the script arguments.

Jest Sharding Caveats

• Version: Sharding is officially supported from Jest 27+. Ensure your Jest version is up to date.
• Test Discoverability: Jest's sharding works best when all test files are discoverable at the start. If your tests are generated dynamically or depend on specific states, sharding might be less effective.
• Balanced Shards: While Jest aims for balance, very long-running individual test files can still cause one shard to finish significantly later than others. Consider splitting such large test files if this becomes an issue.

Pytest Parallelization and Splitting

Pytest, the ubiquitous Python testing framework, doesn't have native sharding built-in like Jest. However, it offers powerful plugins for parallelization and splitting:

• pytest-xdist: For parallelizing tests across multiple CPUs or remote hosts.
• pytest-split: For splitting tests into a fixed number of evenly distributed parts.

Pytest-xdist for Parallel Execution

pytest-xdist allows you to run tests in parallel using multiple worker processes. It supports various distribution strategies.

Installation

pip install pytest-xdist

Usage with GitHub Actions

To use pytest-xdist with the GitHub Actions matrix, you'll combine its features with a method to select specific tests for each shard. A common approach is to use the --dist=loadgroup strategy or to combine it with pytest-split.

name: Python CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ['3.9', '3.10', '3.11'] # Example: Test across Python versions
        shard: [1, 2, 3, 4] # Define 4 shards

    env:
      TOTAL_SHARDS: 4

    steps:
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install poetry # Or pip install -r requirements.txt
        poetry install # Or pip install -e . ".[dev]"
        pip install pytest pytest-xdist pytest-split # Install parallelization plugins

    - name: Run tests with pytest-split (Shard ${{ matrix.shard }})
      # pytest-split requires PYTEST_SPLIT_INDEX and PYTEST_SPLIT_TOTAL environment variables
      run: |
        PYTEST_SPLIT_INDEX=${{ matrix.shard }} \
        PYTEST_SPLIT_TOTAL=${{ env.TOTAL_SHARDS }} \
        pytest --split-by=file --durations=10 --strict-markers -n auto # -n auto leverages pytest-xdist
      # --split-by=file: splits based on files
      # --durations=10: show 10 slowest tests
      # --strict-markers: prevents typos in markers
      # -n auto: tells pytest-xdist to use as many worker processes as CPU cores

Pytest-split for Static Sharding

pytest-split is designed specifically for splitting a test suite into N parts. It can be particularly useful when you need more explicit control over which tests run in which shard or if you're not using pytest-xdist's parallelization on a single runner but rather distributing tests across multiple CI jobs (as in the GitHub Actions matrix). It works by setting environment variables.

Installation

pip install pytest-split

Usage

As demonstrated in the GitHub Actions snippet above, pytest-split relies on two environment variables:

• PYTEST_SPLIT_INDEX: The 1-based index of the current shard (e.g., 1, 2, 3).
• PYTEST_SPLIT_TOTAL: The total number of shards (e.g., 4).

When these are set, pytest-split intercepts the test collection phase and filters tests according to the specified shard. You can combine it with pytest -n auto to also parallelize within each shard if your CI runner has multiple cores.

Pytest Sharding Strategies and Trade-offs

For most scenarios, combining GitHub Actions matrix with pytest-split for coarse-grained sharding across CI jobs and pytest -n auto (powered by pytest-xdist) for fine-grained parallelism within each job offers the best balance of simplicity and efficiency.

Common Pitfalls

• Unbalanced Shards: If one shard consistently takes much longer than others, it becomes the bottleneck, negating much of the benefit. Investigate long-running tests within that shard. Consider dynamic sharding solutions or manually adjusting test file distribution if possible.
• Shared Resources/State: Parallelizing tests that share or modify global state (e.g., databases, file systems) without proper isolation can lead to flaky tests or race conditions. Ensure tests are truly independent.
• CI Runner Resource Limits: Running too many parallel jobs or shards on CI runners with limited CPU or memory can degrade performance or cause failures, even if the tests themselves are optimized. Monitor CI agent resource utilization.
• Setup/Teardown Overhead: Each parallel job or worker might incur setup (e.g., dependency installation) and teardown overhead. For very fast tests, this overhead can sometimes outweigh the benefits of parallelization.
• Test Dependencies: If tests in one shard depend on the successful completion of tests in another, parallelization will break the logic. Ensure your tests are atomic and independent.