Stackvora.tech

Sign and Verify Artifacts With Sigstore

Securing the software supply chain has become a critical concern for organizations deploying applications. Attack vectors have shifted from traditional…

Securing the software supply chain has become a critical concern for organizations deploying applications. Attack vectors have shifted from traditional network perimeter defenses to vulnerabilities introduced during development, build, and deployment processes. Sigstore provides a suite of tools designed to address these challenges by enabling cryptographic signing of software artifacts, verifying their authenticity, and ensuring their integrity throughout their lifecycle. This article details how to integrate Sigstore, specifically using cosign, into CI/CD pipelines for keyless signing and verification, and how to enforce these policies in a Kubernetes environment.

The core components of Sigstore include Fulcio, Rekor, and Cosign. Fulcio acts as a root Certificate Authority (CA) that issues short-lived X.509 certificates to identified entities (developers, CI/CD systems) based on OIDC authentication. Rekor is an immutable, tamper-resistant transparency log that records all signing events and associated metadata. Cosign is the CLI tool used to generate, sign, verify, and manage signatures and attestations for container images and other artifacts.

Keyless Signing with OIDC and Cosign

Traditional artifact signing often relies on long-lived private keys, which present significant management overhead, including secure storage, rotation policies, and revocation procedures. Sigstore's keyless signing mechanism eliminates this by leveraging OpenID Connect (OIDC) identities. Instead of static keys, a short-lived signing certificate is issued by Fulcio based on a successful OIDC authentication against an identity provider (IdP) like GitHub, Google, or your organization's Okta instance. This certificate is then used to sign the artifact, and the signature, certificate, and a Merkle tree hash of the artifact are recorded in Rekor.

Prerequisites for Keyless Signing

  • Cosign CLI: Version 2.0.0 or later. Install via Homebrew (brew install cosign), Go (go install github.com/sigstore/cosign/cmd/cosign@latest), or download from GitHub releases.
  • OIDC Provider: Access to an OIDC provider (e.g., GitHub Actions, GitLab CI, Google Cloud Identity, AWS IAM).
  • Registry Access: Write access to an OCI-compatible registry (e.g., Docker Hub, Google Container Registry, JFrog Artifactory).

Integrating Keyless Signing in CI/CD

The process involves authenticating with an OIDC provider, obtaining a token, and then using cosign sign. Most CI/CD platforms can natively integrate with OIDC to provide a federated identity token. Here's an example for GitHub Actions:

name: Sign Artifacts with Sigstore

on:
  push:
    branches:
      - main

jobs:
  build-and-sign:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write # Required for OIDC authentication

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Cosign
        uses: sigstore/cosign-installer@v3.5.0 # Use the latest stable version
        with:
          cosign-release: 'v2.2.3' # Specify a precise Cosign version

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push image
        id: build-image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}/my-app:latest

      - name: Sign the container image
        env:
          COSIGN_EXPERIMENTAL: "true" # Required for some experimental features or older Cosign versions
        run: |
          cosign sign --yes ghcr.io/${{ github.repository }}/my-app:latest

      - name: Verify the signature immediately (optional, for pipeline self-check)
        run: |
          cosign verify ghcr.io/${{ github.repository }}/my-app:latest

The id-token: write permission is crucial for GitHub Actions to issue an OIDC token that Cosign can use to authenticate with Fulcio. The cosign sign --yes command performs the keyless signing process:

  1. Cosign requests an OIDC token from the CI/CD environment.
  2. It then presents this token to Fulcio.
  3. Fulcio verifies the token with the OIDC provider and, if valid, issues a short-lived X.509 certificate to Cosign. The certificate subject will contain the OIDC identity (e.g., https://github.com/org/repo/.github/workflows/workflow.yaml@refs/heads/main).
  4. Cosign uses this certificate to sign the image manifest digest.
  5. The generated signature and the certificate are uploaded to Rekor. Cosign also automatically uploads the signature to the OCI registry alongside the image (typically as a .sig or .att suffix on the image digest).

The --yes flag is important for non-interactive environments as it bypasses confirmation prompts.

Verifying Artifacts in Deployment Pipelines

Verification is as critical as signing. Before deploying an artifact to production, its signature must be validated to ensure it originates from a trusted source and has not been tampered with. This typically happens in a deployment pipeline or an admission controller.

Cosign Verification Process

# Verify an image signed with keyless (Fulcio)
cosign verify ghcr.io/myorg/my-app:latest

# Verify an image signed with keyless and specify a Rekor URL if not using public instance
# cosign verify --rekor-url https://rekor.my-private-instance.com ghcr.io/myorg/my-app:latest

# Verify an image and check the OIDC identity used to sign it
# This is crucial for policy enforcement!
cosign verify --certificate-identity "https://github.com/myorg/my-repo/.github/workflows/build-and-sign.yml@refs/heads/main" \
              --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
              ghcr.io/myorg/my-app:latest

# Verifying a specific digest (more secure for production)
cosign verify myregistry.com/my/image@sha256:abcd...efgh

When you run cosign verify without specifying a public key, Cosign performs the following steps:

  1. It retrieves the signature(s) and associated certificate(s) from the OCI registry.
  2. It queries Rekor to find the entry for the artifact's digest and its signature.
  3. It verifies that the certificate used to sign the artifact was issued by Fulcio and is valid according to the Rekor entry.
  4. It then verifies the signature against the artifact's digest using the public key embedded in the certificate.
  5. Finally, it checks if the Rekor entry itself is valid and untampered (by verifying Rekor's own signature against the Sigstore root key).

For stricter policy enforcement, always include --certificate-identity and --certificate-oidc-issuer. These flags ensure that the signing certificate was issued to a specific OIDC identity (e.g., a particular GitHub Actions workflow) and from a trusted OIDC issuer, preventing unauthorized identities from signing artifacts.

Enforcing Signed Images in Kubernetes with Admission Controllers

Verifying signatures in a deployment script is a good first step, but for robust supply chain security, policy enforcement should be automated at the Kubernetes cluster level. This is achieved using mutating and validating admission controllers. Tools like Kyverno or OPA Gatekeeper can consume Cosign policies to block the deployment of unsigned or improperly signed images.

Using Kyverno for Cosign Policy Enforcement

Kyverno is a Kubernetes native policy engine. It can validate image signatures and attestations directly within admission requests. Here's a basic Kyverno ClusterPolicy to enforce that all images pulled into a cluster must be signed by a specific OIDC identity and issuer.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-image-signature
  annotations:
    policies.kyverno.io/category: Software Supply Chain Security
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      This policy enforces that container images must be signed using Cosign
      and verified against a specific OIDC identity and issuer.
spec:
  validationFailureAction: Enforce # Or Audit for testing
  rules:
    - name: verify-image-signature-and-identity
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
        - image: "*" # Apply to all images in the Pod spec
          keyless:
            # Fulcio root certificate, typically sourced from a ConfigMap or a direct URL
            # The public Sigstore root for Fulcio is often pre-configured in Kyverno or
            # can be referenced from a public URL.
            # Refer to Kyverno documentation for the exact root certificate handling (e.g., `sigstore.dev/policy-controller/ca-cert-configmap`)
            ctlog:
              rekorUrl: https://rekor.sigstore.dev # Public Rekor instance
            attestations:
              # For more advanced SLSA attestation verification, specify predicates
              # - predicate:
              #     json:
              #       # Example: Check for a specific builder ID in the attestation
              #       # You would need to sign with --attest and generate SLSA provenance first
              #       - path: "$.predicate.builder.id"
              #         operator: Equals
              #         value: "https://github.com/tektoncd/chains/tekton-chains-git-set-digest@sha256:..."
            # Policy rules for the certificate itself
            # This ensures only images signed by your specific CI/CD workflow are allowed
            policy:
              - subject:
                  # Replace with your actual GitHub Actions workflow identity
                  # For GitHub Actions, this pattern is common:
                  # https://github.com/ORG/REPO/.github/workflows/WORKFLOW.yml@refs/heads/BRANCH
                  # For GitLab CI:
                  # https://gitlab.com/GROUP/PROJECT/-/blob/main/.gitlab-ci.yml@...
                  # For Google Cloud Build:
                  # https://cloudbuild.googleapis.com/google.com/projects/PROJECT_NUMBER/locations/global/triggers/TRIGGER_ID
                  # Use `cosign verify --json ` to inspect the identity field
                  # or inspect the Fulcio cert directly from Rekor.
                  identity: "https://github.com/myorg/my-repo/.github/workflows/build-and-sign.yml@refs/heads/main"
                  issuer: "https://token.actions.githubusercontent.com"
                  # OIDC Issuer for GitHub Actions. Adjust for other providers.
                  # https://accounts.google.com for Google
                  # https://sts.amazonaws.com for AWS
                  # https://gitlab.com for GitLab
                  # Your custom OIDC provider URL

Deploying this Kyverno policy will prevent any Pod from being created if its container images are not signed according to the specified OIDC identity and issuer. The validationFailureAction: Enforce means the admission request will be denied. Set it to Audit during initial testing to see policy violations without blocking deployments.

Note: Kyverno's Sigstore integration automatically fetches the necessary Sigstore root certificates for Fulcio and Rekor, simplifying the setup considerably compared to manual key management.

SLSA Attestations with Cosign

Beyond simple signature verification, organizations need assurance about how an artifact was built. Supply-chain Levels for Software Artifacts (SLSA) is a framework that specifies a set of security guidelines and controls to prevent tampering and improve the integrity of software. Cosign can generate and attach SLSA-compliant build attestations to images, recording details like the source code repository, build commands, and dependencies.

Generating SLSA Provenance

The cosign generate-provenance command can create SLSA provenance in-toto attestations. This often integrates with build systems like Tekton Chains, GitHub Actions' OIDC provider, or cloud build services to automatically capture build details.

# Example: Generate SLSA provenance for a GitHub Actions build
# This typically happens as part of a dedicated GitHub Action, e.g., sigstore/gh-action-helm-charts-sigstore
# Or manually with `cosign attest` after building:

# Build the image and push
docker build -t ghcr.io/myorg/my-app:latest .
docker push ghcr.io/myorg/my-app:latest

# Then generate attestation (this example is simplified, usually requires a builder integration)
# A more realistic scenario involves a builder like Tekton Chains or GitHub Actions' OIDC
# which automatically generates the JSON payload.
# Here's a conceptual manual step:
echo '{"_type":"https://in-toto.io/Statement/v0.1","predicateType":"https://slsa.dev/provenance/v0.2","subject":[{"name":"my-app","digest":{"sha256":"<IMAGE_DIGEST>"}}],"predicate":{"builder":{"id":"https://github.com/myorg/my-repo/actions/runs/BUILD_RUN_ID"},"invocation":{"parameters":{"build_command":["docker","build","-t","ghcr.io/myorg/my-app:latest","."]},"environment":{"git_commit":"abcdef"}},"materials":[{"uri":"git+https://github.com/myorg/my-repo","digest":{"sha1":"abcdef"}}],"metadata":{"buildStartedOn":"2023-01-01T12:00:00Z"}}}' > provenance.json

# Sign and attach the attestation
COSIGN_EXPERIMENTAL=1 cosign attest --yes --predicate provenance.json ghcr.io/myorg/my-app:latest

The COSIGN_EXPERIMENTAL=1 environment variable is often needed for newer Cosign features, including certain attestation types. The --predicate flag points to a JSON file containing the attestation payload. This payload typically follows the in-toto spec and the SLSA provenance predicate type. These attestations are also stored in Rekor and associated with the image.

Verifying SLSA Attestations

Verification of attestations ensures that the build process itself met certain security standards. This is critical for achieving higher SLSA levels.

# Retrieve and display attestations
cosign attest --type slsaprovenance ghcr.io/myorg/my-app:latest

# Verify both signature and SLSA attestation
cosign verify-attestation --type slsaprovenance \
                          --certificate-identity "https://github.com/myorg/my-repo/.github/workflows/build-and-sign.yml@refs/heads/main" \
                          --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
                          ghcr.io/myorg/my-app:latest

For automated policy enforcement, Kyverno can verify specific fields within SLSA attestations. For instance, you could enforce that the builder ID in the attestation matches a known, trusted builder or that a specific source repository was used.

Common Pitfalls

  • Missing id-token: write permission: This is a common error in GitHub Actions, preventing OIDC token generation. The cosign sign command will fail with authentication errors.
  • Incorrect OIDC Identity/Issuer in Verification: If the --certificate-identity or --certificate-oidc-issuer in cosign verify (or Kyverno policy) does not exactly match what was recorded in Fulcio/Rekor during signing, verification will fail. Double-check these values. Use cosign verify --json <image> to inspect the actual certificate identity and issuer.
  • Firewall Issues: Ensure your CI/CD environment and Kubernetes cluster can reach fulcio.sigstore.dev, rekor.sigstore.dev, and your OIDC provider's endpoints (typically TCP port 443).
  • Cosign Version Mismatches: Different Cosign versions can sometimes have slightly different behaviors or flag requirements (e.g., COSIGN_EXPERIMENTAL). Always test with the specific version you plan to use. As of Cosign v2.0+, COSIGN_EXPERIMENTAL is generally not required for basic keyless signing but might be for newer attestation features.
  • Registry Authentication: Ensure Cosign has appropriate credentials to push signatures to your OCI registry. This usually means configuring Docker login or environment variables for registry authentication.
  • Slow Rekor Propagation: In rare cases, there might be a slight delay (seconds) between a successful sign operation and the Rekor entry being fully queryable globally. This is usually not an issue but can rarely cause verification failures immediately after signing.

Back to the knowledge base · Ask the AI assistant