Kubernetes / Platform intermediate 11 min read

Sigstore and Cosign: Keyless Container Image Signing and Verification

sigstore cosign container-signing supply-chain rekor keyless-signing

Sigstore and Cosign: Keyless Container Image Signing and Verification

Problem

Container image supply chains fail silently. Without signing, a Kubernetes cluster has no way to distinguish an image built from a verified commit in a controlled pipeline from one pushed directly by a compromised account or a registry MITM. Traditional signing approaches move the problem rather than solving it: long-lived GPG or RSA private keys become high-value targets, require rotation procedures nobody actually follows, and get embedded in CI secrets that leak in build logs.

Sigstore eliminates long-lived private keys from the signing workflow entirely. The keyless model ties signatures to the ephemeral identity of a CI job — specifically its OIDC token — not to a persistent key that someone must protect. The full signing event is recorded in a tamper-evident log that anyone can query.

Target systems: cosign 2.x, Rekor public instance (rekor.sigstore.dev), Fulcio public CA (fulcio.sigstore.dev), Kyverno 1.12+, OPA Gatekeeper 3.16+, Kubernetes 1.28+, GitHub Actions.

Sigstore Components

Sigstore is not a single tool. It is a collection of services and specifications that work together:

Cosign is the client. It builds and verifies signatures, attaches them to OCI registries as attached artifacts (using the sha256-<digest>.sig tag convention), and orchestrates the interaction with Fulcio and Rekor.

Fulcio is a certificate authority. Given a valid OIDC token, Fulcio issues a short-lived X.509 certificate (valid for 10 minutes) binding the signing public key to the OIDC subject — the GitHub Actions job identity, the Google service account email, or any other supported OIDC provider identity. The private key never leaves the signer; Fulcio only ever sees the public key.

Rekor is an append-only transparency log. After signing, cosign submits the signature, the certificate, and the artifact digest to Rekor. Rekor returns a signed entry timestamp (SET) proving the log accepted the record. The inclusion in the log is what enables later verification without the original certificate still being valid — by the time you verify, the 10-minute Fulcio cert is expired, but the Rekor entry proves the cert was valid at signing time.

The trust chain: you trust the Sigstore root of trust (or your own private Sigstore deployment), Fulcio inherits from it, and Rekor’s signed tree root is verifiable against it. All of this is coordinated through the Sigstore TUF (The Update Framework) root, which cosign fetches and pins automatically.

Keyless Signing Flow

The full flow for a GitHub Actions keyless signing event:

  1. The GitHub Actions runner requests an OIDC token from GitHub’s OIDC provider. The token contains claims including sub (the workflow ref), iss (https://token.actions.githubusercontent.com), and the repository/branch.
  2. Cosign generates a transient ephemeral keypair in memory.
  3. Cosign exchanges the OIDC token with Fulcio for a certificate binding the ephemeral public key to the OIDC subject.
  4. Cosign signs the image digest with the ephemeral private key and immediately discards it.
  5. Cosign uploads the signature (with the Fulcio cert embedded) to the OCI registry attached to the image digest.
  6. Cosign submits a Rekor log entry containing the signature and cert; Rekor returns a SET.
  7. Cosign embeds the SET in the OCI signature artifact.

At verification time: cosign fetches the signature artifact, extracts the Fulcio cert, verifies the cert chain against the Sigstore CA, checks the Rekor inclusion proof, verifies the image digest matches what was signed, and confirms the OIDC subject matches your expected identity (the --certificate-identity and --certificate-oidc-issuer flags).

Keyless Signing in GitHub Actions

Add id-token: write to the job permissions. Without it, GitHub will not issue an OIDC token to the runner.

name: Build and Sign

on:
  push:
    branches: [main]

jobs:
  build-sign:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Install cosign
        uses: sigstore/cosign-installer@v3

      - name: Log in to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        id: build-push
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

      - name: Sign image
        run: |
          cosign sign --yes \
            ghcr.io/${{ github.repository }}@${{ steps.build-push.outputs.digest }}

The --yes flag accepts the Rekor transparency log upload without an interactive prompt. Always sign by digest, not by tag — tags are mutable, digests are not.

To verify locally that the signature was produced by a specific GitHub Actions workflow:

cosign verify \
  --certificate-identity "https://github.com/myorg/myrepo/.github/workflows/build.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/myorg/myrepo@sha256:abc123...

The output includes the Rekor log index and the certificate subject. A non-zero exit code means verification failed — treat it as a hard failure in scripts.

Key-Based Signing When OIDC Is Unavailable

Some environments cannot use OIDC: air-gapped clusters, on-premise GitLab without OIDC configuration, or local developer workflows. Cosign supports key-based signing as a fallback.

Generate a keypair. The private key is encrypted with a passphrase:

cosign generate-key-pair
# Produces cosign.key (encrypted private key) and cosign.pub

Store cosign.key in your secrets manager (Vault, AWS Secrets Manager, or as a CI secret). Never commit it.

Sign:

COSIGN_PASSWORD=<passphrase> cosign sign --key cosign.key \
  ghcr.io/myorg/myrepo@sha256:abc123...

Verify:

cosign verify --key cosign.pub \
  ghcr.io/myorg/myrepo@sha256:abc123...

With key-based signing, Rekor upload still happens by default and can be suppressed with --tlog-upload=false — appropriate for air-gapped environments where the Rekor endpoint is unreachable. If you suppress Rekor, you lose the transparency guarantee; verification must be done against the public key directly and there is no independent audit trail.

For hardware-backed keys (YubiKey, HSM), cosign supports PKCS11 via cosign sign --key pkcs11:.... This keeps the private key in hardware while retaining the rest of the cosign workflow.

Enforcing Signatures with Kyverno

Kyverno’s ImageVerify rule integrates directly with cosign verification. The policy runs as an admission webhook: when a Pod is submitted, Kyverno fetches the image signature from the registry and verifies it before allowing the Pod to be created. See the Kyverno controller security article for controller hardening.

Keyless verification policy — verifies the image was signed by a specific GitHub Actions workflow:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: check-image-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [production, staging]
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/myrepo:*"
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/myorg/myrepo/.github/workflows/build.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Key-based verification policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images-keyed
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: check-image-signature-key
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [production]
      verifyImages:
        - imageReferences:
            - "registry.internal.corp/*"
          attestors:
            - count: 1
              entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                      -----END PUBLIC KEY-----

Set background: false on all verifyImages policies. Kyverno cannot re-verify already-running pods in background mode, so background processing is irrelevant and enabling it adds noise without safety.

The imageReferences glob must be explicit. Using * to match all images in all registries will break system pods that pull from registry.k8s.io and have no cosign signatures. Match only the registries you control.

Enforcing Signatures with OPA Gatekeeper

Gatekeeper does not have native cosign integration. External data (via externalData in Gatekeeper 3.12+) or a custom external-data provider is required. The provider queries cosign verification and returns a boolean result that the Rego policy evaluates.

Define the provider:

apiVersion: externaldata.gatekeeper.sh/v1beta1
kind: Provider
metadata:
  name: cosign-verifier
spec:
  url: https://cosign-verifier.gatekeeper-system.svc:8090/verify
  timeout: 10
  caBundle: <base64-encoded-CA>

The provider is a small HTTP service that wraps cosign verify and returns {"responses": [{"uid": "<image-digest>", "data": "true"}]} on success or an error response on failure.

Rego constraint template:

package signedimages

violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  image := container.image
  response := external_data({"provider": "cosign-verifier", "keys": [image]})
  result := response.responses[_]
  result[0] == image
  result[1] == "false"
  msg := sprintf("image %v is not signed", [image])
}

For most teams, Kyverno’s native ImageVerify integration is simpler to operate than the Gatekeeper external data provider. Use Gatekeeper if you already have a large Rego policy library and want to consolidate on a single policy engine. See the SLSA build provenance article for how signature enforcement integrates with a full SLSA level 3 pipeline.

Multi-Arch Image Signing

Multi-architecture images are stored as an OCI image index (manifest list). The index references multiple platform-specific manifests. You must sign the image index digest, not individual platform digests.

After a multi-arch build with docker buildx:

# Get the index digest explicitly
IMAGE_INDEX_DIGEST=$(crane digest ghcr.io/myorg/myrepo:latest)

cosign sign --yes \
  ghcr.io/myorg/myrepo@${IMAGE_INDEX_DIGEST}

Verify against the index:

cosign verify \
  --certificate-identity "..." \
  --certificate-oidc-issuer "..." \
  ghcr.io/myorg/myrepo@${IMAGE_INDEX_DIGEST}

If you sign individual platform manifests instead of the index, verification against the index digest will fail. Kyverno’s ImageVerify resolves the tag to a digest before verification — it will use the index digest, so sign the index.

When using docker/build-push-action in GitHub Actions, the outputs.digest is the index digest when platforms is set to multiple values. Use that output directly as shown in the workflow example above.

Rekor Log Monitoring

The Rekor transparency log is public. Any entry signed by your Fulcio certificate or containing your image digest can be found. This cuts both ways: it enables independent auditing, but it also means you should monitor for unexpected entries — unauthorized signing events that appear in the log indicate a credential or OIDC identity compromise.

The rekor-monitor tool queries the Rekor log for entries matching your identities and alerts on unexpected signatures.

Install:

go install github.com/sigstore/rekor-monitor/cmd/rekor-monitor@latest

Run with a config file specifying watched subjects:

# rekor-monitor-config.yaml
logInfoFile: /var/run/rekor-monitor/checkpoint.json
monitoredValues:
  certSubjects:
    - "https://github.com/myorg/myrepo/.github/workflows/build.yml@refs/heads/main"
  subjects:
    - "ghcr.io/myorg/myrepo"
startIndex: 0

rekor-monitor --config rekor-monitor-config.yaml \
  --rekor-server https://rekor.sigstore.dev

Run this on a schedule (cron or a Kubernetes CronJob). Output entries that were not expected — entries signed by your subject but from a workflow branch or ref you did not authorize, or entries on image digests that were never built by your pipeline.

The checkpoint file persists the last verified log index, so subsequent runs only scan new entries. Without it, rekor-monitor scans from index 0 on every run, which is expensive as the log grows.

For production monitoring, pipe the output to your SIEM. The Sigstore community maintains a reference deployment of rekor-monitor as a GitHub Actions workflow that posts alerts to a Slack webhook.

Common Pitfalls

Clock skew. Fulcio certificates are valid for 10 minutes. If the signing machine’s clock is skewed forward by more than the certificate validity window, Rekor will reject the entry because the certificate appears expired at submission time. Ensure NTP synchronization on all CI runners. GitHub-hosted runners are fine; self-hosted runners with misconfigured NTP are a common failure mode.

OIDC token expiry. In long-running builds, the OIDC token issued at job start may expire before the signing step runs. GitHub OIDC tokens are valid for 5 minutes. If your build takes longer than that before the cosign sign step, the token will be rejected by Fulcio. Structure workflows to sign immediately after push, not after additional long-running steps. If the token has expired, the error from cosign is OIDC identity token is expired — the fix is restructuring the job, not retrying with the same token.

Verifying the wrong digest. Tags are mutable. If you verify by tag (cosign verify ghcr.io/myrepo/myapp:latest) and the tag has been moved since signing, you are verifying the wrong image. Always verify by digest. In Kyverno, ImageVerify resolves the digest before verification automatically; in manual verification, use crane digest to resolve first, then pass the digest directly.

Signing after tag mutation. The inverse problem: you sign myimage:latest, then immediately overwrite the tag with a new push. Now the signature exists in the registry but points to the old digest. The new image behind latest is unsigned. Sign by digest in CI, not by tag. The build-push action’s outputs.digest gives you the immutable reference.

Registry support. Cosign attaches signatures to the registry using the OCI referrers API or the sha256-<digest>.sig tag fallback. Most registries support this: GHCR, ECR, GCR, Docker Hub, Harbor 2.x. Azure Container Registry requires the oras.artifact.manifest API which cosign uses automatically. Some older Nexus and Artifactory configurations block unknown tag formats — test your registry before enforcing signature policies in production.

Namespace mismatches in Kyverno policies. A ClusterPolicy with namespaces in the match block only applies to those namespaces. If you add a new namespace and forget to update the policy, pods in that namespace are admitted without signature verification. Use a deny-by-default approach: an additional ClusterPolicy that blocks all pods in unlisted namespaces unless they match an exception list. Audit quarterly.

Rekor unavailability. The public Rekor instance (rekor.sigstore.dev) has had outages. If cosign cannot reach Rekor during signing, the sign command fails. For production pipelines, consider mirroring to a private Rekor instance or using --rekor-url to point to a HA deployment. At verification time, Rekor unavailability is a configurable failure mode — cosign’s --insecure-ignore-tlog flag skips Rekor verification entirely, which should never be set in policy enforcement contexts.