Enforcing Software Supply Chain Security Policies at Deploy Time
Problem
CI pipelines enforce a great deal: linting, unit tests, SAST scans, vulnerability checks, image signing. The temptation is to treat a passing pipeline as a security guarantee. It is not. CI is advisory. An attacker who gains access to a CI runner, a registry credential, or a pipeline YAML file can bypass, modify, or disable every check that lives inside the pipeline. A malicious image can be pushed directly to the registry. A signed image can be replaced by an unsigned one after the signing step completes. A workflow file can be edited to skip the vulnerability scan.
The deploy-time gate is different. Kubernetes admission control runs inside the cluster, outside the pipeline’s control plane. Every Pod creation request — regardless of origin, regardless of whether it came from a GitOps controller, a manual kubectl apply, a Helm release, or a compromised CI system — must pass through the admission webhook before a container starts. This makes it the only control that cannot be bypassed by attacking the pipeline.
The gaps that only deploy-time policy can close:
- A developer pushes an image directly to the registry with
docker push, bypassing CI entirely. - A CI job is modified to skip signing; the unsigned image reaches the registry.
- An image is signed correctly but then replaced in the registry by an attacker with registry write access. The replaced image has no valid signature.
- A GitOps operator syncs a manifests change that points to a different (older, unpatched) image digest than the one tested in CI.
- An emergency rollback targets an image that was never scanned for recent CVEs.
Target systems: Kubernetes 1.28+, Sigstore policy-controller 0.9+, Kyverno 1.12+, OPA Gatekeeper 3.15+, slsa-verifier 2.4+, Cosign 2.4+, Syft 1.x, Grype 0.77+.
Threat Model
- Adversary 1 — Pipeline bypass: An attacker with CI runner access or repository write access disables signing and vulnerability scanning steps, then pushes a malicious image to the registry. Without admission-time verification, the image deploys to production.
- Adversary 2 — Registry substitution: An attacker with registry write access replaces a legitimately signed image with a malicious one after the image has passed CI. The image tag still resolves, but the digest (and therefore the signature) no longer matches.
- Adversary 3 — Dependency confusion / base image compromise: A publicly available base image used in production is compromised after the last CI scan. A new pod using an older manifest pulls the compromised image. Without deploy-time CVE gating, the compromised image runs.
- Adversary 4 — Banned package introduction: A developer introduces a dependency that is on the organization’s banned package list (known-malicious, license-incompatible, or from a sanctioned vendor). The SBOM attestation includes the banned package; an SBOM-aware admission check blocks the deployment.
- Adversary 5 — Provenance downgrade: An image is built outside the approved build system and achieves only SLSA Level 1 (or none). A policy that requires SLSA Level 3 blocks it; without that policy, it reaches production.
- Access level: Adversaries 1–2 have CI or registry credentials. Adversary 3 operates through the public package ecosystem. Adversaries 4–5 operate through the normal development workflow.
- Objective: Run unauthorized, unverified, or vulnerable code in production.
- Blast radius: Container escape, data exfiltration, lateral movement to adjacent cluster workloads.
What to Enforce at Deploy Time
Before configuring any specific tool, establish the set of properties that every image must prove before it is allowed to run. These properties map directly to policy rules:
| Property | Evidence type | Why it belongs here |
|---|---|---|
| Image was signed by the expected build pipeline | Cosign signature in OCI registry | Proves image origin; blocks direct registry pushes |
| SLSA provenance meets minimum level (≥ L2) | SLSA provenance attestation | Proves the build was non-falsifiable and auditable |
| SBOM is attached and contains no banned packages | SPDX/CycloneDX attestation | Enables real-time license and supply chain policy enforcement |
| Vulnerability scan shows no critical CVEs | Grype/Trivy scan attestation | Last line of defence before execution |
| Image was built from an approved base image | Provenance fromImage field or base image attestation |
Prevents stale or unapproved base images reaching production |
Each property should be enforced independently. A policy that requires all five is stronger than a policy that requires only signing: an attacker who can sign their malicious image (e.g. by using a stolen signing key) is still blocked by the CVE check and the provenance level requirement.
Configuration
Step 1: Sigstore Policy-Controller — ClusterImagePolicy
The Sigstore policy-controller is a dedicated Kubernetes admission controller for supply chain policy. It exposes a ClusterImagePolicy CRD that maps image reference patterns to signing and attestation requirements.
Install the policy-controller via Helm:
helm repo add sigstore https://sigstore.github.io/helm-charts
helm repo update
helm install policy-controller sigstore/policy-controller \
--namespace cosign-system \
--create-namespace \
--set policyController.failurePolicy=Fail
Create a namespace-level enforcement label. The policy-controller only enforces on namespaces that opt in:
kubectl label namespace production \
policy.sigstore.dev/include=true
Define a ClusterImagePolicy that requires keyless signing from the expected workflow:
# cluster-image-policy-production.yaml
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
name: production-signing-policy
spec:
images:
# Match all images in the organization's registry.
- glob: "ghcr.io/myorg/**"
authorities:
- keyless:
url: https://fulcio.sigstore.dev
identities:
# Only accept signatures from the production build workflow on main.
- issuer: "https://token.actions.githubusercontent.com"
subjectRegExp: "^https://github\\.com/myorg/[^/]+/\\.github/workflows/build\\.yml@refs/heads/main$"
# Every signature must have a Rekor transparency log entry.
ctlog:
url: https://rekor.sigstore.dev
# Also require the SLSA provenance attestation.
attestations:
- name: slsa-provenance
predicateType: "https://slsa.dev/provenance/v1"
policy:
type: cue
data: |
// Minimum SLSA Level 2: hosted runner, non-falsifiable provenance.
import "list"
predicateType: "https://slsa.dev/provenance/v1"
predicate: {
buildDefinition: {
buildType: =~ "^https://slsa\\.dev/container-based-build/v0\\."
}
}
Per-namespace overrides are possible by creating additional ClusterImagePolicy resources scoped to non-production namespaces with relaxed requirements. This supports a progressive rollout: enforce strictly in production first, then extend to staging.
Step 2: Kyverno Policies for Supply Chain
Kyverno provides a flexible policy engine that complements the policy-controller with richer condition logic and the ability to enforce SBOM content.
Install Kyverno:
helm install kyverno kyverno/kyverno \
--namespace kyverno \
--create-namespace \
--set admissionController.replicas=3
Policy 1: Require signed image with digest pinning.
Kyverno’s verifyImages rule automatically mutates the pod’s image reference to use the verified digest, preventing tag-based attacks after signing:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-signed-image
spec:
validationFailureAction: Enforce
background: false
rules:
- name: check-image-signature
match:
any:
- resources:
kinds: [Pod]
namespaces: [production, staging]
verifyImages:
- imageReferences:
- "ghcr.io/myorg/*"
mutateDigest: true # Pin the image to the verified digest.
required: true
attestors:
- count: 1
entries:
- keyless:
subject: "https://github.com/myorg/*/github/workflows/build.yml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"
rekor:
url: "https://rekor.sigstore.dev"
Policy 2: Require SBOM attestation.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-sbom-attestation
spec:
validationFailureAction: Enforce
background: false
rules:
- name: check-sbom-attached
match:
any:
- resources:
kinds: [Pod]
namespaces: [production]
verifyImages:
- imageReferences:
- "ghcr.io/myorg/*"
attestors:
- count: 1
entries:
- keyless:
subject: "https://github.com/myorg/*/github/workflows/build.yml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"
attestations:
- predicateType: "https://spdx.dev/Document"
attestors:
- count: 1
entries:
- keyless:
subject: "https://github.com/myorg/*/github/workflows/build.yml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"
Policy 3: Require SLSA provenance at level 2 or higher.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-slsa-provenance
spec:
validationFailureAction: Enforce
background: false
rules:
- name: check-slsa-provenance
match:
any:
- resources:
kinds: [Pod]
namespaces: [production]
verifyImages:
- imageReferences:
- "ghcr.io/myorg/*"
attestors:
- count: 1
entries:
- keyless:
issuer: "https://token.actions.githubusercontent.com"
subject: "https://github.com/myorg/*/github/workflows/build.yml@refs/heads/main"
attestations:
- predicateType: "https://slsa.dev/provenance/v1"
conditions:
- all:
# Builder must be the SLSA GitHub Generator at a pinned version.
- key: "{{ predicate.builder.id }}"
operator: Equals
value: "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v2.0.0"
Step 3: OPA Gatekeeper Constraint for Image Signing
For organizations already running OPA Gatekeeper, a custom ConstraintTemplate enforces image signing requirements without deploying an additional controller:
# constraint-template-signed-images.yaml
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: requiresignedimages
spec:
crd:
spec:
names:
kind: RequireSignedImages
validation:
openAPIV3Schema:
type: object
properties:
allowedRegistries:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package requiresignedimages
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
image := container.image
# Image must include a digest (sha256:...) — unsigned images
# typically reference only a tag.
not regex.match("@sha256:[a-f0-9]{64}$", image)
msg := sprintf(
"Container image '%v' must reference a digest, not a mutable tag. Sign the image and pin to digest.", [image]
)
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
image := container.image
# Image must come from an approved registry.
not startswith(image, "ghcr.io/myorg/")
not startswith(image, "registry.internal.myorg.com/")
msg := sprintf(
"Container image '%v' is not from an approved registry.", [image]
)
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireSignedImages
metadata:
name: require-signed-images-production
spec:
enforcementAction: deny
match:
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces: ["production", "staging"]
parameters:
allowedRegistries:
- "ghcr.io/myorg/"
- "registry.internal.myorg.com/"
Note: OPA Gatekeeper validates the image reference at admission time but does not itself verify the cryptographic signature. Pair it with the Sigstore policy-controller (which does cryptographic verification) and use Gatekeeper for additional structural checks (registry allowlisting, digest pinning enforcement, label requirements).
Step 4: slsa-verifier in CD Scripts
For non-Kubernetes deployments — serverless functions, bare-metal, VM-based workloads — the slsa-verifier CLI provides standalone provenance verification that can run as a pre-deployment gate inside any CD script:
#!/usr/bin/env bash
# deploy-gate.sh — run before any deployment; exits non-zero on policy failure.
set -euo pipefail
IMAGE_DIGEST="${1:?Usage: $0 <image-digest>}"
EXPECTED_REPO="github.com/myorg/myapp"
EXPECTED_BUILDER="https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v2.0.0"
echo "==> Verifying SLSA provenance for ${IMAGE_DIGEST}"
slsa-verifier verify-image \
--source-uri "${EXPECTED_REPO}" \
--builder-id "${EXPECTED_BUILDER}" \
--print-provenance \
"ghcr.io/myorg/myapp@${IMAGE_DIGEST}"
echo "==> Verifying image signature"
cosign verify \
--certificate-identity-regexp "^https://github\\.com/myorg/myapp/\\.github/workflows/build\\.yml@refs/heads/main$" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
"ghcr.io/myorg/myapp@${IMAGE_DIGEST}"
echo "==> All supply chain gates passed. Proceeding with deployment."
Integrate this gate into Argo CD via a PreSync hook:
# argocd-presync-hook.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: supply-chain-gate
annotations:
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
template:
spec:
restartPolicy: Never
containers:
- name: gate
image: ghcr.io/myorg/deploy-gate-tools:latest
command: ["/scripts/deploy-gate.sh"]
args: ["$(IMAGE_DIGEST)"]
env:
- name: IMAGE_DIGEST
valueFrom:
configMapKeyRef:
name: release-metadata
key: image_digest
The PreSync hook runs before Argo CD applies any manifests. A non-zero exit blocks the sync.
Step 5: SBOM Analysis — Checking for Banned Packages
Attaching an SBOM attestation proves the SBOM exists; it does not automatically check the SBOM’s contents. Add a banned-package check as a deployment gate step:
#!/usr/bin/env bash
# check-sbom-banned.sh — extract SBOM from attestation and check for banned packages.
set -euo pipefail
IMAGE_REF="${1:?Usage: $0 <image-ref>}"
BANNED_PACKAGES_FILE="${2:-/etc/policy/banned-packages.txt}"
echo "==> Extracting SBOM attestation for ${IMAGE_REF}"
cosign verify-attestation \
--certificate-identity-regexp "^https://github\\.com/myorg/.*@refs/heads/main$" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
--type spdxjson \
"${IMAGE_REF}" \
| jq -r '.payload' \
| base64 -d \
| jq -r '.predicate.packages[].name' \
> /tmp/sbom-packages.txt
echo "==> Checking for banned packages"
VIOLATIONS=0
while IFS= read -r banned; do
if grep -qxF "${banned}" /tmp/sbom-packages.txt; then
echo "BLOCKED: banned package '${banned}' found in SBOM"
VIOLATIONS=$((VIOLATIONS + 1))
fi
done < "${BANNED_PACKAGES_FILE}"
if [[ "${VIOLATIONS}" -gt 0 ]]; then
echo "Supply chain gate FAILED: ${VIOLATIONS} banned package(s) found."
exit 1
fi
echo "==> SBOM check passed. No banned packages found."
The banned packages file (/etc/policy/banned-packages.txt) is managed as a ConfigMap in the cluster, version-controlled in a policy repository, and updated by the security team independently of application code. This separation means the security team can add a package to the banned list without modifying any application pipeline.
Step 6: Progressive Enforcement — Audit Mode Then Enforce Mode
Switching directly to enforce mode in production causes outages if any existing workloads lack the required attestations. Use audit mode to measure the compliance gap first.
Kyverno audit mode:
spec:
validationFailureAction: Audit # Log violations; do not block.
# Query Kyverno policy reports to measure the violation rate.
kubectl get policyreport -A -o json \
| jq '[.items[].results[] | select(.result == "fail")] | length'
# List all failing pods by policy.
kubectl get policyreport -A -o json \
| jq -r '.items[] | .namespace as $ns | .results[] | select(.result == "fail") | "\($ns)/\(.resources[0].name): \(.policy)/\(.rule)"'
Sigstore policy-controller audit mode:
Add the warn label instead of include:
kubectl label namespace production \
policy.sigstore.dev/warn=true # Warn on violation; do not block.
Transition checklist:
- Deploy all policies in audit/warn mode.
- Run for one full deployment cycle (at least one week in production).
- Query violation reports; remediate each failing image.
- Set a compliance threshold (e.g. zero violations for 72 hours).
- Switch policies to enforce mode namespace by namespace, starting with non-production.
- Monitor pod admission error rates in the cluster for 24 hours after each namespace switches.
Step 7: Emergency Rollbacks — Pre-Signed Rollback Images
An enforcement policy that blocks unsigned images creates an operational risk: if a rollback is needed urgently and the rollback target image lacks a valid signature, the rollback itself is blocked. Prevent this by pre-signing rollback images at build time.
The principle: every image that might ever need to be deployed — including images intended for rollback — must pass through the signing pipeline. Never rely on the ability to bypass policy as a rollback mechanism.
Implementation:
# In the build workflow, tag the previous release as a rollback candidate and sign it.
- name: Tag and sign rollback candidate
run: |
PREVIOUS_DIGEST=$(curl -sH "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
"https://api.github.com/repos/${{ github.repository }}/releases/latest" \
| jq -r '.tag_name')
# Retag the previous release image as rollback.
cosign copy \
ghcr.io/${{ github.repository }}:${PREVIOUS_DIGEST} \
ghcr.io/${{ github.repository }}:rollback
# The copy preserves all signatures and attestations. No re-signing needed.
# Verify the rollback tag has a valid signature.
cosign verify \
--certificate-identity-regexp "^https://github\\.com/${{ github.repository }}/.*@refs/heads/main$" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
ghcr.io/${{ github.repository }}:rollback
For GitOps-based rollbacks, maintain a rollback-manifest.yaml in the repository that references a known-good, pre-signed digest. Reverting the GitOps repository to a previous commit automatically targets the pre-signed image.
Never create a break-glass bypass that disables admission-time verification for rollbacks. If a genuine emergency requires deploying an unsigned image, the incident must be declared, a time-limited namespace exemption must be applied with full audit logging, and the image must be re-signed and re-deployed within four hours.
Step 8: Air-Gapped Environments — Local Rekor and Fulcio
Air-gapped clusters cannot reach rekor.sigstore.dev or fulcio.sigstore.dev. The sigstore-scaffold project provides Helm charts for an internal Sigstore stack.
Deploy a local Sigstore stack:
# Install the sigstore scaffolding charts.
helm repo add sigstore https://sigstore.github.io/helm-charts
helm repo update
# Deploy Trillian (the transparency log backend).
helm install trillian sigstore/trillian \
--namespace sigstore-system \
--create-namespace
# Deploy Rekor (transparency log API).
helm install rekor sigstore/rekor \
--namespace sigstore-system \
--set server.extraArgs='{--trillian_log_server.address=trillian-logserver.sigstore-system:8090}'
# Deploy Fulcio (certificate authority).
helm install fulcio sigstore/fulcio \
--namespace sigstore-system \
--set config.OIDCIssuers[0].IssuerURL=https://kubernetes.default.svc \
--set config.OIDCIssuers[0].ClientID=sigstore \
--set config.OIDCIssuers[0].Type=kubernetes
# Deploy CTLog (certificate transparency log for Fulcio certificates).
helm install ctlog sigstore/ctlog \
--namespace sigstore-system
# Deploy a local TUF root (distributes the trust bundle to clients).
helm install tuf sigstore/tuf \
--namespace sigstore-system
Configure CI and clients to use the internal stack:
# Export environment variables before running cosign.
export SIGSTORE_REKOR_API_URL=https://rekor.sigstore-system.svc.cluster.local
export SIGSTORE_CT_LOG_PUBLIC_KEY_FILE=/etc/sigstore/ctlog-pub.pem
export SIGSTORE_ROOT_FILE=/etc/sigstore/root.json # Local TUF root.
export FULCIO_URL=https://fulcio.sigstore-system.svc.cluster.local
export COSIGN_MIRROR=https://tuf.sigstore-system.svc.cluster.local
cosign sign --yes ghcr.io/myorg/myapp@sha256:abc123...
Configure the policy-controller to use the internal stack:
# policy-controller-values.yaml (passed to helm install)
policyController:
cosignSystemConfigMap:
rekorURL: "https://rekor.sigstore-system.svc.cluster.local"
fulcioURL: "https://fulcio.sigstore-system.svc.cluster.local"
tufMirror: "https://tuf.sigstore-system.svc.cluster.local"
tufRoot: "/etc/sigstore/tuf-root.json"
Mirror the TUF root and public keys to each cluster node using a DaemonSet that runs at boot, ensuring every node has the trust material required to verify signatures without an external network call.
Expected Behaviour
| Scenario | Without deploy-time policy | With deploy-time policy |
|---|---|---|
| Unsigned image pushed directly to registry | Deploys silently | Rejected by admission webhook |
| Image signed by wrong workflow (fork, wrong branch) | Indistinguishable from valid | Identity mismatch; admission blocked |
| Image with critical CVE deployed after scan grace period | Deploys silently | Blocked by vulnerability attestation check |
| SBOM contains banned package | Deploys silently | Blocked by SBOM policy gate |
| SLSA Level 1 image (no hermetic build guarantee) | Deploys silently | Blocked by provenance level constraint |
| Rollback to pre-signed image | Works | Works (pre-signed images pass policy) |
| Emergency rollback to unsigned image | Works (silently dangerous) | Blocked unless break-glass activated with audit log |
Trade-offs
| Aspect | Benefit | Cost | Mitigation |
|---|---|---|---|
| Admission-time enforcement | Cannot be bypassed from the pipeline | Adds 100–300ms to pod admission; webhook outage blocks all scheduling | Run policy-controller in HA (3+ replicas); set failurePolicy: Fail with HA; set failurePolicy: Ignore only if availability outweighs security for the namespace. |
| Requiring all five properties | Maximum supply chain coverage | Some third-party images will never have SLSA provenance | Scope strict policies to first-party images; use weaker policies (signature only) for third-party namespaces. |
| Audit mode transition | Safe rollout with no outages | Compliance theatre if teams never graduate to enforce mode | Set a documented, time-limited audit window. Track the policy violation count as a metric with an SLO. |
| Local Sigstore (air-gapped) | No dependency on external services | Operational overhead: Rekor, Fulcio, CTLog, TUF — four additional services to maintain | Use sigstore-scaffold; deploy all components from the same Helm chart set. Back up the Trillian database. |
| Pre-signed rollback images | Rollback always works under policy | Requires build pipeline to sign rollback candidates proactively | Automate rollback tagging in the build workflow; verify rollback signatures in the deployment readiness gate. |
Failure Modes
| Failure | Symptom | Detection | Recovery |
|---|---|---|---|
| Policy-controller webhook unavailable | All pod creations blocked (if failurePolicy: Fail) |
Pod events: admission webhook timeout |
Restore policy-controller replicas; maintain 3+ replicas to avoid single points of failure. |
| Rekor unavailable (public or internal) | cosign verify fails; existing running pods unaffected; new pods blocked |
Signing and verification errors in CI and webhook logs | Use offline verification (cosign verify --offline) if Rekor checkpoint was previously fetched; restore Rekor if using internal stack. |
| Image has valid signature but no SBOM attestation | Pod blocked by SBOM policy | Kyverno policy report shows require-sbom-attestation/check-sbom-attached failure |
Re-run CI to generate and attach the SBOM attestation; or temporarily relax SBOM policy in audit mode while remediating. |
| Banned package added to approved base image | All images derived from that base are blocked at deploy time | SBOM policy gate blocks all pods using the base image | Update base image to remove the banned package; rebuild and re-sign all derived images; emergency: add time-limited exception with security team approval. |
Kyverno mutateDigest rewrites image to unexpected digest |
Pod uses different image than intended | Image digest in pod spec does not match intended release | This is correct behaviour: the policy pinned the tag to the digest of the signed image. Investigate if the running digest differs from the expected release digest. |
| slsa-verifier version mismatch with attestation format | PreSync hook fails on valid image | Hook job logs show unsupported predicate type |
Pin slsa-verifier to the version that matches the generator version used in CI. |