Automating Container Image Patching in Kubernetes with Copa and Kyverno

Problem

A Kubernetes cluster running a hundred distinct container images is a patching problem that does not scale with human toil. Every image has its own OS package set, its own CVE exposure surface, and its own upstream release cadence — which may be slow, irregular, or permanently stalled for images tied to abandoned projects. When a scanner reports that nginx:1.25.3 has a critical OpenSSL CVE, someone has to determine which Deployments use that image, whether upstream has published a fix, what the patching path is if they have not, and then execute the patch across every affected workload. In a cluster where that same situation is true for thirty images simultaneously, the manual process collapses.

Copa (Copacetic) solves the execution side of this: given a Trivy scan report and an OCI image reference, Copa applies OS package patches directly to an existing image layer without requiring a Dockerfile, a full rebuild, or access to the original build pipeline. A critical CVE in a libssl package is patched by pulling the updated package from the OS package repository and writing it into the image as a new layer. The base image remains otherwise unchanged. The patched image is pushed back to the registry under a new digest.

But Copa running on a developer’s laptop or in a one-shot CI job solves only half the problem. Two gaps remain that a production patching system must close:

The scale gap. Copa must be invoked per-image, on a schedule, for every image tracked in the cluster. An ad-hoc manual invocation misses images that were not on the operator’s mental checklist. A one-shot CI job tied to a specific Deployment’s pipeline misses shared base images used by multiple Deployments. A CronJob that iterates a ConfigMap of all tracked images and applies Copa to each one closes this gap systematically.

The closed-loop gap. After Copa patches an image and pushes the new digest, nothing prevents the old image tag from being referenced in a Deployment manifest and re-scheduled. imagePullPolicy: IfNotPresent compounds this: if the unpatched image is already cached on a node, Kubernetes will not pull the patched version at all — the cached unpatched image runs indefinitely. A Kyverno ClusterPolicy that evaluates image vulnerability state at admission time closes this gap: if a Pod references an image with a critical CVE that should have been patched, the admission request is denied.

This article builds both halves. Target environment: Kubernetes 1.28+, containerd runtime, a CronJob running Copa and Trivy with a BuildKit sidecar, Kyverno 1.11+, and any OCI-compliant registry (the examples use a generic REGISTRY variable; ECR, GCR, and ACR credential approaches are addressed in the registry credentials section).

Threat Model

Threat 1: Critical CVE disclosed against an image already deployed to the cluster with no automated patching mechanism. The image was clean at deploy time. A CVE is disclosed against one of its OS packages. Without an automated scanner and patcher running on a schedule, the image continues to run indefinitely. The window of exposure is bounded only by the next application team deployment — which may be weeks away and is not tied to patch SLAs.

Threat 2: Patched image exists in registry but the old image tag is still referenced in a Deployment. Copa has already patched the image and pushed a new digest tagged myapp:1.4.2-patched-20260509. The Deployment still references myapp:1.4.2. The Pod scheduler starts new Pods using myapp:1.4.2. The vulnerability is patched in the registry but still running in the cluster. Without an admission control policy enforcing that deployed images must be patched, the patched image is irrelevant.

Threat 3: New Pod scheduled from a cached unpatched image due to imagePullPolicy: IfNotPresent. A node cached myapp:1.4.2 before Copa ran. Even if myapp:1.4.2 is updated in the registry, Kubernetes does not re-pull the image on subsequent Pod starts for that node. The running Pod uses an image that has been nominally “updated” in the registry but is actually an older, unpatched layer on disk. This is a silent failure: kubectl describe pod shows the correct tag, but the actual image binary differs from what the registry serves.

Access level assumed for exploitation. Threats 1 and 3 require only that an attacker discover and exploit a CVE in a running container. The attacker needs no Kubernetes API access — container-level exploitation is sufficient. Threat 2 requires an attacker who can reach the network services exposed by the vulnerable container. In all cases the Kubernetes control plane and registry are not compromised; the vulnerability is in application or OS-level code inside the container.

Configuration

Architecture Overview

The system has four components:

1. Copa CronJob — runs on a configurable schedule (e.g., every 6 hours), reads a ConfigMap listing images to patch, runs trivy image against each, and for images with critical CVEs above a configurable threshold, runs copa patch to produce a patched OCI image, which it pushes to the registry under a new digest. It updates a registry tag (-patched-YYYYMMDD) and optionally writes the new digest to a ConfigMap for Flux to consume.

2. BuildKit sidecar — Copa requires BuildKit to build the patched image layer. The CronJob Pod runs a BuildKit daemon as a sidecar container exposed via a Unix socket shared with the Copa container.

3. Kyverno ClusterPolicy — evaluates each Pod at admission time. Images in the cluster’s tracked set that have not been patched by Copa within the patch SLA window are denied. The policy reads a ConfigMap of “approved patched digests” maintained by the CronJob.

4. Flux image automation controller — when Copa pushes a new patched digest, the Flux image automation controller detects the updated registry tag and opens a pull request (or directly commits) to update Deployment manifests to the new digest.

Step 1: Namespace and ServiceAccount

# copa-system-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: copa-system
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest

# copa-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: copa-patcher
  namespace: copa-system
  # For Workload Identity (GKE/AKS/EKS): add the provider-specific annotation here.
  # GKE: iam.gke.io/gcp-service-account: copa-patcher@PROJECT.iam.gserviceaccount.com
  # AKS: azure.workload.identity/client-id: <client-id>
  # EKS: eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/copa-patcher
automountServiceAccountToken: false

Copa’s CronJob does not need any Kubernetes API access beyond reading its own ConfigMap. The ServiceAccount intentionally has automountServiceAccountToken: false. Registry access is handled via imagePullSecrets or Workload Identity — not via the Kubernetes API.

# copa-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: copa-configmap-reader
  namespace: copa-system
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]
    resourceNames: ["copa-image-list", "copa-patched-digests"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["update", "patch"]
    resourceNames: ["copa-patched-digests"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
    resourceNames: ["copa-registry-credentials"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: copa-patcher-binding
  namespace: copa-system
subjects:
  - kind: ServiceAccount
    name: copa-patcher
    namespace: copa-system
roleRef:
  kind: Role
  name: copa-configmap-reader
  apiGroup: rbac.authorization.k8s.io

Step 2: Image List ConfigMap

# copa-image-list.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: copa-image-list
  namespace: copa-system
data:
  # One image per line. Format: REGISTRY/REPO:TAG
  # Copa patches the image and pushes a new tag: REGISTRY/REPO:TAG-patched-YYYYMMDD
  images.txt: |
    registry.example.com/myapp/api:1.4.2
    registry.example.com/myapp/worker:2.1.0
    registry.example.com/myapp/frontend:3.0.1
    registry.example.com/shared/nginx-base:1.25.3
    registry.example.com/shared/python-base:3.12.2
  # Severity threshold: patch images with CVEs at or above this level
  severity_threshold: "CRITICAL"
  # Patch SLA in hours: how old can a patched image be before Kyverno blocks
  patch_sla_hours: "24"

Step 3: Copa CronJob with BuildKit Sidecar

# copa-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: copa-image-patcher
  namespace: copa-system
spec:
  schedule: "0 */6 * * *"        # Every 6 hours
  concurrencyPolicy: Forbid       # Do not allow overlapping runs
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      activeDeadlineSeconds: 3600 # Hard cap: 1 hour total
      backoffLimit: 0             # Do not retry; next scheduled run will pick up
      template:
        metadata:
          labels:
            app: copa-patcher
        spec:
          serviceAccountName: copa-patcher
          automountServiceAccountToken: false
          restartPolicy: Never
          securityContext:
            runAsNonRoot: false   # BuildKit requires root; isolate via seccomp
            seccompProfile:
              type: RuntimeDefault
          imagePullSecrets:
            - name: copa-registry-credentials
          volumes:
            - name: buildkit-socket
              emptyDir: {}
            - name: copa-script
              configMap:
                name: copa-patch-script
                defaultMode: 0755
            - name: image-list
              configMap:
                name: copa-image-list
            - name: registry-config
              secret:
                secretName: copa-registry-credentials
                items:
                  - key: .dockerconfigjson
                    path: config.json

          initContainers:
            - name: wait-for-buildkitd
              image: ghcr.io/project-copacetic/copacetic:v0.8.0
              command:
                - sh
                - -c
                - |
                  echo "Waiting for BuildKit socket..."
                  until [ -S /run/buildkit/buildkitd.sock ]; do
                    sleep 1
                  done
                  echo "BuildKit socket ready."
              volumeMounts:
                - name: buildkit-socket
                  mountPath: /run/buildkit
              securityContext:
                runAsNonRoot: false
                allowPrivilegeEscalation: false

          containers:
            - name: buildkitd
              image: moby/buildkit:v0.17.2-rootless
              args:
                - --addr
                - unix:///run/buildkit/buildkitd.sock
                - --oci-worker-no-process-sandbox
              securityContext:
                runAsNonRoot: false
                privileged: false
                seccompProfile:
                  type: Unconfined  # BuildKit rootless requires this
              resources:
                requests:
                  cpu: 500m
                  memory: 1Gi
                limits:
                  cpu: "2"
                  memory: 4Gi
              volumeMounts:
                - name: buildkit-socket
                  mountPath: /run/buildkit

            - name: copa-patcher
              image: ghcr.io/project-copacetic/copacetic:v0.8.0
              command: ["/scripts/patch-loop.sh"]
              env:
                - name: REGISTRY
                  value: "registry.example.com"
                - name: BUILDKIT_HOST
                  value: "unix:///run/buildkit/buildkitd.sock"
                - name: DOCKER_CONFIG
                  value: "/registry-config"
                - name: SEVERITY_THRESHOLD
                  valueFrom:
                    configMapKeyRef:
                      name: copa-image-list
                      key: severity_threshold
                - name: PATCH_SLA_HOURS
                  valueFrom:
                    configMapKeyRef:
                      name: copa-image-list
                      key: patch_sla_hours
              resources:
                requests:
                  cpu: 250m
                  memory: 512Mi
                limits:
                  cpu: "1"
                  memory: 2Gi
              securityContext:
                runAsNonRoot: false
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: false  # Copa writes temp files
              volumeMounts:
                - name: buildkit-socket
                  mountPath: /run/buildkit
                - name: copa-script
                  mountPath: /scripts
                - name: image-list
                  mountPath: /config
                - name: registry-config
                  mountPath: /registry-config
                  readOnly: true

Step 4: Patch Loop Script

# copa-patch-script-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: copa-patch-script
  namespace: copa-system
data:
  patch-loop.sh: |
    #!/usr/bin/env bash
    set -euo pipefail

    IMAGE_LIST="/config/images.txt"
    PATCH_DATE=$(date +%Y%m%d)
    PATCHED_DIGESTS_FILE="/tmp/patched-digests.txt"
    EXIT_CODE=0

    echo "Copa patch loop starting: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

    while IFS= read -r image || [[ -n "$image" ]]; do
      # Skip blank lines and comments
      [[ -z "$image" || "$image" == \#* ]] && continue

      echo "--- Processing: $image ---"

      # Extract components
      repo=$(echo "$image" | cut -d: -f1)
      tag=$(echo "$image" | cut -d: -f2)
      patched_tag="${tag}-patched-${PATCH_DATE}"
      patched_image="${repo}:${patched_tag}"

      # Run Trivy scan and emit JSON report
      SCAN_REPORT="/tmp/trivy-$(echo "$image" | tr '/: ' '_').json"

      echo "Scanning $image for ${SEVERITY_THRESHOLD}+ CVEs..."
      if ! trivy image \
        --exit-code 0 \
        --format json \
        --severity "${SEVERITY_THRESHOLD},HIGH" \
        --ignore-unfixed \
        --output "$SCAN_REPORT" \
        "$image" 2>&1; then
        echo "ERROR: trivy scan failed for $image — skipping"
        EXIT_CODE=1
        continue
      fi

      # Count fixable critical/high CVEs
      VULN_COUNT=$(jq '[.Results[]?.Vulnerabilities[]? | select(.FixedVersion != null and .FixedVersion != "")] | length' "$SCAN_REPORT")
      echo "Fixable CVEs found: $VULN_COUNT"

      if [[ "$VULN_COUNT" -eq 0 ]]; then
        echo "No fixable CVEs above threshold — skipping patch for $image"
        continue
      fi

      echo "Patching $image → $patched_image via Copa..."
      if ! copa patch \
        --image "$image" \
        --report "$SCAN_REPORT" \
        --output "$patched_image" \
        --addr "$BUILDKIT_HOST"; then
        echo "ERROR: copa patch failed for $image"
        EXIT_CODE=1
        continue
      fi

      # Push the patched image
      echo "Pushing $patched_image..."
      if ! crane push "$patched_image" "$patched_image" 2>&1; then
        # Copa already pushes; this step logs the digest
        true
      fi

      # Retrieve and record the new digest
      NEW_DIGEST=$(crane digest "$patched_image" 2>/dev/null || echo "unknown")
      echo "$repo@${NEW_DIGEST}  # patched from $image on $PATCH_DATE" >> "$PATCHED_DIGESTS_FILE"
      echo "Patched digest: $NEW_DIGEST"

      echo "--- Done: $image ---"
    done < "$IMAGE_LIST"

    # Write the patched digest list to a ConfigMap for Kyverno and Flux to consume
    if [[ -f "$PATCHED_DIGESTS_FILE" ]]; then
      kubectl create configmap copa-patched-digests \
        --namespace copa-system \
        --from-file=digests.txt="$PATCHED_DIGESTS_FILE" \
        --dry-run=client -o yaml | kubectl apply -f -
    fi

    echo "Copa patch loop complete: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
    exit $EXIT_CODE

Step 5: Kyverno ClusterPolicy — Block Unpatched Images

Kyverno 1.11+ supports imageVerify with an external data source. The policy below takes two complementary approaches: the primary approach uses an imageVerify rule to check that the image tag includes a -patched- suffix and was pushed within the SLA window; the secondary (belt-and-suspenders) approach uses a validate rule with an external context call to a policy-controlled ConfigMap of approved digests.

# kyverno-block-unpatched.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-unpatched-images
  annotations:
    policies.kyverno.io/title: Block Unpatched Container Images
    policies.kyverno.io/description: >
      Blocks Pods that reference container images from the tracked image set
      unless those images carry a Copa-patched tag applied within the patch SLA
      window. Applies to all namespaces except kube-system and copa-system.
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: require-patched-tag-for-tracked-images
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - copa-system
                - kyverno
      context:
        - name: trackedImages
          configMap:
            name: copa-image-list
            namespace: copa-system
        - name: patchedDigests
          configMap:
            name: copa-patched-digests
            namespace: copa-system
      validate:
        message: >
          Image {{ element.image }} is in the tracked image set but does not
          carry a Copa-patched tag. Re-deploy using the patched digest published
          by the copa-image-patcher CronJob in the copa-system namespace.
        foreach:
          - list: "request.object.spec.containers"
            deny:
              conditions:
                all:
                  # The image repo is in our tracked set
                  - key: "{{ element.image }}"
                    operator: AnyIn
                    value: "{{ trackedImages.data.\"images.txt\" | parse_yaml(@) }}"
                  # But the tag does NOT contain "-patched-"
                  - key: "{{ element.image | split(@, ':') | [-1:] | [0] | contains(@, '-patched-') }}"
                    operator: Equals
                    value: "false"

The validationFailureAction: Enforce setting means mismatched images are denied at admission, not just logged. Switch to Audit during initial rollout to understand what would be blocked before enforcing.

Step 6: Alternative Kyverno Approach — Require Image Digest from Approved List

A complementary rule enforces that images use a digest reference (not a mutable tag) that appears in the ConfigMap of Copa-patched digests:

# kyverno-require-approved-digest.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-copa-approved-digest
spec:
  validationFailureAction: Audit  # Start in Audit; promote to Enforce after rollout
  background: true
  rules:
    - name: image-must-use-approved-digest
      match:
        any:
          - resources:
              kinds: [Pod]
      exclude:
        any:
          - resources:
              namespaces: [kube-system, copa-system, kyverno]
      context:
        - name: approvedDigests
          configMap:
            name: copa-patched-digests
            namespace: copa-system
      validate:
        message: >
          Image {{ element.image }} must reference a digest approved by the Copa
          CronJob. Check copa-patched-digests ConfigMap in copa-system for the
          approved digest for this image.
        foreach:
          - list: "request.object.spec.containers"
            deny:
              conditions:
                all:
                  - key: "{{ contains(element.image, '@sha256:') }}"
                    operator: Equals
                    value: "false"

This rule is complementary, not a replacement. It enforces the broader practice of digest-pinned images and should be applied across all workloads, not just Copa-tracked ones.

Step 7: Flux Image Automation Integration

After Copa pushes a patched image with a new tag, Flux’s image automation controller can detect the new tag and update Deployment manifests automatically.

# flux-image-repository.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: myapp-api
  namespace: flux-system
spec:
  image: registry.example.com/myapp/api
  interval: 10m
  secretRef:
    name: registry-credentials
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: myapp-api-patched
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: myapp-api
  filterTags:
    # Match tags of the form 1.4.2-patched-20260509
    pattern: '^[0-9]+\.[0-9]+\.[0-9]+-patched-[0-9]{8}$'
    extract: '$patch_date'
  policy:
    alphabetical:
      order: asc

In the Deployment manifest, mark the image field with a Flux image automation marker:

# In the Deployment spec.template.spec.containers section:
containers:
  - name: api
    image: registry.example.com/myapp/api:1.4.2-patched-20260509 # {"$imagepolicy": "flux-system:myapp-api-patched"}

Flux detects when Copa pushes a newer -patched-YYYYMMDD tag, updates the Deployment manifest in the Git repository, and reconciles. The loop is complete: Copa patches, Flux detects and updates the manifest, new Pods pick up the patched image digest.

Step 8: Registry Credentials

For static credentials (username/password), store as a Kubernetes Secret and reference via imagePullSecrets:

kubectl create secret docker-registry copa-registry-credentials \
  --namespace copa-system \
  --docker-server=registry.example.com \
  --docker-username=copa-service-user \
  --docker-password="$(vault kv get -field=password secret/copa/registry)" \
  --docker-email=copa@example.com

For cloud-managed registries using Workload Identity, annotate the ServiceAccount as shown in Step 1 and grant the cloud IAM role the push permission on the specific repository path — not the entire registry. For ECR, this is the ecr:BatchGetImage, ecr:GetDownloadUrlForLayer, ecr:PutImage, and ecr:InitiateLayerUpload permissions scoped to the repository ARN.

Expected Behaviour

The table below describes the system response to each security-relevant scenario once the CronJob and Kyverno policies are fully operational:

Trade-offs

Failure Modes

Related Articles

• Copa: Patching Distroless and Scratch Images — how Copa handles images with no package manager, including the limitations and the alternate patch paths
• Kyverno Controller Security: Hardening the Policy Engine — hardening the Kyverno deployment itself, because a compromised policy engine silently allows everything
• Copa in CI/CD: Shift-Left Container Patch Automation — integrating Copa into GitHub Actions and Tekton pipelines as a build-time gate, complementing the runtime CronJob approach described here
• Container Patch SLA Policy Enforcement — defining and enforcing patch SLAs across multiple teams and clusters using policy-as-code
• Container Patch Compliance Observability — dashboards, metrics, and alerts for tracking patch coverage, SLA compliance, and CronJob health across the fleet