Patching Distroless and Minimal Container Images with Copa

Problem

Every container image in production is a snapshot of a package state at build time. The moment a CVE is disclosed for OpenSSL, glibc, zlib, or any other OS-level library embedded in that base image, the clock starts. Threat actors run automated scanners against NVD and scan reachable services for the vulnerable version within minutes of disclosure. The mean time from CVE publication to active exploitation in the wild has dropped from weeks to days for high-severity findings.

The traditional response is straightforward: update the Dockerfile’s base image reference, rebuild, re-test, push. That path requires owning the Dockerfile, having a working build pipeline, access to source dependencies, and a test suite that completes in a reasonable time. It works for first-party application images.

It breaks entirely for several common cases:

• Upstream or third-party images pulled from a registry — you don’t own the Dockerfile, you can’t trigger the upstream rebuild, and the upstream maintainer may take days or weeks to cut a patched release.
• Distroless images (gcr.io/distroless/*, cgr.dev/chainguard/*) — these images contain no shell, no package manager binary, and sometimes no init system. The patching mechanism does not exist inside the image.
• Scratch-based images with a copied OS layer — package metadata may be present in the filesystem but no tooling is available to invoke it.
• Pipeline bottlenecks — even for images you own, a full rebuild through a CI pipeline may take 20–40 minutes, gated behind code review, integration tests, and deployment approvals that don’t account for emergency security response.

Copa (project-copacetic, maintained under the CNCF sandbox) addresses this gap. It operates outside the image: Copa reads a vulnerability scanner report (Trivy JSON format), determines which OS packages have available fixes, fetches those packages from the distribution’s package repository, and applies them as a new OCI layer on top of the existing image using BuildKit. Copa never needs access to the original Dockerfile, build context, or source repository. It never runs a package manager inside the image. It works against a pulled image from any OCI-compliant registry.

For distroless images, Copa extracts the package database directly from the image filesystem — reading the dpkg status file at /var/lib/dpkg/status or the apk database at /lib/apk/db/installed — and uses that metadata to determine what to update without invoking any binary from the image.

Target systems: Any OCI-compliant container image on Debian 10+, Ubuntu 20.04+, Alpine 3.12+, RHEL/UBI 8+, and their distroless variants. Requires BuildKit 0.11+ (Docker Engine 23+ includes a compatible BuildKit). Copa 0.6+.

Threat Model

Adversary 1 — CVE exploitation window against unowned base images. A team uses node:20-slim as their application base. A critical CVE is disclosed in libssl3 bundled in that image. The Docker Hub upstream won’t publish a patched tag for 5 days. The application team has no mechanism to patch the base image without owning its build pipeline. Attackers scan for the vulnerable OpenSSL TLS handshake path within hours.

Adversary 2 — Distroless images bypassing the remediation process entirely. A platform team has standardized on gcr.io/distroless/java21-debian12 for all JVM workloads. A vulnerability is discovered in a bundled libz version. The security team’s standard patch procedure — open Jira, trigger rebuild — fails because distroless images are not built in-house. The security tooling cannot patch them. The image sits unpatched indefinitely because no remediation path exists in the process.

Adversary 3 — Upstream publisher delay as an exploitation window. Google publishes new distroless images roughly weekly. A CVE disclosed on Monday may not appear in an updated distroless tag until the following Friday. For CVEs with public exploits, a 4-day window against production workloads running distroless images is a material risk.

Blast radius comparison: An unpatched distroless image runs in a Kubernetes pod with access to internal APIs, secrets mounted from a secret store, and network access to the data tier. A successful exploit of the vulnerable library executes code in that process context. Copa-patching that image takes 60–120 seconds, requires no code change, no pipeline run, and no deployment approval for the patch layer itself — only for the image promotion step. The patched image can be verified with a Trivy rescan before promotion, confirming CVE count reduction at the package level.

Configuration / Implementation

Installing Copa

Copa ships as a single static binary. Download from GitHub releases:

# Set the desired version.
COPA_VERSION="0.7.1"

# Download the binary for Linux amd64.
curl -sSfL \
  "https://github.com/project-copacetic/copacetic/releases/download/v${COPA_VERSION}/copa_${COPA_VERSION}_linux_amd64.tar.gz" \
  | tar -xz copa

# Move to PATH.
sudo mv copa /usr/local/bin/copa
chmod +x /usr/local/bin/copa

# Verify.
copa version

Alternatively, install from source with Go 1.21+:

go install github.com/project-copacetic/copacetic/cmd/copa@latest

BuildKit Requirement

Copa uses BuildKit as its low-level layer construction engine. BuildKit must be running as an accessible daemon. There are three options:

Option 1: Docker-bundled BuildKit (simplest for local use)

Docker Engine 23+ ships with BuildKit and exposes it on the standard Docker socket. Copa can use this directly:

# Confirm Docker's bundled BuildKit is active.
docker buildx version
# buildx v0.12.0 ...

# Use Copa with the Docker socket (default when DOCKER_HOST is set).
copa patch \
  --image myimage:1.0 \
  --report report.json \
  --tag myimage:1.0-patched

Option 2: Standalone buildkitd daemon

For CI environments or systems without Docker, run buildkitd directly:

# Install buildkitd.
BUILDKIT_VERSION="0.13.2"
curl -sSfL \
  "https://github.com/moby/buildkit/releases/download/v${BUILDKIT_VERSION}/buildkit-v${BUILDKIT_VERSION}.linux-amd64.tar.gz" \
  | tar -xz -C /usr/local

# Start buildkitd (runs as root; use rootless variant for unprivileged).
buildkitd --addr unix:///run/buildkit/buildkitd.sock &

# Verify it's listening.
buildctl --addr unix:///run/buildkit/buildkitd.sock debug workers

Option 3: Remote BuildKit socket in CI

# Point Copa at a remote BuildKit instance.
copa patch \
  --image myimage:1.0 \
  --report report.json \
  --tag myimage:1.0-patched \
  --addr tcp://buildkitd.internal:1234

Generating the Trivy Report

Copa requires a Trivy JSON vulnerability report in a specific schema. The report must include OS package vulnerability data — not just application-level findings:

# Install Trivy if not present.
curl -sSfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh \
  | sh -s -- -b /usr/local/bin

# Generate the report Copa expects.
# --vuln-type os restricts to OS packages (Copa only patches OS packages).
# --ignore-unfixed omits CVEs with no available fix (Copa can only apply fixes that exist).
trivy image \
  --format json \
  --output report.json \
  --vuln-type os \
  --ignore-unfixed \
  myimage:1.0

The --ignore-unfixed flag is important: Copa will error or skip packages if the report contains CVEs for which no fixed version exists in the distribution repository. Filtering them out at the Trivy stage makes Copa output predictable.

Patching a Debian/Ubuntu Image

This walkthrough patches nginx:1.25-bookworm, a Debian 12-based image that typically has several OS package CVEs in its bundled libraries.

# Step 1: Pull the image.
docker pull nginx:1.25-bookworm

# Step 2: Generate the Trivy report.
trivy image \
  --format json \
  --output nginx-report.json \
  --vuln-type os \
  --ignore-unfixed \
  nginx:1.25-bookworm

# Inspect the findings count.
jq '[.Results[].Vulnerabilities // [] | .[]] | length' nginx-report.json
# e.g., 14

# Step 3: Run Copa.
copa patch \
  --image nginx:1.25-bookworm \
  --report nginx-report.json \
  --tag nginx:1.25-bookworm-patched

# Copa output (abbreviated):
# INFO    Loading image nginx:1.25-bookworm
# INFO    Fetching updated packages from debian:bookworm
# INFO    Applying layer for packages: libssl3, libgnutls30, libsystemd0, libzstd1 ...
# INFO    Wrote patched image as nginx:1.25-bookworm-patched

# Step 4: Verify with Trivy rescan.
trivy image \
  --format json \
  --output nginx-patched-report.json \
  --vuln-type os \
  --ignore-unfixed \
  nginx:1.25-bookworm-patched

jq '[.Results[].Vulnerabilities // [] | .[]] | length' nginx-patched-report.json
# e.g., 0 — all OS package CVEs with fixes applied.

Copa resolves the package names from the Trivy report to their apt-compatible counterparts, fetches updated .deb archives directly from the Debian mirror (without running apt inside the container), extracts the package contents, and squashes them into a new layer added to the image manifest.

Patching an Alpine Image

Alpine images use apk and store their package database at /lib/apk/db/installed. Copa handles this path automatically:

# Pull an Alpine-based image.
docker pull python:3.11-alpine3.19

# Generate the Trivy report.
trivy image \
  --format json \
  --output python-alpine-report.json \
  --vuln-type os \
  --ignore-unfixed \
  python:3.11-alpine3.19

# Patch with Copa.
copa patch \
  --image python:3.11-alpine3.19 \
  --report python-alpine-report.json \
  --tag python:3.11-alpine3.19-patched

# Verify.
trivy image \
  --vuln-type os \
  --ignore-unfixed \
  python:3.11-alpine3.19-patched

The key difference for Alpine: Copa fetches .apk packages from the Alpine mirror rather than .deb files, and updates the apk database file inside the new layer so subsequent apk info calls on a running container reflect the patched state (even though no apk binary is invoked during patching).

Patching a Distroless Image

This is the most significant capability gap Copa fills. gcr.io/distroless/base-debian12 contains no shell (/bin/sh), no package manager, and no way to run any remediation tooling inside the container. But it does contain a dpkg status file at /var/lib/dpkg/status that records every installed package and version.

Copa’s distroless patching path:

1. Mounts the image filesystem in memory via BuildKit.
2. Reads /var/lib/dpkg/status to build a package inventory.
3. Cross-references inventory against the Trivy report to identify packages needing updates.
4. Fetches the updated .deb archives from the Debian bookworm repository.
5. Extracts the package contents and updated control files.
6. Writes a new layer containing the updated binaries and the updated dpkg status file.
7. Appends the layer to the image manifest.

# Pull the distroless image.
docker pull gcr.io/distroless/base-debian12:latest

# Generate Trivy report.
# Note: Trivy can scan distroless images — it reads the dpkg status file the same way Copa does.
trivy image \
  --format json \
  --output distroless-base-report.json \
  --vuln-type os \
  --ignore-unfixed \
  gcr.io/distroless/base-debian12:latest

# Check what Copa will patch.
jq '.Results[].Vulnerabilities // [] | .[] | .PkgName' distroless-base-report.json | sort -u
# e.g.: "libssl3", "libgnutls30", "zlib1g"

# Patch with Copa.
copa patch \
  --image gcr.io/distroless/base-debian12:latest \
  --report distroless-base-report.json \
  --tag gcr.io/distroless/base-debian12:patched-20260509

# The patched image still has no shell. Copa never adds one.
# Verify by attempting to exec (should fail — no shell is correct).
docker run --rm --entrypoint /bin/sh gcr.io/distroless/base-debian12:patched-20260509 -c "echo test"
# Error: no such file or directory — correct, distroless properties preserved.

# Verify the CVEs are gone.
trivy image \
  --vuln-type os \
  --ignore-unfixed \
  gcr.io/distroless/base-debian12:patched-20260509
# Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

Copa does not add a shell, a package manager, or any debug tooling to the patched image. The distroless security properties (no shell, minimal attack surface, no package manager to abuse) are fully preserved.

Pushing the Patched Image and Signing with Cosign

After patching, push the image to your registry and sign it to establish provenance:

# Tag for your registry.
docker tag \
  gcr.io/distroless/base-debian12:patched-20260509 \
  registry.example.com/base/distroless-base-debian12:patched-20260509

# Push.
docker push registry.example.com/base/distroless-base-debian12:patched-20260509

# Sign with cosign (keyless via Sigstore, requires OIDC token in CI).
cosign sign \
  --yes \
  registry.example.com/base/distroless-base-debian12:patched-20260509

# Alternatively, sign with a key file.
cosign sign \
  --key cosign.key \
  registry.example.com/base/distroless-base-debian12:patched-20260509

# Attach the Trivy SBOM/report as an attestation.
cosign attest \
  --key cosign.key \
  --type vuln \
  --predicate distroless-patched-report.json \
  registry.example.com/base/distroless-base-debian12:patched-20260509

Full CI Workflow with Remote BuildKit

In a CI pipeline (GitHub Actions, GitLab CI, Tekton), the full sequence runs as a single job:

#!/usr/bin/env bash
set -euo pipefail

IMAGE="${1}"          # e.g., registry.example.com/app/myservice:1.4.2
PATCHED_TAG="${2}"    # e.g., registry.example.com/app/myservice:1.4.2-patched
BUILDKIT_ADDR="${BUILDKIT_ADDR:-tcp://buildkitd.internal:1234}"

# Step 1: Generate vulnerability report.
trivy image \
  --format json \
  --output /tmp/report.json \
  --vuln-type os \
  --ignore-unfixed \
  "${IMAGE}"

# Step 2: Check if there are any patchable CVEs — skip patching if clean.
VULN_COUNT=$(jq '[.Results[].Vulnerabilities // [] | .[]] | length' /tmp/report.json)
if [ "${VULN_COUNT}" -eq 0 ]; then
  echo "No OS vulnerabilities with fixes — skipping Copa patch."
  exit 0
fi

echo "Found ${VULN_COUNT} patchable OS vulnerabilities. Running Copa."

# Step 3: Patch.
copa patch \
  --image "${IMAGE}" \
  --report /tmp/report.json \
  --tag "${PATCHED_TAG}" \
  --addr "${BUILDKIT_ADDR}"

# Step 4: Verify.
trivy image \
  --format json \
  --output /tmp/report-patched.json \
  --vuln-type os \
  --ignore-unfixed \
  "${PATCHED_TAG}"

REMAINING=$(jq '[.Results[].Vulnerabilities // [] | .[]] | length' /tmp/report-patched.json)
echo "Remaining OS CVEs after patch: ${REMAINING}"

# Step 5: Push and sign.
docker push "${PATCHED_TAG}"
cosign sign --yes "${PATCHED_TAG}"

Expected Behaviour

Trade-offs

Failure Modes

Related Articles

• Container Vulnerability Scanning in CI — setting up Trivy in your pipeline to generate the reports Copa consumes, and establishing vulnerability gates before promotion.
• Automating Copa Patching in CI/CD Pipelines — building Copa into GitHub Actions, GitLab CI, and Tekton to patch images automatically on CVE disclosure, with registry promotion policies.
• Copa-Based Automated Patching in Kubernetes — running Copa as a Kubernetes Job or CronJob to patch running workloads in-cluster, and updating Deployment image references via the Kubernetes API.
• SBOM Generation and Consumption — using SBOMs alongside Copa to understand the full package inventory before and after patching, and to verify patch completeness.
• Container Patch SLA Policy Enforcement — defining and enforcing SLA policies (e.g., critical CVEs patched within 24 hours) using Copa as the remediation mechanism and OPA/Gatekeeper as the enforcement layer.