Copa in CI/CD: Automated Container Patch Pipelines with Trivy, cosign, and GitHub Actions
Problem
Container scanning in CI solves the wrong half of the problem. Trivy, Grype, and similar tools tell you which OS packages in your image are vulnerable. They do not fix them. The conventional response is to rebuild: update the base image, bump a FROM line, and let the pipeline run again. That works when your team owns the Dockerfile. It fails completely when the vulnerable packages live in a third-party base image you do not control — which is the case for most production workloads.
The traditional scan-build-push loop also has a timing problem. An image scanned clean at build time can accumulate CVEs between build and deploy. If your deployment cadence is weekly but new CVEs are disclosed daily, you are perpetually deploying images that were only clean at the moment of the scan. There is no in-pipeline mechanism to close that gap.
Copa (Copacetic) addresses both problems directly. It operates on the already-built image, using BuildKit to apply updated OS packages without rebuilding from a Dockerfile. The input is a Trivy JSON report; the output is a new image digest with the vulnerable packages replaced by patched versions. The process runs entirely within the pipeline, on images you did not build, sourced from registries you do not own.
The practical result is a scan-patch-verify loop that runs on any OCI-compliant image:
- Trivy scans the image and produces a structured vulnerability report.
- Copa reads the report and patches OS package CVEs in place via BuildKit.
- Trivy re-scans the patched image to confirm CVE elimination.
- cosign signs the verified patched digest with keyless OIDC attestation.
- The signed, verified image is promoted to the production registry.
This article covers the full implementation in GitHub Actions and Tekton Pipelines, including patch failure handling, promotion gating, and scheduled nightly re-patching of production images.
Target systems: Copa 0.6+; Trivy 0.50+; cosign 2.x; GitHub Actions; Tekton Pipelines v1 API; BuildKit 0.12+; any OCI-compliant registry; Debian, Alpine, and RHEL-based image OS packages.
Threat Model
Threat 1 — CVE disclosed between build and deploy. An image builds clean on Monday. A critical CVE in libssl is published Wednesday. The image deploys Friday with no additional scanning or patching. Without an in-pipeline patching mechanism, the build-time scan result is the only signal — and it is already stale.
Threat 2 — Critical CVE in a third-party base image. An application team uses FROM nginx:1.25 or a distroless base from a vendor. A new CVE affects a package in that base image. The application team has no upstream Dockerfile access. The conventional fix — modify the FROM line — is not available. The vulnerability persists until the upstream maintainer releases a new base image tag.
Threat 3 — Patched image pushed without re-verification. Copa runs and exits zero. The pipeline treats exit zero as success and promotes the image. However, Copa only patches packages for which a fix is available in the upstream package repository. If no fix exists, Copa may patch a subset of CVEs and leave others in place. Without a Trivy re-scan of the patched digest, the pipeline has no confirmation that the CVEs it intended to fix are actually gone.
Threat 4 — Patched image pushed unsigned. An admission controller in the production cluster requires cosign signature verification before allowing an image to run. A patched image pushed without cosign signing fails admission. Alternatively, without signing, there is no provenance record linking the patched digest to the pipeline run that produced it — any image with the right tag could be pulled.
Access level for all threats: No attacker action required for Threats 1–3; these are configuration gaps. Threat 4 requires a registry write access. The blast radius in each case is a production container running with exploitable OS packages, or a patched image that cannot pass admission control.
Configuration
GitHub Actions: Full Scan-Patch-Verify-Sign Workflow
The workflow is structured as four sequential jobs. Each job passes data forward using GitHub Actions artifacts and job outputs.
# .github/workflows/copa-patch.yml
name: Copa Container Patch Pipeline
on:
push:
branches: [main]
workflow_dispatch:
inputs:
image:
description: "Image to patch (e.g. ghcr.io/org/app:sha-abc123)"
required: true
type: string
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
permissions:
contents: read
packages: write
id-token: write # required for cosign OIDC keyless signing
issues: write # required for creating issues on patch failure
jobs:
scan:
name: Trivy Initial Scan
runs-on: ubuntu-24.04
outputs:
image-ref: ${{ steps.set-image.outputs.image-ref }}
critical-count: ${{ steps.parse.outputs.critical-count }}
steps:
- name: Log in to registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set image reference
id: set-image
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
echo "image-ref=${{ inputs.image }}" >> "$GITHUB_OUTPUT"
else
echo "image-ref=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}" >> "$GITHUB_OUTPUT"
fi
- name: Run Trivy scan
uses: aquasecurity/trivy-action@0.28.0
with:
image-ref: ${{ steps.set-image.outputs.image-ref }}
format: json
output: trivy-report.json
severity: CRITICAL,HIGH
exit-code: "0" # do not fail here; Copa will patch
- name: Parse critical CVE count
id: parse
run: |
COUNT=$(jq '[.Results[]?.Vulnerabilities[]? | select(.Severity == "CRITICAL")] | length' trivy-report.json)
echo "critical-count=${COUNT}" >> "$GITHUB_OUTPUT"
echo "Found ${COUNT} CRITICAL CVEs before patching"
- name: Upload Trivy report
uses: actions/upload-artifact@v4
with:
name: trivy-report-pre-patch
path: trivy-report.json
retention-days: 7
patch:
name: Copa Patch
runs-on: ubuntu-24.04
needs: scan
outputs:
patched-image: ${{ steps.copa.outputs.patched-image }}
steps:
- name: Download Trivy report
uses: actions/download-artifact@v4
with:
name: trivy-report-pre-patch
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
driver: docker-container
use: true
- name: Log in to registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Run Copa patch
id: copa
uses: project-copacetic/copa-action@v0.6.0
with:
image: ${{ needs.scan.outputs.image-ref }}
image-report: trivy-report.json
patched-tag: ${{ github.sha }}-patched
output: patched-image
buildkit-version: latest
copa-version: latest
- name: Verify Copa produced a patched image
run: |
PATCHED="${{ steps.copa.outputs.patched-image }}"
if [ -z "${PATCHED}" ]; then
echo "Copa did not produce a patched image. Checking for no-fix scenario."
exit 1
fi
echo "Patched image: ${PATCHED}"
verify:
name: Trivy Post-Patch Verification
runs-on: ubuntu-24.04
needs: patch
outputs:
residual-critical: ${{ steps.parse.outputs.residual-critical }}
steps:
- name: Log in to registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Run Trivy scan on patched image
uses: aquasecurity/trivy-action@0.28.0
with:
image-ref: ${{ needs.patch.outputs.patched-image }}
format: json
output: trivy-report-post.json
severity: CRITICAL,HIGH
exit-code: "0"
- name: Check for residual critical CVEs
id: parse
run: |
COUNT=$(jq '[.Results[]?.Vulnerabilities[]? | select(.Severity == "CRITICAL")] | length' trivy-report-post.json)
echo "residual-critical=${COUNT}" >> "$GITHUB_OUTPUT"
if [ "${COUNT}" -gt "0" ]; then
echo "ERROR: ${COUNT} CRITICAL CVEs remain after Copa patch."
jq '.Results[]?.Vulnerabilities[]? | select(.Severity == "CRITICAL") | {id: .VulnerabilityID, pkg: .PkgName, fixed: .FixedVersion}' trivy-report-post.json
exit 1
fi
echo "Verification passed: 0 CRITICAL CVEs in patched image."
- name: Upload post-patch Trivy report
uses: actions/upload-artifact@v4
with:
name: trivy-report-post-patch
path: trivy-report-post.json
retention-days: 30
sign-and-push:
name: cosign Sign and Push
runs-on: ubuntu-24.04
needs: [patch, verify]
steps:
- name: Install cosign
uses: sigstore/cosign-installer@v3
- name: Log in to registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Sign patched image with OIDC keyless signing
env:
COSIGN_EXPERIMENTAL: "1"
run: |
PATCHED="${{ needs.patch.outputs.patched-image }}"
# Resolve the image digest before signing
DIGEST=$(docker buildx imagetools inspect "${PATCHED}" --format '{{.Manifest.Digest}}')
DIGEST_REF="${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${DIGEST}"
cosign sign --yes \
--rekor-url https://rekor.sigstore.dev \
--fulcio-url https://fulcio.sigstore.dev \
"${DIGEST_REF}"
echo "Signed: ${DIGEST_REF}"
echo "PATCHED_DIGEST_REF=${DIGEST_REF}" >> "$GITHUB_ENV"
- name: Promote to production registry
run: |
# Only runs after verify job passed — crane copy is the promotion gate
crane copy \
"${{ needs.patch.outputs.patched-image }}" \
"${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest-patched"
env:
COSIGN_EXPERIMENTAL: "1"
Copa Action Inputs and Outputs
The project-copacetic/copa-action action wraps the copa patch CLI command. The key inputs:
- uses: project-copacetic/copa-action@v0.6.0
with:
image: ghcr.io/org/app:sha-abc123 # source image to patch
image-report: trivy-report.json # Trivy JSON report (--format json)
patched-tag: sha-abc123-patched # tag for the output image
output: patched-image # name of the output variable
buildkit-version: latest # BuildKit daemon version
copa-version: latest # copa binary version
timeout: "10m" # patch timeout (optional)
The action outputs a single value, patched-image, which is the full image reference including registry, repository, and the patched tag. This value is the input to both the verify and sign jobs. Passing it between jobs requires the outputs block on the patch job and needs.patch.outputs.patched-image in downstream jobs.
Copa requires BuildKit because it uses BuildKit’s image manipulation API to create a new layer with patched packages. The docker/setup-buildx-action with driver: docker-container starts a BuildKit daemon that Copa can connect to. Without this step, Copa exits with a buildkitd connection error.
Handling Copa Patch Failures
Copa can fail in two distinct ways: a hard failure (BuildKit error, registry auth failure, malformed Trivy report) and a soft failure (no fix available for one or more CVEs). Both require different handling.
The following workflow job handles Copa failures gracefully, creating a GitHub Issue when no fix is available:
patch-with-fallback:
name: Copa Patch with Failure Handling
runs-on: ubuntu-24.04
needs: scan
outputs:
patched-image: ${{ steps.copa.outputs.patched-image }}
patch-status: ${{ steps.copa-status.outputs.status }}
steps:
- name: Download Trivy report
uses: actions/download-artifact@v4
with:
name: trivy-report-pre-patch
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
with:
driver: docker-container
use: true
- name: Log in to registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Run Copa patch
id: copa
continue-on-error: true
uses: project-copacetic/copa-action@v0.6.0
with:
image: ${{ needs.scan.outputs.image-ref }}
image-report: trivy-report.json
patched-tag: ${{ github.sha }}-patched
- name: Determine patch status
id: copa-status
run: |
if [ "${{ steps.copa.outcome }}" = "success" ]; then
echo "status=patched" >> "$GITHUB_OUTPUT"
else
# Check if failure is due to no-fix-available vs hard error
# Copa exits with a non-zero code and "no updates available" message
# when all CVEs lack a fix in the upstream package repo
echo "status=no-fix" >> "$GITHUB_OUTPUT"
fi
- name: Create GitHub Issue for unfixable CVEs
if: steps.copa-status.outputs.status == 'no-fix'
uses: actions/github-script@v7
with:
script: |
const report = require('fs').readFileSync('trivy-report.json', 'utf8');
const parsed = JSON.parse(report);
const criticals = parsed.Results?.flatMap(r => r.Vulnerabilities || [])
.filter(v => v.Severity === 'CRITICAL' && !v.FixedVersion)
.map(v => `- **${v.VulnerabilityID}** in \`${v.PkgName}\` (no fix available)`)
.join('\n') || 'None';
await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title: `[Copa] Unfixable CVEs in ${context.sha.slice(0, 8)} — manual remediation required`,
labels: ['security', 'vulnerability', 'needs-triage'],
body: `## Copa patch failed: no fix available\n\n**Image:** \`${{ needs.scan.outputs.image-ref }}\`\n**Workflow run:** ${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}\n\n### CVEs with no fix available\n${criticals}\n\n### Recommended actions\n1. Check if a new base image tag has been released.\n2. Consider switching to an alternative base image.\n3. Add affected CVE IDs to \`.copa-exceptions.txt\` if accepted risk.`
});
- name: Fail if no fix and not in exceptions list
if: steps.copa-status.outputs.status == 'no-fix'
run: |
# Check known-exceptions list
if [ -f .copa-exceptions.txt ]; then
UNFIXED=$(jq -r '.Results[]?.Vulnerabilities[]? | select(.Severity == "CRITICAL" and (.FixedVersion == null or .FixedVersion == "")) | .VulnerabilityID' trivy-report.json)
while IFS= read -r cve; do
if ! grep -q "^${cve}$" .copa-exceptions.txt; then
echo "CVE ${cve} is not in the exceptions list. Failing pipeline."
exit 1
fi
done <<< "${UNFIXED}"
echo "All unfixable CVEs are in the exceptions list. Continuing."
else
echo "No exceptions list found. Failing pipeline due to unfixable CVEs."
exit 1
fi
The .copa-exceptions.txt file is a plain-text list of CVE IDs (one per line) that have been reviewed and accepted as known exceptions, typically because no fix exists and the application code does not exercise the vulnerable code path. This file should be committed to the repository and reviewed in pull requests.
Promotion Gate
The promotion gate is enforced by job dependency ordering. The sign-and-push job only runs if the verify job passes. If Trivy finds residual CRITICAL CVEs after patching, the verify job exits non-zero, and the sign-and-push job is skipped.
For multi-environment promotion, use crane copy with explicit digest pinning:
promote-to-production:
name: Promote Patched Image to Production Registry
runs-on: ubuntu-24.04
needs: [verify, sign-and-push]
if: needs.verify.outputs.residual-critical == '0'
environment: production
steps:
- name: Install crane
uses: imjasonh/setup-crane@v0.4
- name: Log in to staging registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Log in to production registry
uses: docker/login-action@v3
with:
registry: registry.prod.example.com
username: ${{ secrets.PROD_REGISTRY_USER }}
password: ${{ secrets.PROD_REGISTRY_TOKEN }}
- name: Copy patched image to production registry
run: |
PATCHED="${{ needs.patch.outputs.patched-image }}"
# Resolve digest to avoid tag mutability
DIGEST=$(crane digest "${PATCHED}")
SRC="${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${DIGEST}"
DST="registry.prod.example.com/${{ env.IMAGE_NAME }}:${{ github.sha }}-patched"
crane copy "${SRC}" "${DST}"
echo "Promoted ${SRC} to ${DST}"
The environment: production gate adds a required reviewer approval step if configured in repository settings — no patched image reaches production without both Copa verification passing and a human approval.
Scheduled Nightly Re-Patching Pipeline
This separate workflow re-scans and re-patches all production images on a schedule, closing the gap between build time and new CVE disclosure:
# .github/workflows/copa-scheduled-patch.yml
name: Nightly Copa Re-Patch
on:
schedule:
- cron: "0 2 * * *" # 02:00 UTC nightly
workflow_dispatch:
permissions:
contents: read
packages: write
id-token: write
issues: write
jobs:
list-production-images:
name: List Production Images
runs-on: ubuntu-24.04
outputs:
matrix: ${{ steps.build-matrix.outputs.matrix }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Build image matrix from production manifest
id: build-matrix
run: |
# Read production images from a manifest file
# Format: one image reference per line
IMAGES=$(jq -R -s 'split("\n") | map(select(length > 0))' production-images.txt)
echo "matrix={\"image\": ${IMAGES}}" >> "$GITHUB_OUTPUT"
nightly-patch:
name: Patch ${{ matrix.image }}
runs-on: ubuntu-24.04
needs: list-production-images
strategy:
matrix: ${{ fromJson(needs.list-production-images.outputs.matrix) }}
fail-fast: false # patch all images even if one fails
steps:
- name: Log in to registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Run Trivy scan
uses: aquasecurity/trivy-action@0.28.0
with:
image-ref: ${{ matrix.image }}
format: json
output: trivy-report.json
severity: CRITICAL,HIGH
exit-code: "0"
- name: Check if patching is needed
id: check
run: |
COUNT=$(jq '[.Results[]?.Vulnerabilities[]? | select(.Severity == "CRITICAL")] | length' trivy-report.json)
echo "needs-patch=$([ "${COUNT}" -gt 0 ] && echo true || echo false)" >> "$GITHUB_OUTPUT"
echo "critical-count=${COUNT}" >> "$GITHUB_OUTPUT"
- name: Set up Docker Buildx
if: steps.check.outputs.needs-patch == 'true'
uses: docker/setup-buildx-action@v3
with:
driver: docker-container
use: true
- name: Run Copa patch
if: steps.check.outputs.needs-patch == 'true'
id: copa
continue-on-error: true
uses: project-copacetic/copa-action@v0.6.0
with:
image: ${{ matrix.image }}
image-report: trivy-report.json
patched-tag: nightly-${{ github.run_id }}
- name: Verify and sign if patched
if: steps.check.outputs.needs-patch == 'true' && steps.copa.outcome == 'success'
run: |
PATCHED="${{ steps.copa.outputs.patched-image }}"
# Re-scan
trivy image --format json -o trivy-post.json --severity CRITICAL "${PATCHED}"
COUNT=$(jq '[.Results[]?.Vulnerabilities[]? | select(.Severity == "CRITICAL")] | length' trivy-post.json)
if [ "${COUNT}" -gt "0" ]; then
echo "Residual CVEs after nightly patch: ${COUNT}"
exit 1
fi
# Sign
cosign sign --yes "${PATCHED}"
echo "Nightly patch complete for ${{ matrix.image }}"
env:
COSIGN_EXPERIMENTAL: "1"
Tekton Pipeline Implementation
For teams running Tekton Pipelines on Kubernetes, the same scan-patch-verify-sign loop maps to four Tasks connected by a Pipeline:
# tekton/tasks/trivy-scan-task.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: trivy-scan
namespace: cicd
spec:
params:
- name: image-ref
type: string
description: Full image reference to scan
- name: severity
type: string
default: "CRITICAL,HIGH"
results:
- name: critical-count
description: Number of CRITICAL CVEs found
workspaces:
- name: reports
description: Shared workspace for scan reports
steps:
- name: scan
image: aquasec/trivy:0.50.4
script: |
#!/usr/bin/env sh
trivy image \
--format json \
--output "$(workspaces.reports.path)/trivy-report.json" \
--severity "$(params.severity)" \
--exit-code 0 \
"$(params.image-ref)"
COUNT=$(jq '[.Results[]?.Vulnerabilities[]? | select(.Severity == "CRITICAL")] | length' \
"$(workspaces.reports.path)/trivy-report.json")
printf '%s' "${COUNT}" > "$(results.critical-count.path)"
# tekton/tasks/copa-patch-task.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: copa-patch
namespace: cicd
spec:
params:
- name: image-ref
type: string
- name: patched-tag
type: string
results:
- name: patched-image
description: Full reference of the patched image
workspaces:
- name: reports
description: Workspace containing trivy-report.json
- name: dockerconfig
description: Docker config for registry auth
mountPath: /kaniko/.docker
sidecars:
- name: buildkitd
image: moby/buildkit:v0.12.5-rootless
securityContext:
seccompProfile:
type: Unconfined
runAsUser: 1000
runAsGroup: 1000
readinessProbe:
exec:
command: ["buildctl", "debug", "workers"]
initialDelaySeconds: 5
periodSeconds: 3
steps:
- name: patch
image: ghcr.io/project-copacetic/copacetic:v0.6.0
script: |
#!/usr/bin/env sh
copa patch \
--image "$(params.image-ref)" \
--report "$(workspaces.reports.path)/trivy-report.json" \
--addr buildkit:///run/buildkit/buildkitd.sock \
--tag "$(params.patched-tag)" \
--output "$(results.patched-image.path)"
env:
- name: DOCKER_CONFIG
value: /kaniko/.docker
# tekton/tasks/cosign-sign-task.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: cosign-sign
namespace: cicd
spec:
params:
- name: image-ref
type: string
description: Image reference to sign (should be digest-pinned)
steps:
- name: sign
image: gcr.io/projectsigstore/cosign:v2.2.3
script: |
#!/usr/bin/env sh
cosign sign --yes \
--rekor-url https://rekor.sigstore.dev \
--fulcio-url https://fulcio.sigstore.dev \
"$(params.image-ref)"
env:
- name: COSIGN_EXPERIMENTAL
value: "1"
# tekton/pipelines/copa-pipeline.yaml
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: copa-patch-pipeline
namespace: cicd
spec:
params:
- name: image-ref
type: string
- name: patched-tag
type: string
workspaces:
- name: reports
- name: dockerconfig
tasks:
- name: scan
taskRef:
name: trivy-scan
params:
- name: image-ref
value: $(params.image-ref)
workspaces:
- name: reports
workspace: reports
- name: patch
taskRef:
name: copa-patch
runAfter: [scan]
params:
- name: image-ref
value: $(params.image-ref)
- name: patched-tag
value: $(params.patched-tag)
workspaces:
- name: reports
workspace: reports
- name: dockerconfig
workspace: dockerconfig
- name: verify
taskRef:
name: trivy-scan
runAfter: [patch]
params:
- name: image-ref
value: $(tasks.patch.results.patched-image)
- name: severity
value: "CRITICAL"
workspaces:
- name: reports
workspace: reports
- name: sign
taskRef:
name: cosign-sign
runAfter: [verify]
params:
- name: image-ref
value: $(tasks.patch.results.patched-image)
results:
- name: patched-image
value: $(tasks.patch.results.patched-image)
- name: post-patch-critical-count
value: $(tasks.verify.results.critical-count)
The Tekton Pipeline passes the patched image reference between tasks using tasks.patch.results.patched-image. The verify Task re-runs the TrivyScanTask against this output rather than the original image. The runAfter field enforces sequential execution.
Expected Behaviour
The following table describes pipeline outcomes for the primary scenarios:
| Scenario | Copa Action | Pipeline Outcome | Notification |
|---|---|---|---|
| No CVEs found in initial Trivy scan | Copa is skipped; original image is promoted | Pipeline passes; original image is signed and promoted | None required |
| CVEs found; Copa patches all successfully; re-scan clean | Copa patches; verify passes | Pipeline passes; patched image is signed and promoted | Slack/PR comment with CVE count patched |
| CVEs found; Copa patches some; re-scan shows residual CRITICAL CVEs | Copa exits zero (partial patch) | verify job fails; pipeline stops; no promotion |
GitHub Actions job failure notification; Trivy report uploaded as artifact |
| CVEs found; no fix available in upstream package repo | Copa exits non-zero; patch job fails |
patch-with-fallback creates GitHub Issue; checks exceptions list; fails pipeline unless CVEs are in exceptions list |
GitHub Issue created with CVE details and remediation options |
| Copa exits non-zero due to BuildKit error | Hard failure | patch job fails; pipeline stops |
GitHub Actions job failure; runner logs contain BuildKit error |
| Cosign OIDC auth fails | cosign exits non-zero | sign-and-push job fails; patched image is in registry but unsigned |
GitHub Actions job failure; operator must investigate OIDC configuration |
Trade-offs
| Dimension | Option A | Option B | Recommendation |
|---|---|---|---|
| Copa placement in pipeline | Copa runs in the main build pipeline after every push | Copa runs in a separate scheduled pipeline (nightly/weekly) | Use both: main pipeline for new CVEs introduced by the PR; scheduled pipeline for CVEs disclosed after build |
| GitHub Actions vs. Tekton for Copa | GitHub Actions: simpler setup, hosted runners, no BuildKit sidecar complexity | Tekton: Kubernetes-native, integrates with Tekton Chains for supply chain attestation, more control over runner resources | GitHub Actions for most teams; Tekton when you already run Tekton Pipelines and need Chains integration |
| copa-action vs. manual copa CLI steps | copa-action abstracts BuildKit setup, simplifies version pinning, provides structured outputs |
Manual copa patch CLI gives full control over flags, easier to debug, no dependency on third-party action |
Use copa-action for standard cases; use manual CLI when you need custom BuildKit socket paths or non-standard registry configurations |
| Scan-then-patch vs. build-then-scan-patch | Scan the base image before building; patch only the base if needed | Build the full application image first; scan and patch the final image | Scan-patch the final image: application layers can introduce OS packages that the base-only scan misses; Copa operates on the final image regardless |
| Keyless vs. key-based cosign signing | Keyless OIDC: no key management, signatures tied to GitHub OIDC identity, ephemeral certificates | Key-based: works outside GitHub Actions, portable, requires key management and rotation | Keyless in GitHub Actions; key-based in Tekton or self-hosted runners where OIDC is not available |
Failure Modes
| Failure | Symptom | Root Cause | Fix |
|---|---|---|---|
| BuildKit not available in runner | Copa exits: failed to connect to buildkitd: context deadline exceeded |
docker/setup-buildx-action not called before copa-action, or driver: docker-container not set |
Add docker/setup-buildx-action@v3 with driver: docker-container before the Copa step; verify the action runs before Copa in the job |
| Trivy report format version mismatch with Copa | Copa exits: unsupported report schema version or failed to parse vulnerability report |
Trivy JSON schema changes between Trivy versions; Copa expects a specific schema revision | Pin Trivy to a version known to work with your Copa version; check Copa release notes for supported Trivy versions; use --format json not --format sarif |
| cosign OIDC fails in non-GitHub runner | cosign exits: error getting Fulcio roots: failed to get certificate or OIDC token fetch error |
ACTIONS_ID_TOKEN_REQUEST_URL is not set outside GitHub-hosted runners; self-hosted runners behind a proxy may block Fulcio/Rekor endpoints |
On self-hosted runners, verify outbound access to fulcio.sigstore.dev and rekor.sigstore.dev; add id-token: write to workflow permissions; consider key-based signing as fallback |
| Patched image digest not passed correctly between jobs | Downstream job uses original image reference; verify scans original image not patched image; false clean result | outputs block on patch job missing or incorrectly named; steps.copa.outputs.patched-image referenced before Copa action completes |
Verify the patch job has outputs: patched-image: ${{ steps.copa.outputs.patched-image }}; verify downstream jobs reference needs.patch.outputs.patched-image not a hardcoded tag |
| Copa patches but does not update the manifest index | crane copy pulls the wrong architecture |
Copa creates a single-platform patched image; multi-arch source manifests result in a single-arch patched image | Set COPA_BUILDKIT_PLATFORM env var or pass --platform to Copa; run Copa once per platform and re-assemble a manifest list with crane index append |
| Registry rate limiting during nightly matrix | Multiple parallel Copa jobs hit registry pull rate limits | Nightly scheduled job runs fail-fast: false with many images simultaneously |
Add max-parallel: 3 to the matrix strategy; use a registry mirror or pull-through cache; add exponential backoff retry logic around the Trivy scan step |