Copa in Air-Gapped Environments: Container Patching Without Internet Access

The Problem

Copa (Copacetic) patches container images by fetching updated OS packages at runtime — it runs Trivy to identify vulnerable packages in a container layer, then calls the package manager inside a BuildKit worker to pull and apply those updates. The patched layer is committed back to the image and pushed to a registry. The whole loop takes seconds on an internet-connected build host.

In air-gapped environments, every external network call in that loop fails. The apt/apk/yum fetch step cannot reach upstream mirrors. Trivy’s vulnerability database cannot be updated from db.trivy.aquasec.com. The image push targets a registry that may be internal but cannot pull base layers from Docker Hub. Change control processes in these environments add further friction: importing new packages may require a formal approval cycle that can take days to weeks.

The environments where this matters most are exactly the ones with the lowest tolerance for unpatched containers. Defence networks classified at IL4/IL5 or above run containerised workloads on Kubernetes but operate behind unidirectional gateways or physical data diodes. Financial trading infrastructure has microsecond-latency constraints that preclude external network calls during operational hours. ICS/OT environments operate under strict network isolation rules with explicit inventory of every permitted data flow. Government compute environments following NIST SP 800-53 SC-7 boundary controls may allow no egress whatsoever from the production zone.

The standard Copa workflow assumes internet connectivity at three points: Trivy database update, OS package fetch, and image push. All three must be redirected to internal equivalents before Copa can function in an air-gapped zone. This article builds the complete architecture for doing that — a staging zone that prepares patch bundles and vulnerability databases, a one-way transfer pipeline into the production zone, and the Copa/BuildKit/registry stack that operates entirely on internal resources.

Threat Model

CVE disclosure followed by delayed patching due to import backlog. A critical CVE is disclosed against libssl in an Alpine-based container running in the production zone. The patched package is available in the Alpine CDN within hours. In an air-gapped environment without a defined import pipeline, the patching cycle depends on when an administrator next performs a manual import run — potentially days later — and then on whether the change control board has scheduled the import for that window. During that interval the container is running a known-vulnerable package. An attacker with access to the environment — whether via a compromised endpoint or a persistent presence pre-dating the isolation — can exploit the window. The risk is compounded because security teams may not realise that the Trivy scan result from the previous day remains valid: the image has not been patched since the last scan.

Stale Trivy vulnerability database producing false negatives. Trivy’s offline database is a snapshot. If the snapshot was taken before a CVE was published to NVD, Trivy will not detect the vulnerability. In internet-connected environments, trivy db update runs daily and the database is never more than 24 hours stale. In an air-gapped environment where the database is manually imported on an ad-hoc basis, the gap can extend to weeks. An image scanned with a 30-day-old Trivy database may appear clean when it has 15 undetected CVEs against packages disclosed in the last month. The security team acts on “no findings” as an indicator of hygiene when the scanner is blind.

Data diode misconfiguration allowing bidirectional traffic through the patching import channel. The import pipeline introduces a controlled data flow from the staging zone into the production zone. If that channel is misconfigured — for example, a firewall rule that allows traffic in both directions, or a transfer host that is dual-homed and running an SSH server accessible from both zones — an attacker who compromises an asset in the production zone can use the import channel to exfiltrate data or reach the internet through the staging zone. The import pipeline must be the most tightly controlled data flow into the production zone: receive-only, with cryptographic integrity verification on every transferred artifact.

Architecture and Implementation

Zone Architecture

The air-gapped patching architecture uses two zones with a one-way transfer boundary between them.

The staging zone has internet access, at minimum outbound HTTPS to OS package mirrors and Trivy’s database endpoint. It runs:

• An apt-mirror or aptly instance syncing Debian/Ubuntu repositories
• An apk fetch job pulling Alpine packages into a local HTTP server
• A scheduled Trivy database download job
• A bundle generation pipeline that packages the above as OCI artifacts
• A transfer agent that copies artifacts across the boundary using the data diode mechanism

The production zone has no internet access. It runs:

• A private OCI registry (Harbor or Zot) receiving the transferred artifacts
• A local HTTP server or Nexus proxy serving OS packages from the imported mirror snapshot
• A BuildKit daemon (buildkitd) configured to use internal package sources only
• A Copa job runner that scans and patches images using internal Trivy DB and internal package mirrors
• Kubernetes network policies restricting Copa/BuildKit pods to internal IP ranges only

┌─────────────────────────────────────────────────────────────────────┐
│                        STAGING ZONE                                 │
│  ┌──────────────┐   ┌──────────────┐   ┌────────────────────────┐  │
│  │  apt-mirror   │   │  apk fetcher │   │  trivy db downloader   │  │
│  │  (daily sync) │   │  (daily sync)│   │  (daily snapshot)      │  │
│  └──────┬───────┘   └──────┬───────┘   └───────────┬────────────┘  │
│         │                  │                        │               │
│         └──────────────────┴────────────────────────┘               │
│                             │                                        │
│                    ┌────────▼────────┐                              │
│                    │  bundle packer  │  crane export → tarball      │
│                    │  + sha256sum    │                              │
│                    └────────┬────────┘                              │
└─────────────────────────────┼───────────────────────────────────────┘
                              │  DATA DIODE (receive-only → prod)
                              ▼
┌─────────────────────────────────────────────────────────────────────┐
│                      PRODUCTION ZONE                                │
│  ┌────────────────┐  ┌───────────┐  ┌──────────────┐              │
│  │ import verifier│  │  Harbor/  │  │  Nexus proxy  │              │
│  │ (sha256 check) │→ │   Zot     │  │  (pkg mirror) │              │
│  └────────────────┘  └─────┬─────┘  └──────┬───────┘              │
│                             │               │                        │
│                    ┌────────▼───────────────▼───────┐              │
│                    │   Copa + BuildKit + Trivy       │              │
│                    │   (all internal endpoints)      │              │
│                    └─────────────────────────────────┘              │
└─────────────────────────────────────────────────────────────────────┘

Mirroring apt Repositories (Debian/Ubuntu)

Use aptly rather than apt-mirror for air-gapped work. aptly creates immutable snapshots that can be published at a stable URL and versioned, which is critical for reproducibility: a Copa patch run in week 1 and week 2 should use the same package versions unless the snapshot is explicitly updated.

Install aptly in the staging zone and create a managed mirror:

# /etc/aptly.conf (staging zone)
{
  "rootDir": "/srv/aptly",
  "downloadConcurrency": 4,
  "architectures": ["amd64", "arm64"],
  "gpgDisableSign": false,
  "gpgDisableVerify": false
}

# Create a mirror of the Ubuntu 22.04 security repo
aptly mirror create \
  ubuntu-jammy-security \
  http://security.ubuntu.com/ubuntu \
  jammy-security \
  main restricted universe

# Create a mirror of the main repo for dependency resolution
aptly mirror create \
  ubuntu-jammy-main \
  http://archive.ubuntu.com/ubuntu \
  jammy jammy-updates \
  main restricted universe multiverse

# Update and snapshot
aptly mirror update ubuntu-jammy-security
aptly mirror update ubuntu-jammy-main

# Create a dated snapshot — do this on the daily sync schedule
SNAP_DATE=$(date +%Y%m%d)
aptly snapshot create "ubuntu-jammy-security-${SNAP_DATE}" \
  from mirror ubuntu-jammy-security
aptly snapshot create "ubuntu-jammy-main-${SNAP_DATE}" \
  from mirror ubuntu-jammy-main

# Merge and publish
aptly snapshot merge "ubuntu-jammy-${SNAP_DATE}" \
  "ubuntu-jammy-security-${SNAP_DATE}" \
  "ubuntu-jammy-main-${SNAP_DATE}"

aptly publish snapshot \
  -distribution=jammy \
  -component=main \
  "ubuntu-jammy-${SNAP_DATE}" \
  filesystem:prod-mirror:ubuntu

The published snapshot is available under /srv/aptly/public/ as a complete apt repository tree. This directory is the artifact to transfer to the production zone.

Mirroring apk Repositories (Alpine)

Alpine package management does not have a direct mirror tool, but apk fetch can pull packages by name. For a complete mirror, use wget in recursive mode against the Alpine CDN:

#!/usr/bin/env bash
# /opt/scripts/mirror-alpine.sh (staging zone, runs daily)
set -euo pipefail

ALPINE_VERSION="3.19"
ARCH="x86_64"
MIRROR_ROOT="/srv/alpine-mirror"
ALPINE_CDN="https://dl-cdn.alpinelinux.org/alpine"

for repo in main community; do
  DEST="${MIRROR_ROOT}/v${ALPINE_VERSION}/${repo}/${ARCH}"
  mkdir -p "${DEST}"
  wget \
    --mirror \
    --no-parent \
    --no-host-directories \
    --cut-dirs=3 \
    --reject "index.html*" \
    --directory-prefix="${DEST}" \
    "${ALPINE_CDN}/v${ALPINE_VERSION}/${repo}/${ARCH}/"
done

# Regenerate the APKINDEX for the local mirror
# apk does not require this if serving the original files, but verify checksums
find "${MIRROR_ROOT}" -name "*.apk" -newer "${MIRROR_ROOT}/.last_sync" \
  | wc -l | xargs -I{} echo "New packages: {}"

touch "${MIRROR_ROOT}/.last_sync"

Serve the mirror root with a simple nginx container in the production zone after transfer:

# /etc/nginx/conf.d/alpine-mirror.conf (production zone)
server {
    listen 8080;
    server_name alpine-mirror.internal;

    root /srv/alpine-mirror;
    autoindex on;

    location / {
        try_files $uri $uri/ =404;
    }
}

Syncing the Trivy Offline Vulnerability Database

Trivy’s offline database is distributed as an OCI artifact. Download it in the staging zone and bundle it for transfer:

#!/usr/bin/env bash
# /opt/scripts/sync-trivy-db.sh (staging zone)
set -euo pipefail

DB_DIR="/srv/trivy-db"
mkdir -p "${DB_DIR}"

# Download the full vulnerability database (not just the light DB)
# trivy db download requires TRIVY_CACHE_DIR to be set
export TRIVY_CACHE_DIR="${DB_DIR}"
trivy image --download-db-only --no-progress

# The DB is now at ${DB_DIR}/db/trivy.db
# Bundle it as a tar.gz with a manifest for integrity checking
DB_DATE=$(date +%Y%m%d-%H%M%S)
BUNDLE_NAME="trivy-db-${DB_DATE}.tar.gz"

tar -czf "/srv/bundles/${BUNDLE_NAME}" \
  -C "${DB_DIR}" db/

sha256sum "/srv/bundles/${BUNDLE_NAME}" \
  > "/srv/bundles/${BUNDLE_NAME}.sha256"

echo "Bundle ready: /srv/bundles/${BUNDLE_NAME}"

Alternatively, use crane to export the Trivy DB OCI artifact directly, which preserves the OCI manifest structure that Trivy expects:

# Pull the Trivy DB OCI artifact and export it as a tarball
crane pull \
  ghcr.io/aquasecurity/trivy-db:2 \
  /srv/bundles/trivy-db-oci-$(date +%Y%m%d).tar

sha256sum /srv/bundles/trivy-db-oci-$(date +%Y%m%d).tar \
  > /srv/bundles/trivy-db-oci-$(date +%Y%m%d).tar.sha256

In the production zone, import it directly into the internal registry:

# Production zone: import and push to internal Harbor
crane push \
  /srv/imports/trivy-db-oci-20260509.tar \
  harbor.internal.corp/security/trivy-db:2

# Configure Trivy to use the internal registry for DB
export TRIVY_DB_REPOSITORY=harbor.internal.corp/security/trivy-db:2
export TRIVY_SKIP_DB_UPDATE=false  # still update, but from internal source

Private BuildKit Instance in the Air-Gapped Zone

BuildKit is the image building backend Copa uses for applying patches. Run buildkitd as a Kubernetes DaemonSet with access only to internal network resources:

# buildkitd-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: buildkitd
  namespace: copa-system
spec:
  selector:
    matchLabels:
      app: buildkitd
  template:
    metadata:
      labels:
        app: buildkitd
    spec:
      hostPID: false
      hostNetwork: false
      containers:
      - name: buildkitd
        image: harbor.internal.corp/moby/buildkit:v0.15.2-rootless
        args:
          - --addr=unix:///run/buildkit/buildkitd.sock
          - --addr=tcp://0.0.0.0:1234
          - --allow-insecure-entitlement=network.host=false
          - --allow-insecure-entitlement=security.insecure=false
        securityContext:
          seccompProfile:
            type: Unconfined
          runAsUser: 1000
          runAsNonRoot: true
        volumeMounts:
        - name: run-buildkit
          mountPath: /run/buildkit
        - name: buildkit-config
          mountPath: /etc/buildkit
      volumes:
      - name: run-buildkit
        emptyDir: {}
      - name: buildkit-config
        configMap:
          name: buildkit-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: buildkit-config
  namespace: copa-system
data:
  buildkitd.toml: |
    [worker.oci]
      enabled = true
      gc = true

    [registry."harbor.internal.corp"]
      http = false
      insecure = false
      ca = ["/etc/ssl/certs/internal-ca.crt"]

Running Copa with Internal Package Mirrors

Copa uses environment variables to override the package manager proxy. For apt-based images, Copa respects http_proxy / https_proxy / no_proxy when set. Point these at the internal Nexus repository manager or the aptly server:

#!/usr/bin/env bash
# /opt/scripts/copa-patch.sh (production zone)
set -euo pipefail

IMAGE="${1:?usage: copa-patch.sh <image:tag>}"
PATCHED_IMAGE="${IMAGE%-*}-patched:$(date +%Y%m%d)"

# Internal endpoints
TRIVY_SERVER="http://trivy.copa-system.svc.cluster.local:4954"
BUILDKIT_ADDR="tcp://buildkitd.copa-system.svc.cluster.local:1234"
INTERNAL_REGISTRY="harbor.internal.corp"

# Trivy scan using internal DB — produce the report Copa needs
trivy image \
  --server "${TRIVY_SERVER}" \
  --format json \
  --output /tmp/trivy-report.json \
  --ignore-unfixed \
  --pkg-types os \
  "${IMAGE}"

# Run Copa against the BuildKit instance, using the internal apt proxy
# Copa inherits these env vars and passes them to BuildKit worker processes
export http_proxy="http://nexus.internal.corp:8081/repository/apt-proxy/"
export https_proxy="http://nexus.internal.corp:8081/repository/apt-proxy/"
export no_proxy="harbor.internal.corp,buildkitd.copa-system.svc.cluster.local"

# For Alpine images Copa uses the apk package manager:
# point APK_REPOSITORY at the internal mirror via a custom /etc/apk/repositories
# injected at build time. Copa supports --ignore-errors to skip packages
# that cannot be fetched — do NOT use this flag in production; it masks failures.

copa patch \
  --image "${IMAGE}" \
  --report /tmp/trivy-report.json \
  --output "${PATCHED_IMAGE}" \
  --addr "${BUILDKIT_ADDR}" \
  --timeout 15m

echo "Patched image: ${PATCHED_IMAGE}"

# Push to internal registry
crane push "${PATCHED_IMAGE}" \
  "${INTERNAL_REGISTRY}/workloads/${PATCHED_IMAGE##*/}"

A note on --ignore-errors: Copa’s --ignore-errors flag instructs Copa to continue patching if a specific package cannot be fetched. In internet-connected environments this is sometimes used for packages not available in the standard repos. In air-gapped environments, do not use this flag. A fetch failure in an air-gapped Copa run almost always means the package mirror is stale or the package is missing from the import bundle. Silently skipping that package means the vulnerability is not patched and the patched image is pushed as if it were clean. Treat every fetch failure as a mirror synchronisation failure that must be investigated.

One-Way Data Transfer Pipeline

The transfer mechanism depends on the physical isolation technology in use. Common patterns:

Unidirectional security gateway (data diode). Products such as Owl Cyber Defense, Waterfall Security, and Forcepoint ONE handle physical one-way transfer. The staging zone writes artifacts to a shared folder or SFTP endpoint on the high-side of the diode; the production zone reads from the low-side receive path. The diode hardware enforces the direction in the optical layer — no software misconfiguration can reverse it.

For software-defined isolation (firewall policy rather than physical diode), implement the transfer as a pull-only mechanism from the production zone:

#!/usr/bin/env bash
# /opt/scripts/import-bundles.sh (production zone — runs on import host)
set -euo pipefail

STAGING_SFTP="import-user@staging-transfer.corp:/srv/bundles"
IMPORT_DIR="/srv/imports"
REGISTRY="harbor.internal.corp"

# Pull new bundles from staging (SFTP with host key pinning, no password)
rsync \
  --archive \
  --checksum \
  --compress \
  --rsh="ssh -i /etc/import-keys/import_ed25519 \
    -o StrictHostKeyChecking=yes \
    -o UserKnownHostsFile=/etc/import-keys/known_hosts \
    -o PasswordAuthentication=no" \
  "${STAGING_SFTP}/" \
  "${IMPORT_DIR}/"

# Verify every bundle before import
for bundle in "${IMPORT_DIR}"/*.tar; do
  sha_file="${bundle}.sha256"
  if [[ ! -f "${sha_file}" ]]; then
    echo "ERROR: No checksum file for ${bundle}" >&2
    exit 1
  fi
  if ! sha256sum --check --strict "${sha_file}"; then
    echo "ERROR: Checksum mismatch for ${bundle}" >&2
    exit 1
  fi

  # Import OCI artifact into internal registry
  artifact_name=$(basename "${bundle}" .tar)
  case "${artifact_name}" in
    trivy-db-oci-*)
      crane push "${bundle}" \
        "${REGISTRY}/security/trivy-db:2"
      ;;
    ubuntu-mirror-*)
      # Extract and rsync into Nexus raw repository via API
      tar -xzf "${bundle}" -C /srv/pkg-mirror/ubuntu/
      ;;
    alpine-mirror-*)
      tar -xzf "${bundle}" -C /srv/pkg-mirror/alpine/
      ;;
  esac

  # Move to processed
  mv "${bundle}" "${IMPORT_DIR}/processed/"
  mv "${sha_file}" "${IMPORT_DIR}/processed/"
done

The import host must have no route back to the staging zone other than the approved SFTP path. Add an explicit firewall rule blocking all outbound traffic from the import host except TCP 22 to the staging SFTP server, and block that after the import window closes.

Network Policy for Copa and BuildKit Pods

Kubernetes NetworkPolicy restricts Copa and BuildKit egress to only the internal endpoints they require:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: copa-egress-internal-only
  namespace: copa-system
spec:
  podSelector:
    matchLabels:
      app: copa
  policyTypes:
  - Egress
  egress:
  # Internal OCI registry
  - ports:
    - port: 443
      protocol: TCP
    to:
    - ipBlock:
        cidr: 10.100.10.50/32  # Harbor VIP
  # Internal package mirror (Nexus)
  - ports:
    - port: 8081
      protocol: TCP
    to:
    - ipBlock:
        cidr: 10.100.10.60/32  # Nexus
  # BuildKit
  - ports:
    - port: 1234
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: copa-system
      podSelector:
        matchLabels:
          app: buildkitd
  # Trivy server
  - ports:
    - port: 4954
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: copa-system
      podSelector:
        matchLabels:
          app: trivy-server
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: buildkit-egress-internal-only
  namespace: copa-system
spec:
  podSelector:
    matchLabels:
      app: buildkitd
  policyTypes:
  - Egress
  egress:
  # Package mirror only — no public internet
  - ports:
    - port: 8081
      protocol: TCP
    to:
    - ipBlock:
        cidr: 10.100.10.60/32
  # Alpine mirror nginx
  - ports:
    - port: 8080
      protocol: TCP
    to:
    - ipBlock:
        cidr: 10.100.10.61/32
  # Internal registry (to push patched layers)
  - ports:
    - port: 443
      protocol: TCP
    to:
    - ipBlock:
        cidr: 10.100.10.50/32
  # CoreDNS for internal name resolution
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
    to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns

Expected Behaviour

The following table maps each Copa/Trivy operation to its internet-connected behaviour and the air-gapped equivalent, with the verification step for each:

Trade-offs

Failure Modes

Related Articles

• Copa for Distroless Images: Patching Without a Shell
• Copa CI/CD Patch Automation: Integrating Copacetic into Your Pipeline
• Artifact Registry Security: Hardening Your Container Image Supply Chain
• Container Patch SLA Policy Enforcement: Automating Compliance Gates
• Zero Trust Network Access with WireGuard: Replacing VPN with Per-Resource Tunnels