Automated ingress-nginx Version Management and CVE Response

Problem

The ingress-nginx controller has one of the highest CVE rates among core Kubernetes ecosystem components. CVE-2021-25742 allowed annotation injection to modify NGINX configuration and expose secrets. CVE-2023-5044 (nginx.ingress.kubernetes.io/configuration-snippet) allowed arbitrary NGINX config injection. The ingress-nginx advisory from 2025 (CVE-2025-1097, CVE-2025-1098, CVE-2025-1974) disclosed multiple annotation injection paths that could expose all secrets in the cluster.

Unlike application CVEs, ingress-nginx CVEs affect the boundary component that all inbound traffic passes through. A vulnerable ingress controller can leak TLS private keys, inject malicious headers into upstream requests, or allow unauthenticated access to internal services. The blast radius is cluster-wide.

The operational problem is that ingress-nginx version management is typically manual or semi-automated:

Helm chart pinning creates drift. Teams pin to a specific chart version for stability, then fall behind. A cluster running ingress-nginx 1.8.0 today may not have been updated in six months. When a critical CVE is published, the team must identify the current version across every cluster, determine the patched version, test the upgrade, and roll it out — often taking one to three weeks.

Multi-cluster environments magnify the problem. A platform team managing 10+ clusters cannot manually track ingress-nginx versions across all of them. Version drift means different clusters are at different risk levels, and there is no centralised visibility into which clusters are running vulnerable versions.

GitOps repositories accumulate stale versions. Helm chart versions in values.yaml or Chart.yaml become stale silently. There is no automatic notification when a new version is available, and there is no link between a CVE announcement and the specific chart version that patches it.

Canary risk in a shared ingress. A single ingress controller serves all namespaces in a cluster. An upgrade that breaks the controller takes down all inbound traffic, not just one service. Teams are reluctant to upgrade frequently because the risk of outage affects every tenant.

Target systems: any Kubernetes cluster using ingress-nginx; multi-cluster platforms where ingress-nginx is managed via Helm and GitOps; platform teams responsible for shared ingress infrastructure.

Threat Model

Adversary 1 — CVE exploitation before patch. A critical ingress-nginx CVE is published. Clusters running the vulnerable version are actively targeted. The window between CVE publication and patch deployment is measured in weeks due to manual processes. Attackers exploit annotation injection to read cluster secrets via the NGINX configuration.

Adversary 2 — Annotation injection via Ingress object. A user with CREATE Ingress permission in any namespace uses the nginx.ingress.kubernetes.io/configuration-snippet annotation to inject NGINX config that proxies internal traffic to an external server. Without annotation validation and controller version management, this vulnerability persists.

Adversary 3 — Supply chain substitution. A malicious Helm chart version is published to a compromised chart repository. Without pinned chart digests and signature verification, automated update tools pull and deploy the malicious version.

Configuration / Implementation

Step 1 — Establish current version inventory across clusters

#!/bin/bash
# ingress-nginx-inventory.sh
# Report ingress-nginx version across all clusters in kubeconfig

CLUSTERS=$(kubectl config get-contexts -o name 2>/dev/null)

echo "Cluster | Namespace | Chart Version | App Version | Image Tag"
echo "--------|-----------|---------------|-------------|----------"

for cluster in $CLUSTERS; do
  kubectl config use-context "$cluster" &>/dev/null
  
  # Find all ingress-nginx Helm releases
  helm list -A 2>/dev/null | grep "ingress-nginx" | while read -r release namespace revision updated status chart app_version; do
    # Get the controller image tag
    image_tag=$(kubectl get deployment -n "$namespace" \
      -l "app.kubernetes.io/name=ingress-nginx" \
      -o jsonpath='{.items[0].spec.template.spec.containers[0].image}' 2>/dev/null | \
      cut -d: -f2)
    
    echo "$cluster | $namespace | $chart | $app_version | $image_tag"
  done
done

# Check if any clusters are running known-vulnerable versions
# CVE-2025-1097/1098/1974 affect versions < 1.12.1 and < 1.11.5
VULNERABLE_VERSIONS=("1.8" "1.9" "1.10" "1.11.0" "1.11.1" "1.11.2" "1.11.3" "1.11.4")

kubectl get pods -n ingress-nginx \
  -o jsonpath='{.items[*].spec.containers[*].image}' | \
  tr ' ' '\n' | grep "ingress-nginx/controller" | sort -u
# Compare output against vulnerable version list

Step 2 — Configure Renovate for automated Helm chart updates

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:base"],
  
  "helm-values": {
    "fileMatch": ["helm/.*values.*\\.ya?ml$", "clusters/.*\\.ya?ml$"]
  },
  
  "packageRules": [
    {
      "matchPackageNames": ["ingress-nginx"],
      "matchDatasources": ["helm"],
      "groupName": "ingress-nginx",
      "reviewers": ["platform-team"],
      "labels": ["security", "ingress", "helm"],
      
      "automerge": false,
      
      "prPriority": 10,
      
      "prTitle": "chore(ingress): update ingress-nginx {{newVersion}} (was {{currentVersion}})",
      
      "prBody": "## ingress-nginx Update\n\nThis PR updates ingress-nginx from `{{currentVersion}}` to `{{newVersion}}`.\n\n### Security Check\n- [ ] Check [ingress-nginx releases](https://github.com/kubernetes/ingress-nginx/releases) for CVEs fixed in this version\n- [ ] Verify no breaking changes in NGINX configuration\n- [ ] Run staging cluster upgrade before merging to production\n\n**Action required before merge:** Deploy to staging cluster and verify ingress functionality."
    },
    
    {
      "matchPackageNames": ["ingress-nginx"],
      "matchDatasources": ["helm"],
      "matchUpdateTypes": ["patch"],
      "automerge": false,
      "prPriority": 20,
      "labels": ["security", "patch-release"]
    }
  ],
  
  "vulnerabilityAlerts": {
    "enabled": true,
    "labels": ["security", "vulnerability"]
  }
}

Step 3 — Implement staged rollout with canary testing

# ingress-nginx-upgrade-pipeline.yaml
# ArgoCD ApplicationSet for staged ingress-nginx rollout

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ingress-nginx-staged
  namespace: argocd
spec:
  generators:
  # Ordered cluster list — staging first, then production clusters by risk tier
  - list:
      elements:
      - cluster: staging
        url: https://staging.k8s.example.com
        tier: "1"
        autoSync: "true"
      - cluster: production-us-west
        url: https://prod-usw.k8s.example.com
        tier: "2"
        autoSync: "false"
      - cluster: production-eu
        url: https://prod-eu.k8s.example.com
        tier: "3"
        autoSync: "false"
  
  template:
    metadata:
      name: "ingress-nginx-{{cluster}}"
      labels:
        tier: "{{tier}}"
    spec:
      project: platform
      source:
        repoURL: https://kubernetes.github.io/ingress-nginx
        chart: ingress-nginx
        targetRevision: "4.12.1"  # Pin to specific version; Renovate updates this
        helm:
          valuesObject:
            controller:
              replicaCount: 3
              minAvailable: 2
              image:
                digest: "sha256:abc123..."  # Pin image digest for supply chain integrity
      destination:
        server: "{{url}}"
        namespace: ingress-nginx
      syncPolicy:
        automated:
          prune: false
          selfHeal: "{{autoSync}}"

Step 4 — Add pre-upgrade validation gates

#!/bin/bash
# ingress-nginx-pre-upgrade-check.sh
# Run before upgrading ingress-nginx in any cluster

NAMESPACE=${1:-ingress-nginx}
NEW_VERSION=${2:?Usage: $0 <namespace> <new-version>}

echo "=== Pre-upgrade validation for ingress-nginx $NEW_VERSION ==="

# 1. Verify the new version patches known CVEs
echo "--- CVE status for version $NEW_VERSION ---"
CHANGELOG_URL="https://github.com/kubernetes/ingress-nginx/releases/tag/controller-v${NEW_VERSION}"
echo "Review changelog: $CHANGELOG_URL"

# 2. Check annotation validation is enabled (defence against annotation injection CVEs)
echo ""
echo "--- Annotation validation config ---"
kubectl get configmap -n "$NAMESPACE" ingress-nginx-controller \
  -o jsonpath='{.data}' 2>/dev/null | jq '
  {
    "allow-snippet-annotations": .["allow-snippet-annotations"],
    "annotation-value-word-blocklist": .["annotation-value-word-blocklist"]
  }'

# 3. Verify current ingress objects don't use dangerous annotations
echo ""
echo "--- Scanning for configuration-snippet annotations (all namespaces) ---"
kubectl get ingress -A \
  -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.metadata.annotations}{"\n"}{end}' | \
  grep "configuration-snippet\|server-snippet\|stream-snippet" | \
  grep -v "^$" && echo "WARNING: Found ingress objects with snippet annotations — review before upgrade" || \
  echo "OK: No snippet annotations found"

# 4. Check PodDisruptionBudget is configured
echo ""
echo "--- PodDisruptionBudget ---"
kubectl get pdb -n "$NAMESPACE" 2>/dev/null || \
  echo "WARNING: No PodDisruptionBudget found — upgrade may cause outage"

# 5. Snapshot current NGINX config for comparison post-upgrade
echo ""
echo "--- Snapshotting current NGINX config ---"
kubectl exec -n "$NAMESPACE" \
  "$(kubectl get pod -n "$NAMESPACE" -l "app.kubernetes.io/component=controller" \
     -o jsonpath='{.items[0].metadata.name}')" \
  -- nginx -T 2>/dev/null > "/tmp/nginx-config-pre-upgrade-$(date +%Y%m%d).txt"
echo "Config snapshot saved to /tmp/nginx-config-pre-upgrade-$(date +%Y%m%d).txt"

Step 5 — Disable dangerous annotation classes via controller config

# ingress-nginx-hardening-configmap.yaml
# Apply regardless of version — reduces annotation injection attack surface

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  # Disable configuration-snippet entirely (mitigates CVE-2023-5044 class)
  allow-snippet-annotations: "false"
  
  # Block dangerous words in annotation values
  annotation-value-word-blocklist: "load_module,lua_package,_by_lua,location,root,proxy_pass,serviceaccount"
  
  # Enable annotation validation
  enable-annotation-validation: "true"
  
  # Prevent reading from filesystem paths (limits post-exploit impact)
  server-tokens: "false"

# Apply and verify
kubectl apply -f ingress-nginx-hardening-configmap.yaml

# Confirm the controller picked up the config
kubectl exec -n ingress-nginx \
  "$(kubectl get pod -n ingress-nginx -l "app.kubernetes.io/component=controller" \
     -o jsonpath='{.items[0].metadata.name}')" \
  -- nginx -T 2>/dev/null | grep "allow-snippet\|annotation"

Step 6 — Alert on version drift and vulnerable versions

# Prometheus alert for ingress-nginx version drift
groups:
- name: ingress_nginx_security
  rules:
  - alert: IngressNginxVulnerableVersion
    expr: |
      kube_pod_container_info{
        namespace="ingress-nginx",
        container="controller",
        image=~".*ingress-nginx/controller:v1\.(8|9|10|11\.[0-4]).*"
      } == 1
    for: 1h
    labels:
      severity: critical
    annotations:
      summary: "Cluster running vulnerable ingress-nginx version"
      description: "Pod {{ $labels.pod }} is running a version with known CVEs. Upgrade to v1.11.5+ or v1.12.1+"

  - alert: IngressNginxVersionDrift
    expr: |
      count(
        count by (image) (
          kube_pod_container_info{namespace="ingress-nginx", container="controller"}
        )
      ) > 1
    for: 24h
    labels:
      severity: warning
    annotations:
      summary: "Multiple ingress-nginx versions running across pods"
      description: "Version drift detected — ensure upgrade completed cleanly"

Expected Behaviour

Trade-offs

Failure Modes

Related Articles

• Linux NGINX Worker Privilege Hardening — OS-level containment for NGINX CVEs on bare metal/VM deployments
• Kubernetes CVE Patch Automation — broader CVE patch automation for Kubernetes components beyond ingress-nginx
• Helm Chart Security Scanning — scanning Helm charts for vulnerabilities and misconfigurations before deployment
• NGINX Config Security CI Pipeline — static analysis of NGINX configuration in CI to catch misconfigs before deployment
• NGINX Fleet Patch Management — managing NGINX versions across the full fleet including bare metal and Kubernetes