AI-Powered Security Review of Open-Source Kubernetes Operators Before Deployment

The Problem

A Kubernetes operator is a privileged program running inside your cluster with a service account whose ClusterRole often reads every Secret in every namespace, writes to CRDs cluster-wide, and communicates with the API server on a privileged network path. Teams install operators from OperatorHub, Helm charts, and GitHub without reviewing the permissions the operator requests or the controller code that uses those permissions.

The trust model is implicit: if the project has GitHub stars, recent commits, and is used at major companies, the operator’s RBAC claims are assumed to be legitimate and minimal. In practice, many operators carry permissions that are broader than their current feature set requires — accumulated from past features, cargo-culted from templates, or added defensively by maintainers who prefer not to debug permission errors in production. A get/list/watch on secrets in * (all namespaces) in cert-manager, external-secrets, or the Prometheus operator is not inherently malicious, but it means a single vulnerability in the operator’s controller logic could expose every Secret in the cluster to an attacker.

The controller code itself — often 10,000 to 50,000 lines of Go — processes data that originates from Custom Resource spec fields, ConfigMaps, and upstream API responses. A reconcile loop that takes a user-supplied URL from a CRD field and issues an HTTP request to validate it without allowlisting is an SSRF vector. A controller that writes a received value into a shell command string is a command injection. These are not theoretical: real CVEs have been filed against production operators for exactly these patterns.

Until recently, auditing an operator before cluster installation required a dedicated security engineer, days of reading Go code, and expertise in Kubernetes RBAC semantics. LLM-based code analysis can now review a reconciler function in seconds and explicitly answer questions like “does user-controlled data from a CRD spec field reach an exec call?” or “does this ClusterRole grant the ability to escalate to cluster-admin?” The remaining human work is interpreting results and making risk decisions — not reading every line of controller code.

Threat Model

Over-broad RBAC exfiltration. An operator with get/list/watch on secrets across all namespaces has read access to database passwords, TLS private keys, and service account tokens throughout the cluster. A bug in the operator’s reconcile loop — a path traversal in a template, an improper error-handling branch that logs full secret content, an unintentional write to a status field — can result in that data leaving the cluster. This is compounded when the operator also has egress access: the combination of secrets read access and the ability to make outbound HTTP calls is sufficient for credential exfiltration via a single exploitable vulnerability.

Unsafe CRD input processing. Operators reconcile based on CRD spec fields that a namespace-level user can write. If the reconciler passes a field value to exec.Command, os.Stat, or template.Execute without sanitisation, a malicious CRD spec can achieve RCE in the operator pod. Since the operator pod runs with elevated cluster privileges, RCE in the operator typically means cluster compromise. Kubernetes admission controllers validate that CRD fields match their JSON schema, but schema validation does not prevent passing a valid string containing shell metacharacters.

Supply chain compromise via image update. An attacker who compromises the operator project’s container registry or CI/CD pipeline publishes a new image tag containing malicious code. Organisations that pin to a mutable tag (:latest, :v1.2, :stable) and use automated rollout mechanisms will deploy the malicious version without human review. The malicious version may phone home with cluster credentials, enumerate secrets, or wait for a trigger event.

Permission escalation on version upgrade. A minor version bump of an operator introduces a new ClusterRole rule. Automated Helm upgrades apply the new RBAC without alerting the team that the operator’s permissions have expanded. Over successive upgrades, an operator that started with minimal permissions accumulates patch on deployments, delete on namespaces, and impersonate on serviceaccounts — each added for a new feature, each individually justified, and collectively equivalent to cluster-admin.

Hardening Configuration

Step 1: Automated RBAC Permission Audit

Extract all RBAC definitions from an operator’s Helm chart or YAML manifests and flag high-risk permission patterns before the operator is installed.

#!/usr/bin/env python3
# rbac-audit.py — extract and analyse RBAC from operator manifests
# Usage: helm template my-operator ./chart | python3 rbac-audit.py

import sys
import yaml
import json
from typing import Generator

HIGH_RISK_RULES = [
    # (description, match_fn)
    ("wildcard verbs on any resource",
     lambda r: "*" in r.get("verbs", [])),
    ("secrets read across all namespaces",
     lambda r: "secrets" in r.get("resources", []) and
               any(v in r.get("verbs", []) for v in ["get", "list", "watch", "*"])),
    ("pods/exec access (enables RCE)",
     lambda r: "pods/exec" in r.get("resources", [])),
    ("impersonate verb (enables privilege escalation)",
     lambda r: "impersonate" in r.get("verbs", [])),
    ("wildcard resources",
     lambda r: "*" in r.get("resources", [])),
    ("cluster-scoped write access",
     lambda r: any(v in r.get("verbs", []) for v in ["create", "update", "patch", "delete", "*"])
               and "*" in r.get("apiGroups", [])),
]

def parse_manifests(stream) -> Generator[dict, None, None]:
    for doc in yaml.safe_load_all(stream):
        if doc:
            yield doc

def audit_rbac(manifests: list[dict]) -> list[dict]:
    findings = []

    for doc in manifests:
        kind = doc.get("kind", "")
        if kind not in ("ClusterRole", "Role"):
            continue

        name = doc.get("metadata", {}).get("name", "unknown")
        namespace = doc.get("metadata", {}).get("namespace", "cluster-scoped")
        is_cluster_role = (kind == "ClusterRole")

        for rule in doc.get("rules", []):
            for desc, match_fn in HIGH_RISK_RULES:
                try:
                    if match_fn(rule):
                        findings.append({
                            "severity": "HIGH" if is_cluster_role else "MEDIUM",
                            "role": name,
                            "scope": namespace,
                            "issue": desc,
                            "rule": rule,
                        })
                except Exception:
                    pass

    return findings

def check_clusterrolebindings(manifests: list[dict], findings: list[dict]) -> list[dict]:
    """Flag ClusterRoleBindings that bind high-risk roles to operator service accounts."""
    role_names = {f["role"] for f in findings if f["severity"] == "HIGH"}

    for doc in manifests:
        if doc.get("kind") != "ClusterRoleBinding":
            continue
        ref = doc.get("roleRef", {})
        if ref.get("name") in role_names:
            subjects = doc.get("subjects", [])
            sa_subjects = [s for s in subjects if s.get("kind") == "ServiceAccount"]
            if sa_subjects:
                findings.append({
                    "severity": "HIGH",
                    "role": ref["name"],
                    "scope": "cluster",
                    "issue": f"High-risk ClusterRole bound to operator ServiceAccount(s): "
                             f"{[s['name'] for s in sa_subjects]}",
                    "rule": doc,
                })
    return findings

if __name__ == "__main__":
    raw = sys.stdin.read()
    manifests = list(parse_manifests(raw))
    findings = audit_rbac(manifests)
    findings = check_clusterrolebindings(manifests, findings)

    print(json.dumps({"findings": findings, "total": len(findings)}, indent=2))

    if any(f["severity"] == "HIGH" for f in findings):
        sys.exit(1)

Integrate into a Makefile target for operator evaluation:

# Makefile — operator security gate
OPERATOR_CHART ?= ./charts/my-operator
OPERATOR_VERSION ?= 1.0.0

.PHONY: rbac-audit
rbac-audit:
	helm template audit $(OPERATOR_CHART) --version $(OPERATOR_VERSION) \
	  | python3 scripts/rbac-audit.py 2>&1 | tee reports/rbac-$(OPERATOR_VERSION).json

Step 2: LLM-Powered Go Reconciler Code Review

The most effective use of LLM analysis is directed review of the reconciler’s Reconcile() function, with an explicit security-focused prompt. The following script fetches the operator source and runs targeted analysis.

#!/usr/bin/env python3
# operator-code-review.py — LLM security review of operator reconciler
import anthropic
import subprocess
import sys
import json

def fetch_reconciler_source(repo_url: str, version: str, file_path: str) -> str:
    """Fetch reconciler source from GitHub at a tagged version."""
    raw_url = (
        f"https://raw.githubusercontent.com/"
        f"{repo_url.removeprefix('https://github.com/')}/refs/tags/{version}/{file_path}"
    )
    result = subprocess.run(
        ["curl", "-sf", raw_url],
        capture_output=True, text=True, timeout=30
    )
    if result.returncode != 0:
        raise RuntimeError(f"Failed to fetch {raw_url}: {result.stderr}")
    return result.stdout

def review_with_llm(operator_name: str, source_code: str) -> dict:
    client = anthropic.Anthropic()

    # Limit source to 8000 tokens — focus on the Reconcile function
    code_excerpt = source_code[:20000]

    prompt = f"""You are a senior Kubernetes security engineer conducting a pre-deployment security review of an open-source operator named '{operator_name}'.

Analyse the following Go controller code, focusing specifically on the Reconcile() function and any functions it calls:

```go
{code_excerpt}

Answer each question with specific line references where possible:

1. External data sinks: What external data (from CRD spec fields, ConfigMaps, Annotations, or API responses) reaches exec.Command, http.Get, os.WriteFile, template.Execute, or os.Stat without sanitisation?

2. Credential handling: Are secrets or tokens logged, written to status fields, or passed to external services in ways that could expose them?

3. SSRF risk: Does the reconciler make HTTP/HTTPS requests to URLs derived from CRD spec fields? Is there an allowlist?

4. Race conditions: Does the reconciler modify shared state without locks or use client.Get followed by client.Update without conflict retry logic?

5. Privilege use justification: Which API server calls in the reconciler actually require the permissions in the ClusterRole? Are there permissions that appear to be unused?

Respond as JSON with keys: {{ “external_data_sinks”: [{{“location”: “file:line”, “data_source”: “…”, “sink”: “…”, “sanitised”: true/false}}], “credential_risks”: [{{“description”: “…”, “severity”: “LOW/MEDIUM/HIGH/CRITICAL”}}], “ssrf_risks”: [{{“description”: “…”, “severity”: “…”}}], “race_conditions”: [{{“description”: “…”, “severity”: “…”}}], “unused_permissions”: [“list of permissions that appear unused in the code”], “overall_risk”: “LOW/MEDIUM/HIGH/CRITICAL”, “summary”: “…”, “blocking_issues”: [“issues that should block deployment”] }}“”"

response = client.messages.create( model=“claude-sonnet-4-6”, max_tokens=4096, messages=[{“role”: “user”, “content”: prompt}] )

return json.loads(response.content[0].text)

if name == “main”: operator_name = sys.argv[1] # e.g. “cert-manager” repo_url = sys.argv[2] # e.g. “https://github.com/cert-manager/cert-manager” version = sys.argv[3] # e.g. “v1.15.0” reconciler_path = sys.argv[4] # e.g. “internal/controller/certificaterequest_controller.go”

print(f"[*] Fetching {reconciler_path} at {version}…") source = fetch_reconciler_source(repo_url, version, reconciler_path)

print(“[*] Running LLM security review…”) result = review_with_llm(operator_name, source) print(json.dumps(result, indent=2))

if result[“overall_risk”] in (“HIGH”, “CRITICAL”) or result[“blocking_issues”]: sys.exit(1)

### Step 3: Semgrep Rules for Common Operator Vulnerabilities

```yaml
# operator-security.yml — semgrep rules for Kubernetes operator code
rules:
  - id: operator-exec-command-unsanitised-input
    languages: [go]
    message: |
      exec.Command or exec.CommandContext called with a value that may derive
      from a CRD spec field. Trace data flow from Reconcile() args to this call.
    severity: ERROR
    pattern-either:
      - pattern: exec.Command($CMD, ...)
      - pattern: exec.CommandContext($CTX, $CMD, ...)
    pattern-not:
      - pattern: exec.Command("kubectl", ...)
      - pattern: exec.Command("/usr/bin/...", ...)

  - id: operator-http-ssrf-crd-url
    languages: [go]
    message: |
      HTTP client making request to a URL that may originate from a CRD spec field.
      Verify URL is validated against an allowlist before this call.
    severity: WARNING
    patterns:
      - pattern: |
          $CLIENT.Get($CTX, $URL, ...)
      - pattern-not: |
          $URL = $ALLOWLIST_FUNC(...)

  - id: operator-secret-logged
    languages: [go]
    message: |
      A variable named 'secret', 'password', 'token', or 'credential' may be
      passed to a logging call. Ensure sensitive values are not emitted to logs.
    severity: ERROR
    pattern-either:
      - pattern: log.$FUNC(..., $SECRET, ...) where: {metavariable-regex: {metavariable: $SECRET, regex: '(?i)(secret|password|token|cred)'}}
      - pattern: logger.$FUNC(..., $SECRET, ...) where: {metavariable-regex: {metavariable: $SECRET, regex: '(?i)(secret|password|token|cred)'}}

  - id: operator-status-update-no-retry
    languages: [go]
    message: |
      client.Status().Update() called without retry logic. In high-concurrency
      reconcilers this causes lost updates. Use retry.RetryOnConflict().
    severity: WARNING
    pattern: |
      $R.Client.Status().Update($CTX, ...)
    pattern-not: |
      retry.RetryOnConflict(...)

Run Semgrep in the evaluation pipeline:

#!/usr/bin/env bash
# semgrep-operator-check.sh
set -euo pipefail
OPERATOR_SRC_DIR="$1"
RULES_DIR="$(dirname "$0")/semgrep-rules"

semgrep \
  --config "$RULES_DIR/operator-security.yml" \
  --json \
  --output /tmp/semgrep-results.json \
  "$OPERATOR_SRC_DIR"

# Fail on any ERROR severity findings
ERROR_COUNT=$(jq '[.results[] | select(.extra.severity == "ERROR")] | length' /tmp/semgrep-results.json)
echo "[*] Semgrep found $ERROR_COUNT ERROR-level findings"
[[ "$ERROR_COUNT" -eq 0 ]] || exit 1

Step 4: Image Digest Verification Against SLSA Provenance

#!/usr/bin/env bash
# verify-operator-image.sh — verify operator image against SLSA provenance
set -euo pipefail

IMAGE_REF="$1"  # e.g. quay.io/jetstack/cert-manager-controller:v1.15.0

# Install cosign if not present
command -v cosign &>/dev/null || { echo "[!] cosign not installed"; exit 1; }

echo "[*] Verifying SLSA provenance for $IMAGE_REF"

# Verify signature and provenance with cosign
cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity-regexp "https://github.com/cert-manager/.*" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  "$IMAGE_REF" 2>&1 | tee /tmp/provenance-verification.log

# Extract image digest for pinning
DIGEST=$(crane digest "$IMAGE_REF")
echo "[*] Verified image digest: $DIGEST"

# Pin the digest in kustomization or Helm values
echo "operator_image_digest: $DIGEST" > /tmp/pinned-digest.yaml
echo "[PASS] Image $IMAGE_REF verified and pinned at $DIGEST"

Step 5: Graduated Deployment in Quarantine Namespace

# quarantine-deployment.yaml — strip RBAC, observe API calls before production install
apiVersion: v1
kind: Namespace
metadata:
  name: operator-quarantine
  labels:
    pod-security.kubernetes.io/enforce: restricted
---
# Minimal RBAC: only permissions needed to watch its own CRDs
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: operator-quarantine-minimal
rules:
  - apiGroups: ["example.io"]
    resources: ["mycrdkind"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]
---
# Deploy operator into quarantine namespace with audit logging enabled
# Watch for API calls the operator makes beyond its stripped permissions
# (these will appear as RBAC denial events in the audit log)

Monitor API calls during quarantine:

#!/usr/bin/env bash
# monitor-operator-api-calls.sh — parse audit log for operator SA calls
OPERATOR_SA="system:serviceaccount:operator-quarantine:my-operator"
AUDIT_LOG="/var/log/kubernetes/audit.log"

echo "[*] API calls made by operator service account (last 1 hour):"
jq --arg sa "$OPERATOR_SA" \
  'select(.user.username == $sa) |
   {verb: .verb, resource: .objectRef.resource, namespace: .objectRef.namespace,
    responseCode: .responseStatus.code}' \
  "$AUDIT_LOG" 2>/dev/null | sort | uniq -c | sort -rn

echo ""
echo "[*] RBAC denials (permissions the operator tried to use but doesn't have):"
jq --arg sa "$OPERATOR_SA" \
  'select(.user.username == $sa and .responseStatus.code == 403) |
   {verb: .verb, resource: .objectRef.resource, namespace: .objectRef.namespace}' \
  "$AUDIT_LOG" 2>/dev/null | sort | uniq -c | sort -rn

Step 6: GitHub Actions Workflow for Operator Version Bump PRs

# .github/workflows/operator-security-review.yml
name: Operator Security Review

on:
  pull_request:
    paths:
      - 'operators/**'
      - 'helmfiles/**'

jobs:
  operator-review:
    runs-on: ubuntu-24.04

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Detect changed operators
        id: changes
        run: |
          CHANGED=$(git diff --name-only origin/main...HEAD | grep 'operators/' | head -5)
          echo "changed=$CHANGED" >> "$GITHUB_OUTPUT"

      - name: Extract operator name and version
        id: meta
        run: |
          # Parse operator name and new version from changed Helm values files
          OPERATOR=$(echo "${{ steps.changes.outputs.changed }}" | head -1 | cut -d/ -f2)
          VERSION=$(yq '.image.tag' "operators/$OPERATOR/values.yaml")
          echo "operator=$OPERATOR" >> "$GITHUB_OUTPUT"
          echo "version=$VERSION" >> "$GITHUB_OUTPUT"

      - name: RBAC audit
        run: |
          helm template "${{ steps.meta.outputs.operator }}" \
            "operators/${{ steps.meta.outputs.operator }}" \
            --version "${{ steps.meta.outputs.version }}" \
            | python3 scripts/rbac-audit.py | tee "$GITHUB_STEP_SUMMARY"

      - name: LLM reconciler review
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          python3 scripts/operator-code-review.py \
            "${{ steps.meta.outputs.operator }}" \
            "$(yq '.source.repo' operators/${{ steps.meta.outputs.operator }}/metadata.yaml)" \
            "${{ steps.meta.outputs.version }}" \
            "$(yq '.source.reconcilerPath' operators/${{ steps.meta.outputs.operator }}/metadata.yaml)" \
            | tee /tmp/llm-review.json

      - name: Image digest verification
        run: |
          IMAGE=$(yq '.image.repository' "operators/${{ steps.meta.outputs.operator }}/values.yaml")
          bash scripts/verify-operator-image.sh "$IMAGE:${{ steps.meta.outputs.version }}"

      - name: Post review summary to PR
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const review = JSON.parse(fs.readFileSync('/tmp/llm-review.json', 'utf8'));
            const body = `## Operator Security Review: ${{ steps.meta.outputs.operator }} ${{ steps.meta.outputs.version }}
            **Overall Risk:** ${review.overall_risk}
            **Summary:** ${review.summary}
            ${review.blocking_issues.length ? '### Blocking Issues\n' + review.blocking_issues.map(i => `- ${i}`).join('\n') : '### No blocking issues found'}`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body
            });

Expected Behaviour After Hardening

Every new operator installation or version bump goes through the automated review pipeline before cluster deployment. When a team opens a PR to bump cert-manager from v1.14 to v1.15, the workflow runs within minutes: RBAC audit confirms no new permission additions (or flags them with a diff against the previous version’s ClusterRole), the LLM reconciler review analyses the changed Go files in the new release, and the image digest is verified against GitHub Actions SLSA provenance.

Operators with "overall_risk": "LOW" and no RBAC expansions can be auto-merged if CI passes. Operators flagged as "MEDIUM" require a security engineer to review the LLM output and approve. "HIGH" or "CRITICAL" findings block the PR with a required status check, routing to the security team’s queue.

The quarantine namespace deployment runs for 72 hours for any new operator, generating an API call manifest that is compared against the operator’s declared ClusterRole. Discrepancies — whether the operator uses fewer permissions than declared (candidates for reduction) or attempts capabilities beyond what it was granted (immediate escalation alert) — feed back into the operator’s approval record.

Trade-offs and Operational Considerations

Failure Modes

Related Articles

• Kubernetes Operator Security
• CodeQL Kubernetes Security Queries
• Kubernetes Supply Chain Incident Response
• AI SAST in CI/CD Vulnerability Discovery
• OpenSSF Scorecard Risk Scoring