Hardening the Kubernetes Secrets Store CSI Driver

Hardening the Kubernetes Secrets Store CSI Driver

Problem

The Secrets Store CSI Driver (secrets-store.csi.k8s.io) solves a real problem: it lets pods consume secrets from external stores — AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, HashiCorp Vault — as mounted volume files without storing them as Kubernetes Secrets. This is an improvement over the base Secrets model because the secret exists only in memory (tmpfs mount), rotates automatically, and never touches etcd.

The problem is that the CSI driver introduces its own attack surface, and its default configuration makes several security choices that undermine the isolation it is meant to provide.

The sync-to-Kubernetes-Secret feature. The driver supports an optional syncSecret mode that mirrors the external secret into a Kubernetes Secret object. This is used by teams who want environment variable injection instead of file mounts. When enabled, the secret is now in etcd, visible to any workload with get secrets permissions, and subject to all the same risks as native Kubernetes Secrets. Worse, the Kubernetes Secret created by the sync mechanism is not clearly marked as externally managed, leading teams to treat it as a permanent object even when the sync is not needed.

Provider pod permissions. Each provider (AWS, Azure, GCP, Vault) runs as a DaemonSet. The AWS provider requires an IAM role with secretsmanager:GetSecretValue and ssm:GetParameter on * (in default configurations). The Vault provider requires a Vault token or service account with read access to a broad policy. These DaemonSet pods run on every node; compromise of any one pod gives the attacker access to every secret that any pod on that node might request.

SecretProviderClass RBAC. SecretProviderClass objects define which external secrets to fetch and how to mount them. Any namespace-scoped service account with create secretproviderclasses can configure a mount that fetches any secret in the external store that the provider DaemonSet has access to. This is a lateral movement vector: a compromised application pod can create a new SecretProviderClass and mount secrets belonging to other workloads.

Rotation polling creates a denial of service surface. The driver polls external stores for secret rotation at a configurable interval. Under high pod density, this generates a large number of external API calls. If the external store rate-limits these calls, rotation stops working — silently. Pods continue with stale secrets, and the team may not notice for hours.

Provider DaemonSets on control-plane nodes. Default Helm chart configurations tolerate and schedule provider pods on control-plane nodes. A vulnerability in the provider pod on a control-plane node has a much larger blast radius than on a worker node.

Target systems: Kubernetes 1.24+ with secrets-store-csi-driver ≥1.3; clusters using AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, or HashiCorp Vault providers; any cluster where the sync-to-Secret feature is enabled.


Threat Model

Adversary 1 — Namespace-scoped attacker creating SecretProviderClass. Access level: service account with create secretproviderclasses in a namespace. Objective: create a SecretProviderClass that references secrets belonging to a higher-privilege namespace’s workload; mount the class into a pod; read the secret at the mount path.

Adversary 2 — Compromised provider DaemonSet pod. Access level: code execution inside the AWS/Azure/GCP provider pod. Objective: use the provider’s IAM role (which has broad secretsmanager:GetSecretValue access) to exfiltrate every secret in the account — not just those being used by pods on that node.

Adversary 3 — Sync-to-Secret cross-namespace read. Access level: service account with get secrets in a namespace where a synced Secret exists. Objective: read a synced Kubernetes Secret that mirrors an external secret; the team did not intend for the secret to be readable via standard Kubernetes APIs.

Adversary 4 — SecretProviderClass in shared namespace. Access level: engineer with kubectl apply access to a shared namespace. Objective: create a SecretProviderClass referencing production database credentials, mount it into a test pod, and read the credentials.

Without hardening: the provider’s broad IAM permissions and the sync-to-Secret feature expand the blast radius significantly beyond the intended secret access. With hardening: provider IAM is scoped per-secret; sync-to-Secret is disabled except where genuinely needed; SecretProviderClass creation is restricted to platform team.


Configuration / Implementation

Step 1 — Audit current SecretProviderClass objects and sync status

# List all SecretProviderClasses and check for syncSecret
kubectl get secretproviderclass --all-namespaces -o json | jq -r '
  .items[] |
  {
    namespace: .metadata.namespace,
    name: .metadata.name,
    provider: .spec.provider,
    sync_enabled: ((.spec.secretObjects // []) | length > 0),
    secrets_count: ((.spec.secretObjects // []) | length)
  }
'

# Find all Kubernetes Secrets created by the CSI driver sync
kubectl get secrets --all-namespaces \
  -l "secrets-store.csi.k8s.io/managed=true" \
  -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,AGE:.metadata.creationTimestamp"

# Check provider DaemonSet service accounts and their cloud permissions
kubectl get serviceaccounts --all-namespaces | grep -E "csi|secrets-store|provider"

Step 2 — Disable sync-to-Kubernetes-Secret for workloads using file mounts

For workloads that only need the secret as a file (the preferred pattern), ensure secretObjects is absent from the SecretProviderClass:

# Correct: file-only mount, no sync to Kubernetes Secret
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-credentials
  namespace: production
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/app/db-password"
        objectType: "secretsmanager"
        objectAlias: "db-password"
  # No secretObjects block = no Kubernetes Secret created
---
# Pod spec consuming the file mount
spec:
  volumes:
  - name: secrets-store
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: app-credentials
  containers:
  - name: app
    volumeMounts:
    - name: secrets-store
      mountPath: /mnt/secrets
      readOnly: true
    # Read secret from /mnt/secrets/db-password — never in env or etcd

If sync is required for a legacy workload that only accepts env vars, restrict who can see the synced Secret:

# Limit synced Secret visibility — create in a dedicated namespace with restricted RBAC
apiVersion: v1
kind: Namespace
metadata:
  name: secrets-synced
  labels:
    purpose: synced-secrets-only
---
# RBAC: only specific service accounts can read synced secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: synced-secret-reader
  namespace: secrets-synced
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: secret-reader-only
subjects:
- kind: ServiceAccount
  name: legacy-app-sa
  namespace: production

Step 3 — Scope provider IAM permissions per workload (AWS)

Instead of giving the provider DaemonSet broad secretsmanager:GetSecretValue on *, use IRSA to give each workload’s service account access only to its own secrets:

// IAM policy for a specific application's service account (not the provider DaemonSet)
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:prod/app-name/*"
      ]
    }
  ]
}

Configure the pod’s service account to use IRSA (pod-level IAM) rather than the node’s instance profile:

# Pod service account with scoped IAM role
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-name-sa
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/app-name-secret-reader
---
# SecretProviderClass referencing the pod's own service account credentials
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-name-credentials
  namespace: production
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/app-name/db-password"
        objectType: "secretsmanager"
    # Use the pod's IRSA identity, not the node's instance profile
    region: us-east-1

This way, even if the provider DaemonSet is compromised, it cannot fetch secrets for workloads it is not currently serving.

Step 4 — Restrict SecretProviderClass creation with Kyverno

Prevent arbitrary service accounts from creating SecretProviderClass objects that reference sensitive secrets:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-secretproviderclass-creation
spec:
  validationFailureAction: Enforce
  rules:
  - name: only-platform-team-can-create-spc
    match:
      any:
      - resources:
          kinds: [SecretProviderClass]
          operations: [CREATE, UPDATE]
    validate:
      message: "SecretProviderClass objects must be created by the platform team service account"
      deny:
        conditions:
          all:
          - key: "{{ request.userInfo.username }}"
            operator: NotIn
            value:
            - "system:serviceaccount:platform:platform-operator"
            - "system:serviceaccount:argocd:argocd-application-controller"
            - "system:serviceaccount:flux-system:kustomize-controller"
          - key: "{{ request.userInfo.groups[] | contains(@, 'platform-team') }}"
            operator: NotEquals
            value: true

  # Additionally: block sync-to-Secret unless in approved namespaces
  - name: restrict-sync-to-secret
    match:
      any:
      - resources:
          kinds: [SecretProviderClass]
    validate:
      message: "secretObjects (sync-to-Kubernetes-Secret) only permitted in approved namespaces"
      deny:
        conditions:
          all:
          - key: "{{ request.object.spec.secretObjects | length(@) }}"
            operator: GreaterThan
            value: 0
          - key: "{{ request.object.metadata.namespace }}"
            operator: NotIn
            value: ["legacy-apps", "secrets-synced"]

Step 5 — Restrict provider DaemonSet to worker nodes only

Prevent provider pods from scheduling on control-plane nodes:

# Helm values for secrets-store-csi-driver
# values-hardened.yaml

provider:
  aws:
    nodeSelector:
      node-role.kubernetes.io/worker: "true"
    tolerations: []  # Remove any control-plane tolerations
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]

driver:
  nodeSelector:
    node-role.kubernetes.io/worker: "true"
  tolerations: []

# Disable sync-to-Secret feature driver-wide if not needed
syncSecret:
  enabled: false  # Disable the sync controller entirely

Apply:

helm upgrade secrets-store-csi-driver secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system \
  --values values-hardened.yaml

Step 6 — Monitor for SecretProviderClass anomalies

# Falco rule — alert on SecretProviderClass creation outside platform tooling
- rule: Unexpected SecretProviderClass Created
  desc: SecretProviderClass created by a non-platform service account
  condition: >
    k8s.verb in (create, update) and
    k8s.target.resource = secretproviderclasses and
    not k8s.user.name in (
      system:serviceaccount:platform:platform-operator,
      system:serviceaccount:argocd:argocd-application-controller
    )
  output: >
    SecretProviderClass created by unexpected principal
    (user=%k8s.user.name namespace=%k8s.target.namespace name=%k8s.target.name)
  priority: WARNING

Expected Behaviour

Signal Before hardening After hardening
kubectl get secrets -l secrets-store.csi.k8s.io/managed=true Returns synced secrets in many namespaces Empty (sync disabled) or restricted to approved namespace
Developer creates SecretProviderClass with secretObjects Succeeds Blocked by Kyverno — not in approved namespace
Provider DaemonSet pod on control-plane node Possible Blocked by nodeSelector
Provider IAM role has secretsmanager:* on * Common default Scoped per-workload via IRSA
Falco alert on unexpected SecretProviderClass No alert WARNING fires immediately

Verification:

# Confirm sync-to-Secret is disabled
kubectl get SecretProviderClass --all-namespaces -o json | \
  jq '[.items[] | select(.spec.secretObjects and (.spec.secretObjects | length > 0))] | length'
# Expected: 0 (unless in approved namespace)

# Confirm no provider pods on control-plane
kubectl get pods -n kube-system -l app=csi-secrets-store-provider-aws \
  -o jsonpath='{.items[*].spec.nodeName}' | tr ' ' '\n' | while read node; do
  role=$(kubectl get node "$node" -o jsonpath='{.metadata.labels.node-role\.kubernetes\.io/control-plane}')
  [[ -n "$role" ]] && echo "WARNING: Provider pod on control-plane node: $node"
done

Trade-offs

Aspect Benefit Cost Mitigation
Disable sync-to-Secret globally Eliminates etcd exposure for externally-managed secrets Legacy apps that only accept env vars break Migrate to envFrom with secretKeyRef pointing to a file-mounted secret; or use the External Secrets Operator as a migration path
Per-workload IRSA scoping Blast radius of provider compromise limited to one workload’s secrets More IAM roles to manage Automate role creation via Terraform module in service onboarding
Kyverno restrict SPC creation Prevents lateral movement via rogue SecretProviderClass Slows self-service secret configuration Build a GitOps template for SPC creation; platform team reviews and merges
Provider DaemonSet off control-plane Limits control-plane compromise blast radius None — control-plane nodes have no legitimate need for this DaemonSet Apply universally

Failure Modes

Failure Symptom Detection Recovery
Pod fails to mount secret after IRSA scope change Pod stays in ContainerCreating; events show CSI mount failure kubectl describe pod shows “failed to mount secret”; provider logs show IAM denied Verify IRSA policy includes the specific secret ARN; check the pod’s service account has the IRSA annotation
Kyverno blocks GitOps controller from creating SPC ArgoCD/Flux cannot sync SecretProviderClass manifests; app stays out of sync ArgoCD shows SyncFailed; Kyverno audit log shows denial for argocd SA Add ArgoCD/Flux service account to the Kyverno exception list
Sync disabled but app expects env vars from synced Secret App container fails with missing env var Container logs show KeyError / missing env; pod restarts Either re-enable sync for this specific namespace (with restricted RBAC) or migrate the app to file-based secret consumption
Provider rotation polling rate-limited Pods continue with stale secrets; rotation silently stopped Check provider logs for rate limit errors; CSI driver metrics show stale age Reduce polling frequency; implement exponential backoff in provider config; alert on rotation age exceeding TTL