Hardening the Kubernetes Secrets Store CSI Driver
Problem
The Secrets Store CSI Driver (secrets-store.csi.k8s.io) solves a real problem: it lets pods consume secrets from external stores — AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, HashiCorp Vault — as mounted volume files without storing them as Kubernetes Secrets. This is an improvement over the base Secrets model because the secret exists only in memory (tmpfs mount), rotates automatically, and never touches etcd.
The problem is that the CSI driver introduces its own attack surface, and its default configuration makes several security choices that undermine the isolation it is meant to provide.
The sync-to-Kubernetes-Secret feature. The driver supports an optional syncSecret mode that mirrors the external secret into a Kubernetes Secret object. This is used by teams who want environment variable injection instead of file mounts. When enabled, the secret is now in etcd, visible to any workload with get secrets permissions, and subject to all the same risks as native Kubernetes Secrets. Worse, the Kubernetes Secret created by the sync mechanism is not clearly marked as externally managed, leading teams to treat it as a permanent object even when the sync is not needed.
Provider pod permissions. Each provider (AWS, Azure, GCP, Vault) runs as a DaemonSet. The AWS provider requires an IAM role with secretsmanager:GetSecretValue and ssm:GetParameter on * (in default configurations). The Vault provider requires a Vault token or service account with read access to a broad policy. These DaemonSet pods run on every node; compromise of any one pod gives the attacker access to every secret that any pod on that node might request.
SecretProviderClass RBAC. SecretProviderClass objects define which external secrets to fetch and how to mount them. Any namespace-scoped service account with create secretproviderclasses can configure a mount that fetches any secret in the external store that the provider DaemonSet has access to. This is a lateral movement vector: a compromised application pod can create a new SecretProviderClass and mount secrets belonging to other workloads.
Rotation polling creates a denial of service surface. The driver polls external stores for secret rotation at a configurable interval. Under high pod density, this generates a large number of external API calls. If the external store rate-limits these calls, rotation stops working — silently. Pods continue with stale secrets, and the team may not notice for hours.
Provider DaemonSets on control-plane nodes. Default Helm chart configurations tolerate and schedule provider pods on control-plane nodes. A vulnerability in the provider pod on a control-plane node has a much larger blast radius than on a worker node.
Target systems: Kubernetes 1.24+ with secrets-store-csi-driver ≥1.3; clusters using AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, or HashiCorp Vault providers; any cluster where the sync-to-Secret feature is enabled.
Threat Model
Adversary 1 — Namespace-scoped attacker creating SecretProviderClass. Access level: service account with create secretproviderclasses in a namespace. Objective: create a SecretProviderClass that references secrets belonging to a higher-privilege namespace’s workload; mount the class into a pod; read the secret at the mount path.
Adversary 2 — Compromised provider DaemonSet pod. Access level: code execution inside the AWS/Azure/GCP provider pod. Objective: use the provider’s IAM role (which has broad secretsmanager:GetSecretValue access) to exfiltrate every secret in the account — not just those being used by pods on that node.
Adversary 3 — Sync-to-Secret cross-namespace read. Access level: service account with get secrets in a namespace where a synced Secret exists. Objective: read a synced Kubernetes Secret that mirrors an external secret; the team did not intend for the secret to be readable via standard Kubernetes APIs.
Adversary 4 — SecretProviderClass in shared namespace. Access level: engineer with kubectl apply access to a shared namespace. Objective: create a SecretProviderClass referencing production database credentials, mount it into a test pod, and read the credentials.
Without hardening: the provider’s broad IAM permissions and the sync-to-Secret feature expand the blast radius significantly beyond the intended secret access. With hardening: provider IAM is scoped per-secret; sync-to-Secret is disabled except where genuinely needed; SecretProviderClass creation is restricted to platform team.
Configuration / Implementation
Step 1 — Audit current SecretProviderClass objects and sync status
# List all SecretProviderClasses and check for syncSecret
kubectl get secretproviderclass --all-namespaces -o json | jq -r '
.items[] |
{
namespace: .metadata.namespace,
name: .metadata.name,
provider: .spec.provider,
sync_enabled: ((.spec.secretObjects // []) | length > 0),
secrets_count: ((.spec.secretObjects // []) | length)
}
'
# Find all Kubernetes Secrets created by the CSI driver sync
kubectl get secrets --all-namespaces \
-l "secrets-store.csi.k8s.io/managed=true" \
-o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,AGE:.metadata.creationTimestamp"
# Check provider DaemonSet service accounts and their cloud permissions
kubectl get serviceaccounts --all-namespaces | grep -E "csi|secrets-store|provider"
Step 2 — Disable sync-to-Kubernetes-Secret for workloads using file mounts
For workloads that only need the secret as a file (the preferred pattern), ensure secretObjects is absent from the SecretProviderClass:
# Correct: file-only mount, no sync to Kubernetes Secret
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: app-credentials
namespace: production
spec:
provider: aws
parameters:
objects: |
- objectName: "prod/app/db-password"
objectType: "secretsmanager"
objectAlias: "db-password"
# No secretObjects block = no Kubernetes Secret created
---
# Pod spec consuming the file mount
spec:
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: app-credentials
containers:
- name: app
volumeMounts:
- name: secrets-store
mountPath: /mnt/secrets
readOnly: true
# Read secret from /mnt/secrets/db-password — never in env or etcd
If sync is required for a legacy workload that only accepts env vars, restrict who can see the synced Secret:
# Limit synced Secret visibility — create in a dedicated namespace with restricted RBAC
apiVersion: v1
kind: Namespace
metadata:
name: secrets-synced
labels:
purpose: synced-secrets-only
---
# RBAC: only specific service accounts can read synced secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: synced-secret-reader
namespace: secrets-synced
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: secret-reader-only
subjects:
- kind: ServiceAccount
name: legacy-app-sa
namespace: production
Step 3 — Scope provider IAM permissions per workload (AWS)
Instead of giving the provider DaemonSet broad secretsmanager:GetSecretValue on *, use IRSA to give each workload’s service account access only to its own secrets:
// IAM policy for a specific application's service account (not the provider DaemonSet)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret"
],
"Resource": [
"arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:prod/app-name/*"
]
}
]
}
Configure the pod’s service account to use IRSA (pod-level IAM) rather than the node’s instance profile:
# Pod service account with scoped IAM role
apiVersion: v1
kind: ServiceAccount
metadata:
name: app-name-sa
namespace: production
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/app-name-secret-reader
---
# SecretProviderClass referencing the pod's own service account credentials
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: app-name-credentials
namespace: production
spec:
provider: aws
parameters:
objects: |
- objectName: "prod/app-name/db-password"
objectType: "secretsmanager"
# Use the pod's IRSA identity, not the node's instance profile
region: us-east-1
This way, even if the provider DaemonSet is compromised, it cannot fetch secrets for workloads it is not currently serving.
Step 4 — Restrict SecretProviderClass creation with Kyverno
Prevent arbitrary service accounts from creating SecretProviderClass objects that reference sensitive secrets:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-secretproviderclass-creation
spec:
validationFailureAction: Enforce
rules:
- name: only-platform-team-can-create-spc
match:
any:
- resources:
kinds: [SecretProviderClass]
operations: [CREATE, UPDATE]
validate:
message: "SecretProviderClass objects must be created by the platform team service account"
deny:
conditions:
all:
- key: "{{ request.userInfo.username }}"
operator: NotIn
value:
- "system:serviceaccount:platform:platform-operator"
- "system:serviceaccount:argocd:argocd-application-controller"
- "system:serviceaccount:flux-system:kustomize-controller"
- key: "{{ request.userInfo.groups[] | contains(@, 'platform-team') }}"
operator: NotEquals
value: true
# Additionally: block sync-to-Secret unless in approved namespaces
- name: restrict-sync-to-secret
match:
any:
- resources:
kinds: [SecretProviderClass]
validate:
message: "secretObjects (sync-to-Kubernetes-Secret) only permitted in approved namespaces"
deny:
conditions:
all:
- key: "{{ request.object.spec.secretObjects | length(@) }}"
operator: GreaterThan
value: 0
- key: "{{ request.object.metadata.namespace }}"
operator: NotIn
value: ["legacy-apps", "secrets-synced"]
Step 5 — Restrict provider DaemonSet to worker nodes only
Prevent provider pods from scheduling on control-plane nodes:
# Helm values for secrets-store-csi-driver
# values-hardened.yaml
provider:
aws:
nodeSelector:
node-role.kubernetes.io/worker: "true"
tolerations: [] # Remove any control-plane tolerations
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
driver:
nodeSelector:
node-role.kubernetes.io/worker: "true"
tolerations: []
# Disable sync-to-Secret feature driver-wide if not needed
syncSecret:
enabled: false # Disable the sync controller entirely
Apply:
helm upgrade secrets-store-csi-driver secrets-store-csi-driver/secrets-store-csi-driver \
--namespace kube-system \
--values values-hardened.yaml
Step 6 — Monitor for SecretProviderClass anomalies
# Falco rule — alert on SecretProviderClass creation outside platform tooling
- rule: Unexpected SecretProviderClass Created
desc: SecretProviderClass created by a non-platform service account
condition: >
k8s.verb in (create, update) and
k8s.target.resource = secretproviderclasses and
not k8s.user.name in (
system:serviceaccount:platform:platform-operator,
system:serviceaccount:argocd:argocd-application-controller
)
output: >
SecretProviderClass created by unexpected principal
(user=%k8s.user.name namespace=%k8s.target.namespace name=%k8s.target.name)
priority: WARNING
Expected Behaviour
| Signal | Before hardening | After hardening |
|---|---|---|
kubectl get secrets -l secrets-store.csi.k8s.io/managed=true |
Returns synced secrets in many namespaces | Empty (sync disabled) or restricted to approved namespace |
Developer creates SecretProviderClass with secretObjects |
Succeeds | Blocked by Kyverno — not in approved namespace |
| Provider DaemonSet pod on control-plane node | Possible | Blocked by nodeSelector |
Provider IAM role has secretsmanager:* on * |
Common default | Scoped per-workload via IRSA |
| Falco alert on unexpected SecretProviderClass | No alert | WARNING fires immediately |
Verification:
# Confirm sync-to-Secret is disabled
kubectl get SecretProviderClass --all-namespaces -o json | \
jq '[.items[] | select(.spec.secretObjects and (.spec.secretObjects | length > 0))] | length'
# Expected: 0 (unless in approved namespace)
# Confirm no provider pods on control-plane
kubectl get pods -n kube-system -l app=csi-secrets-store-provider-aws \
-o jsonpath='{.items[*].spec.nodeName}' | tr ' ' '\n' | while read node; do
role=$(kubectl get node "$node" -o jsonpath='{.metadata.labels.node-role\.kubernetes\.io/control-plane}')
[[ -n "$role" ]] && echo "WARNING: Provider pod on control-plane node: $node"
done
Trade-offs
| Aspect | Benefit | Cost | Mitigation |
|---|---|---|---|
| Disable sync-to-Secret globally | Eliminates etcd exposure for externally-managed secrets | Legacy apps that only accept env vars break | Migrate to envFrom with secretKeyRef pointing to a file-mounted secret; or use the External Secrets Operator as a migration path |
| Per-workload IRSA scoping | Blast radius of provider compromise limited to one workload’s secrets | More IAM roles to manage | Automate role creation via Terraform module in service onboarding |
| Kyverno restrict SPC creation | Prevents lateral movement via rogue SecretProviderClass | Slows self-service secret configuration | Build a GitOps template for SPC creation; platform team reviews and merges |
| Provider DaemonSet off control-plane | Limits control-plane compromise blast radius | None — control-plane nodes have no legitimate need for this DaemonSet | Apply universally |
Failure Modes
| Failure | Symptom | Detection | Recovery |
|---|---|---|---|
| Pod fails to mount secret after IRSA scope change | Pod stays in ContainerCreating; events show CSI mount failure | kubectl describe pod shows “failed to mount secret”; provider logs show IAM denied |
Verify IRSA policy includes the specific secret ARN; check the pod’s service account has the IRSA annotation |
| Kyverno blocks GitOps controller from creating SPC | ArgoCD/Flux cannot sync SecretProviderClass manifests; app stays out of sync | ArgoCD shows SyncFailed; Kyverno audit log shows denial for argocd SA | Add ArgoCD/Flux service account to the Kyverno exception list |
| Sync disabled but app expects env vars from synced Secret | App container fails with missing env var | Container logs show KeyError / missing env; pod restarts | Either re-enable sync for this specific namespace (with restricted RBAC) or migrate the app to file-based secret consumption |
| Provider rotation polling rate-limited | Pods continue with stale secrets; rotation silently stopped | Check provider logs for rate limit errors; CSI driver metrics show stale age | Reduce polling frequency; implement exponential backoff in provider config; alert on rotation age exceeding TTL |
Related Articles
- Kubernetes Secrets Management — the broader secrets management landscape including native Secrets, external providers, and Vault integration
- External Secrets Operator Hardening — an alternative to the CSI driver with different security characteristics
- AWS IRSA Workload Identity — scoping pod-level IAM permissions used by the CSI driver provider
- Kyverno Policy Development — writing the policies that restrict SecretProviderClass creation
- Kubernetes Service Account Token Security — the service account tokens used to authenticate to Vault and cloud providers from the CSI driver