Cross-Cutting Guides intermediate 12 min read

Platform Team Secrets Injection: Centralized Patterns for Developer Self-Service

secrets-management platform-engineering external-secrets-operator vault kubernetes cicd

Platform Team Secrets Injection: Centralized Patterns for Developer Self-Service

Problem

Developers need database passwords, API keys, TLS certificates, and service account tokens to run their applications. Left to their own devices, they reach for the path of least resistance: hardcoding credentials in environment variable files, committing them as Kubernetes Secrets in git, passing them through CI environment variables that get logged, or copy-pasting them from Slack. Each of these shortcuts creates a new exposure surface that outlives the intent.

The platform team’s job is to make the secure path the easy path. That means providing patterns where the application gets access to the credential at runtime without any human — including the developer who requested it — ever seeing the raw value. The credential never touches git, never appears in a CI log, never sits in a Kubernetes Secret that a developer with kubectl get secret access can decode.

There are four distinct injection patterns that cover most workload types:

  1. External Secrets Operator — syncs from a secret store to native Kubernetes Secrets, managed by the platform team
  2. Vault Agent sidecar — injects a sidecar that handles authentication and renders secrets to files
  3. Secrets Store CSI driver — mounts secrets directly from an external store as volume files
  4. CI/CD OIDC federation — gives pipelines short-lived credentials via identity federation, no stored secrets

Each pattern targets different workload types and threat models. Most mature platform setups use two or three of them simultaneously.

Pattern 1: External Secrets Operator

External Secrets Operator (ESO) runs as a controller in the cluster. It reads from a SecretStore or ClusterSecretStore and creates or updates native Kubernetes Secret objects on a configurable sync interval. The developer writes an ExternalSecret resource that declares what they need; the platform team’s ClusterSecretStore controls how ESO authenticates to the upstream secret store.

For a detailed ESO hardening walkthrough, see External Secrets Operator Hardening.

ClusterSecretStore for Vault

The platform team provisions the ClusterSecretStore once per cluster. This is the authentication boundary: developers cannot change how the store authenticates.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.internal.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-controller"
          serviceAccountRef:
            name: "external-secrets"
            namespace: "external-secrets-system"

The Vault role eso-controller is configured with a policy that allows read-only access to every path the operator needs to serve. ESO authenticates using its own service account, not any application service account. The controller’s Vault token is never exposed to application namespaces.

ClusterSecretStore for AWS SSM Parameter Store

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-ssm-backend
spec:
  provider:
    aws:
      service: ParameterStore
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: "external-secrets"
            namespace: "external-secrets-system"

With IRSA (IAM Roles for Service Accounts), the ESO controller’s service account maps to an IAM role with ssm:GetParameter and ssm:GetParametersByPath permissions. No AWS credentials are stored in the cluster.

ExternalSecret — developer-facing resource

The developer’s team owns the ExternalSecret in their namespace. They declare what path they need and what Kubernetes Secret to create:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payments-db-credentials
  namespace: payments
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: payments-db-secret
    creationPolicy: Owner
    deletionPolicy: Retain
    template:
      type: Opaque
      data:
        DATABASE_URL: "postgres://{{ .username }}:{{ .password }}@db.internal:5432/payments"
  data:
    - secretKey: username
      remoteRef:
        key: "secret/data/payments/database"
        property: username
    - secretKey: password
      remoteRef:
        key: "secret/data/payments/database"
        property: password

The resulting Kubernetes Secret is owned by the ExternalSecret object. If the ExternalSecret is deleted, the Secret is deleted too (with deletionPolicy: Retain, the Secret persists for safe migration periods). The developer never calls vault kv get or sees the raw credential value — they reference the Kubernetes Secret name in their pod spec.

The refreshInterval governs how often ESO re-reads from the upstream store. For credentials that rotate on a schedule, set the interval shorter than the rotation period. ESO updates the Secret in place; pods need to either watch the Secret volume mount for changes or restart on Secret rotation (use Reloader or a rolling update trigger).

Pattern 2: Vault Agent Sidecar Injection

Vault Agent runs as a sidecar container injected into application pods via a mutating admission webhook. The agent handles Vault authentication transparently, renders secrets to files using Go templates, and keeps the rendered files current by renewing leases and re-rendering on rotation. The application reads secrets from the filesystem — it never calls Vault directly and never needs a Vault token.

For foundational Vault hardening, see Vault Architecture Hardening.

Enabling injection with annotations

The developer adds annotations to their pod spec (typically in a Deployment). The platform team controls the webhook configuration that interprets these annotations.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: payments
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "payments-api"
        vault.hashicorp.com/agent-inject-secret-db.env: "secret/data/payments/database"
        vault.hashicorp.com/agent-inject-template-db.env: |
          {{- with secret "secret/data/payments/database" -}}
          export DATABASE_HOST="{{ .Data.data.host }}"
          export DATABASE_USER="{{ .Data.data.username }}"
          export DATABASE_PASSWORD="{{ .Data.data.password }}"
          {{- end }}
        vault.hashicorp.com/agent-pre-populate-only: "false"
        vault.hashicorp.com/agent-cache-enable: "true"
    spec:
      serviceAccountName: payments-api
      containers:
        - name: api-server
          image: payments-api:v1.2.0
          command: ["/bin/sh", "-c"]
          args:
            - source /vault/secrets/db.env && exec /app/server

The annotation vault.hashicorp.com/role references a Vault Kubernetes auth role. That role maps the pod’s service account (payments-api in namespace payments) to a Vault policy. The injection webhook uses the pod’s projected service account token to authenticate to Vault — no static credentials anywhere in the manifest.

Vault policy and Kubernetes auth role

The platform team creates the policy and auth role via Terraform or the Vault API. The developer never touches these resources.

resource "vault_policy" "payments_api" {
  name = "payments-api"

  policy = <<EOT
path "secret/data/payments/database" {
  capabilities = ["read"]
}

path "secret/metadata/payments/database" {
  capabilities = ["read"]
}
EOT
}

resource "vault_kubernetes_auth_backend_role" "payments_api" {
  backend                          = "kubernetes"
  role_name                        = "payments-api"
  bound_service_account_names      = ["payments-api"]
  bound_service_account_namespaces = ["payments"]
  token_policies                   = ["payments-api"]
  token_ttl                        = 3600
  token_max_ttl                    = 86400
}

The Vault policy is narrowly scoped: the payments-api role can only read the payments/database path. It cannot list other paths, cannot write, and cannot access secrets for other teams. The bound_service_account_namespaces constraint ensures that even if an attacker creates a payments-api service account in a different namespace, it cannot authenticate to this role.

Secret file rendering

Vault Agent writes to /vault/secrets/ in the shared volume between the init container and the main container. The template renders secrets into the exact format the application expects — shell exports, a .env file, a JSON config blob, a PKCS12 bundle. The application’s startup command sources the rendered file. No application code change is needed to consume Vault-injected secrets.

For certificate injection specifically, the agent can render the cert and key into PEM files that TLS libraries read directly:

vault.hashicorp.com/agent-inject-secret-tls.crt: "pki/issue/payments-api"
vault.hashicorp.com/agent-inject-template-tls.crt: |
  {{- with pkiCert "pki/issue/payments-api" "common_name=api.payments.internal" "ttl=72h" -}}
  {{ .Cert }}
  {{- end }}

Pattern 3: Secrets Store CSI Driver

The Secrets Store CSI driver mounts secrets from an external store directly as volume files, without creating a Kubernetes Secret object. This avoids the intermediate Kubernetes Secret entirely — nothing is persisted in etcd. The driver is a DaemonSet that runs on every node; when a pod with a SecretProviderClass volume is scheduled, the CSI driver on that node authenticates to the secret store and mounts the secret content into the pod’s filesystem.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: payments-vault-secrets
  namespace: payments
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.internal.example.com"
    roleName: "payments-api"
    objects: |
      - objectName: "db-password"
        secretPath: "secret/data/payments/database"
        secretKey: "password"
      - objectName: "api-key"
        secretPath: "secret/data/payments/external-api"
        secretKey: "key"

The pod spec references the CSI volume:

spec:
  volumes:
    - name: secrets-store
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "payments-vault-secrets"
  containers:
    - name: api-server
      volumeMounts:
        - name: secrets-store
          mountPath: "/mnt/secrets"
          readOnly: true

At mount time, the CSI driver authenticates to Vault using the pod’s service account token (the same Kubernetes auth flow as Vault Agent), reads the specified paths, and presents the values as files at /mnt/secrets/db-password and /mnt/secrets/api-key. The application reads files. No Vault SDK, no HTTP client code, no token management.

The CSI driver supports Vault, AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager through separate provider plugins. A single SecretProviderClass can pull from multiple stores simultaneously if an application needs credentials from different providers.

The main operational caveat with the CSI driver: secrets are only mounted while the pod is running. They are not automatically updated in place when the upstream secret rotates — the pod must be restarted to pick up new values. For credentials with short TTLs or frequent rotation, Vault Agent (which handles lease renewal and re-rendering) is often the better choice. For static secrets with infrequent rotation, the CSI driver’s simpler operational model is preferable.

Pattern 4: CI/CD Secrets via OIDC Federation

CI/CD pipelines historically required stored secrets: a Vault token, an AWS access key, a service account credential stored as a CI environment variable or repository secret. These long-lived credentials are a persistent risk — they don’t expire automatically, they’re often broader than necessary, and a compromise of the CI system exposes them indefinitely.

OIDC federation eliminates stored credentials in CI. The CI provider (GitHub Actions, GitLab CI, Buildkite) issues a short-lived OIDC token for each job run. The pipeline presents this token to Vault or the cloud provider’s IAM system, which validates the token’s claims and issues a short-lived credential in return. The credential lives only for the duration of the job.

For CI/CD security patterns more broadly, see Golden Path Security.

GitHub Actions → Vault OIDC

The platform team configures Vault’s JWT auth method to trust GitHub’s OIDC issuer:

vault auth enable jwt

vault write auth/jwt/config \
  oidc_discovery_url="https://token.actions.githubusercontent.com" \
  bound_issuer="https://token.actions.githubusercontent.com"

vault write auth/jwt/role/ci-payments \
  role_type="jwt" \
  bound_audiences="https://vault.internal.example.com" \
  user_claim="sub" \
  bound_claims='{"repository": "example-org/payments-service", "ref": "refs/heads/main"}' \
  policies="ci-payments-read" \
  ttl="15m"

The bound_claims constraint locks the role to a specific repository and branch. A workflow running from a fork or a non-main branch cannot authenticate to this role. The ttl of 15 minutes means the Vault token expires shortly after the job ends regardless of whether the workflow explicitly revokes it.

The workflow itself needs no stored credentials:

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - name: Authenticate to Vault
        uses: hashicorp/vault-action@v3
        with:
          url: https://vault.internal.example.com
          role: ci-payments
          method: jwt
          jwtGithubAudience: https://vault.internal.example.com
          secrets: |
            secret/data/payments/deploy token | DEPLOY_TOKEN ;
            secret/data/payments/registry password | REGISTRY_PASSWORD

The vault-action step exchanges the GitHub-issued OIDC token for a Vault token, reads the specified secrets, and injects them as masked environment variables for subsequent steps. The secrets appear in the runner’s environment for the duration of the job and are never written to disk or logged.

For AWS deployments, the same pattern applies directly to IAM without a Vault intermediary. GitHub Actions supports aws-actions/configure-aws-credentials with OIDC, which federates the GitHub token directly to an IAM role.

Governance Model: Secret Lifecycle Ownership

Secrets injection patterns only work reliably when ownership is clear. Without defined ownership, secrets accumulate in stores that nobody audits, rotation schedules slip, and decommissioned services leave orphaned credentials with active IAM permissions.

The governance model that works at scale:

Platform team owns:

  • The secret store infrastructure (Vault clusters, AWS Secrets Manager configuration)
  • The authentication backends (Kubernetes auth, JWT/OIDC auth, AWS auth)
  • The top-level path structure in the secret store (secret/data/<team>/<environment>/<service>)
  • ClusterSecretStore and CSI driver configuration
  • Rotation automation for platform-managed secrets (database passwords for shared databases, TLS CA roots)
  • Audit and compliance reporting

Application team owns:

  • The values written to their allocated paths
  • Their ExternalSecret and SecretProviderClass resources
  • Rotation of secrets they own (application-specific API keys, third-party service credentials)
  • Notification to the platform team when a secret is no longer needed (decommissioning)

Nobody manually reads production secret values. Break-glass access to raw secret values requires an approved access request, is time-limited, and is logged. Regular operations never require a human to see a plaintext credential.

Self-Service Secret Path Provisioning via Terraform

The process for a team to get a new secret path follows a PR-based workflow against the platform team’s Terraform repository. This creates an audit trail, enforces naming conventions, and prevents teams from writing secrets to paths where they don’t have a corresponding policy.

The platform team provides a Terraform module:

module "payments_secret_path" {
  source = "git::https://github.com/example-org/platform-tf-modules.git//vault-secret-path?ref=v2.1.0"

  team        = "payments"
  service     = "api-server"
  environment = "production"
  secret_names = ["database", "stripe-api-key", "internal-signing-key"]

  kubernetes_namespaces    = ["payments"]
  kubernetes_service_accounts = ["payments-api"]

  ci_repository = "example-org/payments-service"
  ci_ref_pattern = "refs/heads/main"
}

The module creates:

  • Vault paths at secret/data/payments/production/api-server/<secret_name> for each entry in secret_names
  • A Vault policy scoped to those exact paths (read-only for the application role, read-write for the team’s human role)
  • A Kubernetes auth role binding payments-api service accounts in the payments namespace to the application policy
  • A JWT auth role binding the CI repository and ref to a CI-scoped policy (read-only, 15-minute TTL)
  • An ESO ClusterSecretStore reference in the output for use in ExternalSecret resources

The application team submits a PR to the platform repository adding this module call. The platform team reviews it for naming convention compliance and appropriate scope, then approves and merges. Terraform apply runs in CI. The team then writes their secret values through the Vault UI or CLI (a write-only operation that doesn’t require reading existing values) and creates their ExternalSecret or pod annotations referencing the provisioned paths.

Secret Sprawl Detection

Provisioning paths and injection patterns don’t prevent sprawl on their own. Applications get decommissioned without cleaning up their Vault policies. ESO ExternalSecret resources reference paths that no longer exist in the upstream store. Vault leases accumulate from CI jobs that were cancelled mid-run without revoking their tokens.

Auditing Vault for unused leases and stale policies

List all active leases and find those that haven’t been renewed recently:

vault list sys/leases/lookup/auth/kubernetes/login
vault lease lookup <lease_id>

For systematic auditing, use the Vault audit log. Enable it if not already active:

vault audit enable file file_path=/var/log/vault/audit.log

Parse the audit log to find Vault roles that haven’t authenticated in 90 days:

jq -r 'select(.type == "response") | select(.request.path | startswith("auth/")) | .auth.metadata.role // empty' /var/log/vault/audit.log \
  | sort | uniq -c | sort -rn

Cross-reference the list of active roles against the current Terraform state to find roles that exist in Vault but are not managed by any Terraform resource — these are candidates for deletion.

Auditing ESO for stale ExternalSecret references

ESO reports sync status on the ExternalSecret resource’s status conditions. A stale or broken reference shows up as Ready: False with a reason like SecretSyncedError:

kubectl get externalsecrets -A \
  -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.conditions[0].reason}{"\n"}{end}' \
  | grep -v SecretSynced

Any ExternalSecret not in SecretSynced state has a broken upstream reference. Either the path was deleted without deleting the ExternalSecret, or the store authentication has drifted. These should be investigated and either fixed or deleted — a broken ExternalSecret is often the residue of a decommissioned service that still has a live Vault policy.

Schedule this audit as a weekly CronJob that posts results to a Slack channel or opens GitHub issues for stale references. Platform teams that let this run manually find it never runs.

Detecting Kubernetes Secrets not owned by ESO

Kubernetes Secrets that exist without an ExternalSecret owner are candidates for sprawl — they may have been created manually or by a Helm chart that hardcoded credentials.

kubectl get secrets -A \
  -o json \
  | jq -r '.items[] | select(.metadata.ownerReferences == null or (.metadata.ownerReferences | map(select(.kind == "ExternalSecret")) | length == 0)) | "\(.metadata.namespace)\t\(.metadata.name)\t\(.type)"' \
  | grep -v "kubernetes.io/service-account-token\|helm.sh/release"

Secrets that don’t belong to ESO, aren’t service account tokens, and aren’t Helm release metadata should be reviewed. Most will be legitimate (TLS secrets managed by cert-manager, Sealed Secrets) but the review surfaces anything manually created or left over from a migration away from hardcoded credentials.

Operational Considerations

Secret rotation and pod restart. ESO updates the Kubernetes Secret in place when the upstream value changes. The pod’s environment variables are not updated — environment variables are injected at pod start and are immutable for the lifetime of the container. If the application reads credentials from environment variables (not from files), it must restart to pick up rotated values. Use Reloader to automatically roll Deployments when their referenced Secrets change. For file-based injection (Vault Agent, CSI), the agent re-renders the file and sends SIGHUP if configured.

Secret store availability and pod scheduling. With the CSI driver, if the secret store is unreachable at pod scheduling time, the pod fails to start — the volume mount blocks. This means a Vault outage prevents new pod scheduling. Design the Vault cluster for high availability (three or five nodes, spread across availability zones). For Vault Agent and ESO, the agent caches the secret locally, so a transient Vault outage doesn’t prevent pod startup if the agent has a valid cached value.

Least-privilege Vault policies. Every Vault policy scoped to an application role should grant only read capability on specific paths. Never use wildcards in production policies (path "secret/data/*" is not acceptable). Use Vault’s vault policy fmt to enforce consistent formatting and vault policy read <name> to audit what each policy actually allows. Review policies during the secret path provisioning PR — it is the most natural point to catch overly broad grants before they reach production.

No developer access to production secret values. The injection patterns described here make it possible to run without developers ever reading production credentials. Enforce this: audit Vault human roles for production read capability and remove it. Break-glass access should require a ServiceNow ticket, create a time-limited policy, and generate an alert to the security team. The fact that developers can deploy applications that use production secrets without being able to read those secrets is the key security property these patterns enable.