HashiCorp Vault Architecture and Hardening: Seal Configuration, Audit Logging, and Root Token Elimination

The Problem

Vault is the most widely deployed secrets management system in production Kubernetes and cloud environments. Its architecture offers strong security guarantees — encryption at rest, tight ACL policies, dynamic secret generation, audit logging — but those guarantees only hold when Vault is configured correctly. Most production Vault deployments are not configured correctly.

The specific failures are predictable: unseal keys stored in a shared password manager or Slack DM, a root token that survived the initial vault operator init session and got pasted into a CI/CD secret, an audit log that was never enabled because it was not obvious how to configure it and the team was busy getting Vault running. Each of these failures individually represents total secret compromise under the right attacker conditions. Together, they mean an attacker with a single foothold — a compromised developer laptop, a leaked S3 bucket containing the Terraform state file, a long-lived AWS credential from a rotations-never provisioner — can read every secret your application stack holds.

This is not hypothetical. The 2024 Internet Archive breach exposed secrets stored in a GitLab configuration file. The 2023 CircleCI breach involved token theft from a compromised engineer laptop that had Vault tokens cached to disk. The pattern is consistent: the secrets manager is never the direct vulnerability — the humans, credentials, and processes surrounding it are.

Vault’s security model has three properties that matter most:

Sealed state. All data in Vault’s storage backend is encrypted using the root key, which is itself encrypted and stored only in memory while Vault is running. When Vault is sealed — immediately after startup, after an explicit vault operator seal, or after a crash — the root key is gone from memory. The storage backend (Raft, Consul, DynamoDB, PostgreSQL) contains only ciphertext. An attacker who compromises the storage backend while Vault is sealed gets an encrypted blob they cannot decrypt without the unseal key. This property is what makes Vault’s data-at-rest story fundamentally different from dropping a database backup on an S3 bucket.

Unseal process. By default, Vault uses Shamir’s Secret Sharing. During vault operator init, you specify -key-shares (N) and -key-threshold (M). Vault generates a random root key, then uses Shamir’s algorithm to split it into N shares such that any M shares can reconstruct the original key. Each share holder receives one share. At startup, Vault accepts shares one at a time: after M shares are submitted, the root key is reconstructed in memory and Vault unseals. The security property: no single person knows the root key, and compromising fewer than M shares is useless — Shamir’s Secret Sharing provides information-theoretic security below the threshold. The operational problem: someone has to submit those shares every time Vault restarts. That someone has to be reachable. Those shares have to be stored somewhere.

Root token. Created during vault operator init. Has unrestricted access to every Vault operation — read any secret, enable or disable any auth method, enable or disable audit logging, modify any ACL policy. ACL policies do not apply to the root token. If audit logging is configured, the root token’s actions are logged, but there is no way to restrict what the root token can do. It is the master credential. If it exists and someone has it, they own the Vault instance completely.

The specific attack scenarios that follow all exploit exactly these three properties when they are misconfigured.

Attack Paths Against Unprotected Vault Instances

Attack 1: Unseal key exfiltrated from shared credential store

A team initialises Vault with five shares and a threshold of three. The five shares are pasted into a shared 1Password vault under “Vault Unseal Keys.” Twelve engineers have access to that 1Password vault. One engineer’s laptop is compromised via a phishing attack that installs a keylogger. The attacker steals the 1Password master password, extracts all five unseal keys, and waits.

Three weeks later, the Vault pod is restarted due to a Kubernetes node rotation. Vault is in sealed state. The team manually unseals it — standard operational procedure. The attacker, who already has the keys, now unseals a second Vault instance they’ve spun up with a backup of the storage backend (extracted from the DynamoDB table or S3 Raft snapshot that Vault was writing to):

# Attacker has the three unseal keys and a copy of Vault's storage backend
# Spin up a standalone Vault instance pointing to the copied backend
vault server -config=/tmp/attacker-vault.hcl &

# Unseal using the stolen keys
vault operator unseal <key_share_1>
vault operator unseal <key_share_2>
vault operator unseal <key_share_3>

# Now logged in with the initial root token from the init output:
export VAULT_TOKEN=hvs.CAESIHJ5...

# Read every secret in the KV store
vault kv list secret/
vault kv get secret/production/database
vault kv get secret/production/aws-credentials
vault kv get secret/production/stripe-api-key

The attacker never touched the production Vault instance. They worked entirely from a backup of the storage backend. The production audit log — if it was even configured — shows nothing. The only detection signal is in the object storage access logs showing the storage backend snapshot being downloaded, which nobody was monitoring.

Attack 2: Root token persisted in CI/CD secrets

The operator who ran vault operator init saved the output to a local file to extract the unseal keys. The root token was in that same file. Standard next step: add the root token to the CI/CD pipeline as VAULT_ROOT_TOKEN so the deployment pipeline can configure Vault. “We’ll remove it once we set up proper auth” — that sentence has never been followed by removing it.

The CI/CD platform is later compromised via a malicious dependency in the build pipeline. The attacker reads the environment variables from a running job:

# Attacker has code execution in the CI runner
env | grep VAULT
# VAULT_ADDR=https://vault.internal.company.com:8200
# VAULT_ROOT_TOKEN=hvs.CAESIHJ5...

# No ACL policies apply to root token
vault login $VAULT_ROOT_TOKEN

# Enumerate the entire secrets hierarchy
vault kv list secret/
vault kv list secret/production/
vault kv list secret/production/databases/

# Read everything
vault kv get -format=json secret/production/databases/postgres-rw

# Disable audit logging before doing anything else
vault audit disable file/
vault audit disable syslog/

# Now read more secrets with no forensic trail
vault kv get secret/production/encryption-keys/customer-data-key

The vault audit disable step is critical. The root token can disable audit logging — an operation that requires root-level privilege, but the root token bypasses all policies. Once audit logging is disabled, all subsequent Vault operations are invisible. The attack proceeds without leaving a forensic trail in Vault’s own audit system.

Attack 3: No audit log means no forensics after breach

A production incident reveals that an application’s database credentials were used from an unexpected IP address. The database credentials are stored in Vault. The security team needs to determine: was Vault breached? When? What else was accessed?

vault audit list
# No audit devices are enabled.

Without an audit log, there is no way to answer any of these questions from Vault’s side. The investigation is limited to application logs (which may not record which secrets were read), database access logs (which show credential usage but not the Vault access that preceded it), and network flow data (which shows connections to Vault’s API port but not what was requested). A forensic investigation that should take hours to determine scope takes weeks of correlation work across fragmentary data sources, and may never produce a conclusive answer.

Threat Model

The blast radius of each failure mode:

Unseal keys compromised (from shared password manager, Slack, email, backup of Terraform state): sealed Vault storage backend fully decryptable. Impact: every secret in Vault exposed to anyone who obtains the keys and a copy of the backend.
Root token not revoked: unrestricted access to all Vault operations, ACL policies bypassed, audit logging disableable. Impact: complete Vault compromise — secrets, auth configuration, and the ability to erase forensic evidence.
Audit logging not configured: all secret read, write, and authentication operations invisible. Impact: forensic blindness — breach scope cannot be determined after the fact.
Storage backend exposed without unseal key compromise: data at rest is encrypted and unreadable. Impact: minimal (encrypted blob only). Storage backend access alone, without unseal keys, does not expose secrets.
Auto-unseal KMS key compromised without Vault instance access: KMS alone is insufficient — Vault still requires decrypting the encrypted root key stored in Vault’s keyring. Impact: low in isolation; high if combined with storage backend access.

The compound failure — root token not revoked, audit log disabled or absent, unseal keys in shared storage — is a common production configuration that provides no meaningful security despite Vault’s underlying cryptographic guarantees.

Hardening Configuration

1. Auto-Unseal with AWS KMS

Auto-unseal replaces the Shamir key-sharing process with a cloud KMS call. Instead of splitting the root key into shares held by humans, Vault wraps the root key using a KMS key. On startup, Vault calls KMS to unwrap the root key and unseals without human intervention. The security boundary shifts from “people who hold key shares” to “AWS IAM policies controlling KMS key access.”

This is a significant improvement. IAM policies are auditable, rotatable, and enforced by AWS’s own infrastructure. Unseal key shares stored in a password manager are none of those things.

# /etc/vault/vault.hcl

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/mrk-abc123def456"
  # Multi-region KMS key (mrk-) enables cross-region redundancy
  # If us-east-1 KMS is unavailable, the same key material is accessible
  # from replica regions configured in KMS key policy
}

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"

  retry_join {
    leader_api_addr = "https://vault-2.internal:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-3.internal:8200"
  }
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault/tls/vault.crt"
  tls_key_file  = "/etc/vault/tls/vault.key"
  tls_min_version = "tls13"

  # Disable unauthenticated health check endpoint leaking version info
  unauthenticated_metrics_access = false
}

api_addr     = "https://vault-1.internal:8200"
cluster_addr = "https://vault-1.internal:8201"

# Prevent core dump exposure of in-memory root key
disable_mlock = false

ui = false  # Disable the web UI if not needed; reduces attack surface

The Vault instance’s IAM role needs exactly two KMS permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/mrk-abc123def456"
    }
  ]
}

No other IAM permissions. The KMS key policy itself should restrict usage to the specific Vault instance’s IAM role and require that requests come from within the VPC using a aws:SourceVpc condition.

With auto-unseal configured, Vault’s startup log shows the difference immediately:

[INFO]  core: stored unseal keys supported, attempting fetch
[INFO]  core: vault is unsealed
[INFO]  core: entering standby mode

There is no prompt for unseal keys, no manual intervention required, and no humans holding key material.

2. Bootstrap Script: Root Token Revocation

The bootstrap script is the most operationally dangerous part of the Vault lifecycle. It runs once, generates the root token, and provides the window to configure initial auth methods before the root token is permanently revoked. The script must be idempotent about whether it runs, it must store recovery keys securely before revoking the root token, and it must revoke the root token as its final authenticated action.

#!/bin/bash
# vault-bootstrap.sh — run once per Vault cluster initialization
# Requires: VAULT_ADDR set, OIDC client credentials in environment
set -euo pipefail

VAULT_ADDR="${VAULT_ADDR:-https://vault.internal:8200}"
INIT_OUTPUT="/tmp/vault-init-$(date +%Y%m%d%H%M%S).json"

# Check if Vault is already initialized
if vault status -format=json 2>/dev/null | jq -e '.initialized == true' > /dev/null; then
  echo "Vault already initialized. Exiting."
  exit 0
fi

echo "Initializing Vault with auto-unseal (KMS) and recovery keys..."
vault operator init \
  -recovery-shares=5 \
  -recovery-threshold=3 \
  -format=json > "$INIT_OUTPUT"

echo "Vault initialized. Extracting credentials..."

ROOT_TOKEN=$(jq -r '.root_token' "$INIT_OUTPUT")
RECOVERY_KEYS=$(jq -r '.recovery_keys_b64[]' "$INIT_OUTPUT")

# Store recovery keys — these are the break-glass credential if KMS is lost
# Each key goes to a different secure location / key holder
# This example writes them to SSM Parameter Store; in practice distribute to people
COUNTER=1
while IFS= read -r key; do
  aws ssm put-parameter \
    --name "/vault/recovery-key-${COUNTER}" \
    --value "$key" \
    --type "SecureString" \
    --key-id "arn:aws:kms:us-east-1:123456789012:key/separate-recovery-key" \
    --overwrite
  echo "Recovery key ${COUNTER} stored in SSM"
  ((COUNTER++))
done <<< "$RECOVERY_KEYS"

export VAULT_TOKEN="$ROOT_TOKEN"

# Enable audit logging FIRST — before any other configuration
# This ensures all subsequent bootstrap operations are logged
vault audit enable file \
  file_path=/var/log/vault/audit.log \
  log_raw=false \
  hmac_accessor=true

vault audit enable syslog \
  tag="vault-audit" \
  facility="AUTH"

echo "Audit logging enabled."

# Enable OIDC auth method for human operators
vault auth enable oidc

vault write auth/oidc/config \
  oidc_discovery_url="https://accounts.google.com" \
  oidc_client_id="${OIDC_CLIENT_ID}" \
  oidc_client_secret="${OIDC_CLIENT_SECRET}" \
  default_role="operator"

vault write auth/oidc/role/operator \
  bound_audiences="${OIDC_CLIENT_ID}" \
  allowed_redirect_uris="${VAULT_ADDR}/ui/vault/auth/oidc/oidc/callback" \
  bound_claims_type="string" \
  bound_claims="{\"hd\": \"company.com\"}" \
  user_claim="email" \
  policies="operator" \
  ttl="8h"

# Enable Kubernetes auth for application workloads
vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Write the operator policy — broad but auditable, not root
vault policy write operator - << 'POLICY'
# Full access to secrets — operators need this for break-glass scenarios
path "secret/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Auth method management
path "auth/*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Policy management
path "sys/policies/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Audit log management — operators can view but not disable without explicit sudo
path "sys/audit" {
  capabilities = ["read", "list"]
}

# Allow renewing own tokens
path "auth/token/renew-self" {
  capabilities = ["update"]
}
POLICY

echo "Policies and auth methods configured."

# Verify OIDC auth works before revoking root token
# In practice: a team member logs in via OIDC and verifies access here
echo "Verify OIDC authentication is functional before proceeding."
echo "Press ENTER to revoke root token after verification..."
read -r

# THE CRITICAL STEP: Revoke the root token
echo "Revoking root token..."
vault token revoke "$ROOT_TOKEN"

# Verify revocation
if vault token lookup "$ROOT_TOKEN" 2>&1 | grep -q "403"; then
  echo "Root token successfully revoked. Verification: 403 returned."
else
  echo "WARNING: Root token revocation verification failed. Investigate immediately."
  exit 1
fi

# Destroy the init output file
shred -vzu "$INIT_OUTPUT"
echo "Initialization file destroyed."

unset VAULT_TOKEN ROOT_TOKEN RECOVERY_KEYS
echo "Bootstrap complete. Root token eliminated."

The sequencing matters: audit logging is enabled before any other configuration, so the bootstrap operations themselves — policy writes, auth method configuration — are captured in the audit trail. The root token is revoked as the last authenticated action, after verifying that an alternative auth method works. Revocation is verified by confirming the lookup returns a 403.

3. Audit Log Configuration and Immutable Shipping

Vault’s audit log records every authenticated operation: secret reads and writes, token creation and renewal, auth method logins, policy changes. Each log entry is a JSON object containing the request path, the operation, the token accessor (not the token itself — accessors are HMAC’d by default), the client IP, and the response status. What the audit log does not record by default is the actual secret value — log_raw=false is the default and must not be changed.

# Verify audit configuration after bootstrap
vault audit list -format=json | jq '.'
# Expected output:
# {
#   "file/": {
#     "type": "file",
#     "description": "",
#     "options": {
#       "file_path": "/var/log/vault/audit.log",
#       "hmac_accessor": "true",
#       "log_raw": "false"
#     }
#   },
#   "syslog/": {
#     "type": "syslog",
#     "options": {
#       "facility": "AUTH",
#       "tag": "vault-audit"
#     }
#   }
# }

The audit log file on disk is a single point of failure if the Vault node is compromised. An attacker with root access to the node can delete or truncate it. Ship the audit log to an immutable external store as the primary forensic record:

# /etc/fluent-bit/conf.d/vault-audit.conf
# Fluent Bit configuration for shipping Vault audit logs to S3 with Object Lock

[INPUT]
    Name              tail
    Path              /var/log/vault/audit.log
    Parser            json
    Tag               vault.audit
    Refresh_Interval  5
    DB                /var/lib/fluent-bit/vault-audit.db
    # DB tracks position; surviving a restart doesn't re-ship old events

[FILTER]
    Name   record_modifier
    Match  vault.audit
    Record host ${HOSTNAME}
    Record environment production

[OUTPUT]
    Name                          s3
    Match                         vault.audit
    bucket                        vault-audit-logs-immutable
    region                        us-east-1
    use_put_object                On
    upload_timeout                60s
    store_dir                     /var/lib/fluent-bit/s3-buffer
    s3_key_format                 /vault-audit/%Y/%m/%d/%H/$UUID.json
    # The S3 bucket must have Object Lock configured in COMPLIANCE mode
    # with a minimum retention of 1 year. Compliance mode prevents even
    # account root from deleting objects before retention expires.

The S3 bucket receiving audit logs must have Object Lock in Compliance mode. Governance mode allows account administrators to override the lock — an attacker with AWS root account access can still delete records. Compliance mode does not allow deletion even by the account root user before the retention period expires:

# Configure S3 bucket for immutable audit log retention
aws s3api create-bucket \
  --bucket vault-audit-logs-immutable \
  --region us-east-1 \
  --object-lock-enabled-for-bucket

aws s3api put-object-lock-configuration \
  --bucket vault-audit-logs-immutable \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Years": 1
      }
    }
  }'

# Block public access
aws s3api put-public-access-block \
  --bucket vault-audit-logs-immutable \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

4. Application ACL Policies — Principle of Least Privilege

Every application that reads secrets from Vault should have its own policy scoped to exactly the paths it needs, with exactly the operations it requires. The common failure is creating a read-all policy and assigning it to all applications because it is easier.

# policies/app-payments-service.hcl
# Payment service: reads DB credentials, reads Stripe key, nothing else

path "secret/data/production/databases/payments-postgres" {
  capabilities = ["read"]
}

path "secret/data/production/stripe" {
  capabilities = ["read"]
}

# Explicitly deny listing — prevents enumeration of secret hierarchy
path "secret/metadata/production/*" {
  capabilities = ["deny"]
}

# Allow the service to renew its own token
path "auth/token/renew-self" {
  capabilities = ["update"]
}

# Kubernetes auth role for the payment service
vault write auth/kubernetes/role/payments-service \
  bound_service_account_names=payments-service \
  bound_service_account_namespaces=production \
  policies=app-payments-service \
  ttl=1h \
  max_ttl=4h \
  token_bound_cidrs="10.0.0.0/16"
  # token_bound_cidrs restricts token usage to the cluster CIDR —
  # a stolen token is unusable from outside the network

5. Detection: Alert on Root Token Regeneration

The most dangerous post-breach Vault operation is root token regeneration. Vault allows generating a new root token from recovery keys — a legitimate break-glass operation. It should never happen in normal operations. Alert on any occurrence:

# Loki LogQL query — alert if this matches any log entry
{job="vault-audit"}
| json
| request_path =~ ".*generate-root.*"
| line_format "CRITICAL: Root token generation initiated. Accessor: {{.auth.accessor}} Client IP: {{.request.remote_address}} Time: {{.time}}"

Equivalent Elasticsearch query for teams shipping audit logs to an ELK stack:

{
  "query": {
    "bool": {
      "should": [
        { "match": { "request.path": "sys/generate-root" }},
        { "match": { "request.path": "sys/generate-root/attempt" }}
      ]
    }
  }
}

Also alert on audit log disable operations — if any sys/audit DELETE request appears in the audit log, an operator-level account attempted to remove the audit trail:

{job="vault-audit"}
| json
| request_operation = "delete"
| request_path =~ ".*sys/audit.*"
| line_format "CRITICAL: Audit log disable attempt. Path: {{.request.path}} Accessor: {{.auth.accessor}}"

Note that this detection depends on audit logging being enabled and configured correctly. If audit logging was never configured, this alert will never fire — there is nothing to alert on. The audit log must be the first thing enabled during bootstrap.

Expected Behaviour After Hardening

After auto-unseal configuration, a Vault pod restart produces no human intervention and no unseal prompts. The startup sequence in Vault logs:

[INFO]  core: stored unseal keys supported, attempting fetch
[INFO]  seal.awskms: decrypting seal key
[INFO]  core: vault is unsealed
[INFO]  core: acquired lock, enabling active operation

After root token revocation:

vault token lookup hvs.CAESIHJ5...
# Error looking up token: Error making API request.
# Code: 403. Errors:
# * permission denied

After audit log configuration:

vault audit list
# Path     Type    Description
# ----     ----    -----------
# file/    file
# syslog/  syslog

A secret read from an application produces an audit log entry in /var/log/vault/audit.log:

{
  "time": "2026-05-09T14:23:01.445Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:a3f4...",
    "accessor": "hmac-sha256:b9c1...",
    "display_name": "kubernetes-production-payments-service",
    "policies": ["default", "app-payments-service"],
    "token_type": "service"
  },
  "request": {
    "id": "abc-123",
    "operation": "read",
    "path": "secret/data/production/databases/payments-postgres",
    "remote_address": "10.0.14.22"
  },
  "response": {
    "data": {
      "metadata": {
        "created_time": "2026-04-01T00:00:00Z",
        "version": 3
      }
    }
  }
}

The secret value is not present in the log — log_raw=false ensures only metadata is recorded. The token itself is HMAC’d — it cannot be extracted from the audit log. The accessor allows correlating which token made the request without exposing the token value.

Trade-offs

Auto-unseal creates a KMS dependency. If AWS KMS is unavailable — regional outage, KMS endpoint throttling, VPC endpoint misconfiguration — Vault cannot unseal on restart. A multi-region KMS key (prefixed mrk-) replicates key material across regions; Vault can be configured with a replica in a secondary region. The trade-off is: manual unseal keys in a password manager can be retrieved even when AWS is down, but they require the humans who hold them to be available and willing. KMS dependency is a hard infrastructure dependency, but it eliminates the human reliability problem.

Root token revocation is permanent without recovery keys. Losing both the root token and all recovery keys means Vault is unrecoverable — every secret inside it is permanently inaccessible. Recovery keys must be distributed across multiple people and secure locations before revoking the root token. The bootstrap script above stores them in SSM; in practice, they should be printed, placed in sealed envelopes, and stored in physically separate locations with separate custodians. The recovery key procedure should be tested in a non-production environment before production bootstrapping.

File audit logs consume disk. A busy Vault instance receiving thousands of secret reads per minute can produce substantial audit log volume. Without rotation and offloading, the disk fills and Vault blocks — by design. Vault will refuse to serve requests rather than silently drop audit log entries. Log rotation with immediate shipping to S3 is not optional; it is part of the operational contract for audit log configuration.

ACL policy scope requires ongoing maintenance. As applications add new secret paths, their policies must be updated. A common response to this friction is widening the policy to cover broader paths. This progressively erodes least-privilege guarantees. Policy changes should go through the same review process as infrastructure changes — a pull request with a reviewer who understands the implication of each additional capabilities entry.

Failure Modes

Auto-unseal configured, recovery keys not stored. If the KMS key is accidentally deleted, the KMS key policy is misconfigured to deny Vault’s IAM role, or the AWS account is suspended, Vault cannot unseal. Without recovery keys, there is no mechanism to recover the data. Every secret in Vault is permanently lost. Mitigation: verify the recovery key storage procedure works before bootstrapping production. Run vault operator generate-root in a test environment with recovery keys to confirm the break-glass path is operational.

Audit log configured on Vault node, not shipped externally. An attacker who gains root access to the Vault EC2 instance or Kubernetes node can delete /var/log/vault/audit.log before investigators arrive. The syslog output, if not forwarded to a remote syslog server, faces the same problem. The audit log’s forensic value depends entirely on it existing somewhere the attacker cannot reach. If the Fluent Bit shipper fails silently and the local log fills the disk, Vault blocks all requests — a self-inflicted denial of service that draws immediate attention but is operationally disruptive.

Root token stored as a CI/CD environment variable “for emergencies.” This pattern appears in every post-mortem where a Vault instance is compromised through a CI system. The correct break-glass procedure is: use Vault operator generate-root with recovery keys to create a time-limited root token, perform the emergency operation, then immediately revoke it. Storing a persistent root token anywhere outside of the initial bootstrap moment is not an emergency procedure — it is a standing compromise waiting to be discovered.

No alerting on root token regeneration. The sys/generate-root endpoint is the most sensitive operation Vault exposes. A sophisticated attacker who gains access to enough recovery keys (3 of 5, in a standard configuration) can generate a new root token and proceed with full, unauditable access. If this operation does not trigger an immediate page, the attacker has an open window to read secrets, disable audit logging, and erase traces. The alert must be configured as part of the initial deployment, not added later when the monitoring stack is “more mature.” There is no “more mature” — there is only the next incident.