Network & API Security advanced 13 min read

Service Mesh mTLS Identity: Istio and Linkerd Certificate Security Deep Dive

service-mesh mtls istio linkerd certificate-management zero-trust

Service Mesh mTLS Identity: Istio and Linkerd Certificate Security Deep Dive

The Problem

Pod-to-pod traffic in Kubernetes is unauthenticated by default. A compromised workload can connect to any other service on the pod network, present no credentials, and receive a response. NetworkPolicy can limit which pod IPs can reach which ports, but it cannot establish cryptographic identity — the receiving service has no way to verify that the sender is who it claims to be.

Mutual TLS solves this. Both sides of every connection present a certificate, and both sides verify the other’s certificate against a shared trust root before exchanging any application data. In a service mesh, the sidecar proxy handles the mTLS handshake transparently without application changes. The security properties of this arrangement depend entirely on the certificate infrastructure beneath it: an exposed root CA key compromises all workload identities; permissive peer authentication lets plaintext connections bypass the identity layer; iptables bypass lets attackers reach the application directly. This article covers the certificate hierarchy, rotation mechanics, external CA integration, and bypass detection.

How the Proxy Intercepts Traffic Without App Changes

Both Istio and Linkerd inject a sidecar proxy into each pod at admission time. The injection webhook patches the pod spec to add the proxy container and an initContainer that modifies the pod’s iptables rules before the application starts.

The initContainer runs with NET_ADMIN and installs rules in the ISTIO_REDIRECT or PROXY_INIT chain. Inbound TCP destined for application ports is redirected to the proxy’s inbound port (15006 in Istio, 4143 in Linkerd). Outbound TCP is redirected to the proxy’s outbound port (15001 in Istio, 4140 in Linkerd). Traffic from the proxy’s own UID is excluded from redirection to avoid loops.

The application never sees a direct connection from another pod. Every inbound connection arrives from the local proxy after mTLS termination and peer certificate verification. The calling service’s identity is available in the Envoy access log and Istio telemetry — not injected as a header by default. Headers can be forged; mTLS certificate identity cannot.

Istio Certificate Hierarchy: Root CA to SPIFFE SVID

Istio’s certificate architecture has three tiers.

Tier 1: Root CA. In a default Istio install, istiod generates a self-signed root CA on first startup and stores the private key in the istio-ca-secret Secret in the istio-system namespace. This is the trust anchor for the entire mesh. Any workload certificate that chains to this root will be accepted as a valid mesh identity.

Tier 2: Intermediate CA (istiod). Istiod acts as a signing CA for workload certificates. In deployments using Istio’s own CA, istiod signs workload certificates directly from the root key (no intermediate in the default path). In production deployments this should be replaced: istiod should hold only an intermediate CA certificate signed by an external root, so the root key never exists in the cluster.

Tier 3: Workload certificates (SPIFFE SVIDs). Each sidecar proxy receives a certificate whose Subject Alternative Name encodes the pod’s SPIFFE identity: spiffe://<trust-domain>/ns/<namespace>/sa/<service-account>. This is a SPIFFE Verifiable Identity Document (SVID) — a standard format that allows policy to be written against workload identity rather than IP addresses or hostnames. See SPIFFE/SPIRE Workload Identity for the broader SPIFFE architecture.

The issuance flow: the Envoy sidecar connects to istiod’s SDS endpoint, generates a private key locally, and sends a CSR with the pod’s service account token as proof of identity. Istiod validates the token against the Kubernetes API, signs the CSR, and returns the signed certificate plus the CA bundle. The certificate is stored in memory only — never written to disk.

The default TTL is 24 hours. The proxy requests renewal at roughly half the TTL elapsed — every 12 hours under normal operation. Rotation is graceful: the proxy holds both old and new certificates during a transition window, completing existing connections on the old cert while establishing new ones with the new cert. No connection drops occur.

Istio External CA Integration

Leaving the root CA private key inside the cluster is a significant risk. If an attacker gains access to the istio-system namespace and can read Secrets, they can issue valid certificates for any service account in the mesh.

cert-manager as Istio CA

cert-manager’s istio-csr component implements the Istio certificate signing API (IstioCertificateService), intercepting istiod’s certificate requests and routing them to a cert-manager Issuer or ClusterIssuer. The intermediate CA certificate used by istiod is issued by cert-manager, which can in turn be backed by Vault, AWS PCA, or any other external PKI.

Install istio-csr and configure it to use a Vault-backed ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.internal:8200
    path: pki_int/sign/istio-role
    auth:
      kubernetes:
        role: cert-manager
        mountPath: /v1/auth/kubernetes
        secretRef:
          name: cert-manager-vault-token
          key: token

The istio-csr Helm values point to this issuer:

app:
  certmanager:
    issuerRef:
      name: vault-issuer
      kind: ClusterIssuer
      group: cert-manager.io
  istio:
    namespace: istio-system
    revisionLabel: default

With this configuration, istiod holds a short-lived intermediate CA certificate issued by Vault. The root CA key lives in Vault (or AWS PCA behind Vault), not in the cluster. Rotating the root requires no cluster downtime — Vault issues a new intermediate, cert-manager delivers it to istio-csr, and istiod picks it up.

Certificate Chain Depth

Regardless of the external CA approach, Envoy validates the full certificate chain. The chain presented by each workload must be complete — leaf → intermediate → root — with the root matching the trust bundle distributed to all proxies via Istio’s cacerts Secret. Vault’s pki_int mount must be configured with the signed intermediate (pki_int/intermediate/set-signed), and the root CA certificate must be included in the bundle so all proxies trust the full chain. A missing intermediate in the chain causes TLS handshake failures that are silent from the application’s perspective but visible in Envoy’s access log as HANDSHAKE_FAILED.

Linkerd Certificate Hierarchy

Linkerd uses a three-tier hierarchy with explicit naming:

Trust anchor — the root CA certificate. Its private key must be kept offline or in an HSM. The trust anchor certificate is configured during linkerd install and distributed to all proxies as the trust root. Compromising the trust anchor key allows issuing valid issuer certs, which in turn sign workload certificates. The blast radius is total.

Issuer certificate — an intermediate CA certificate signed by the trust anchor. The issuer’s private key lives in the cluster (in the linkerd-identity-issuer Secret in the linkerd namespace), and the linkerd-identity control-plane component uses it to sign per-workload leaf certificates. The issuer certificate has a configurable TTL; Buoyant recommends 48 hours for automated rotation setups.

Leaf certificates — short-lived certificates issued per proxy. Default TTL is 24 hours. The proxy requests renewal before expiry. Like Istio, the leaf cert Subject Alternative Name is a SPIFFE URI: spiffe://<cluster-identity>/ns/<namespace>/sa/<service-account>.

Using step-ca as Linkerd’s Issuer

step-ca from Smallstep acts as Linkerd’s online intermediate CA while the trust anchor stays offline. Generate the trust anchor offline with step certificate create root.linkerd.cluster.local ca.crt ca.key --profile root-ca --not-after 87600h, then generate the issuer cert signed by the trust anchor with --profile intermediate-ca --not-after 48h. Configure cert-manager to renew the issuer automatically:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-identity-issuer
  namespace: linkerd
spec:
  secretName: linkerd-identity-issuer
  duration: 48h
  renewBefore: 25h
  issuerRef:
    name: step-ca-issuer
    kind: ClusterIssuer
  commonName: identity.linkerd.cluster.local
  isCA: true
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - identity.linkerd.cluster.local

The linkerd-identity controller watches the Secret and picks up the renewed issuer cert without restart. Leaf certificates rotate on the 24-hour cycle using the updated issuer key.

Enforcing mTLS: PeerAuthentication STRICT Mode

Istio’s PeerAuthentication resource controls whether mTLS is required for inbound traffic to a set of pods. The two modes that matter operationally are STRICT and PERMISSIVE.

PERMISSIVE allows both mTLS and plaintext connections. It is the default during migration: existing services that have not yet been meshed can still reach meshed services. From a security standpoint, permissive mode removes the identity guarantee entirely — a plaintext connection from an unmeshed pod (or from a compromised pod that bypasses the sidecar) is accepted without any certificate validation.

STRICT requires mTLS for all inbound traffic. Connections without a valid client certificate are rejected at the Envoy inbound listener. Apply STRICT mode cluster-wide with a root-namespace policy:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

This policy applies to all pods in all namespaces because it is in the Istio root namespace (istio-system). Namespace-scoped or workload-scoped PeerAuthentication resources can override this for migration purposes, but any override to PERMISSIVE should be treated as a temporary exception with a defined expiry.

For migration, apply STRICT per namespace after verifying that all pods in that namespace are meshed:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: namespace-strict
  namespace: payments
spec:
  mtls:
    mode: STRICT

Verify all pods in the namespace have sidecars before applying STRICT — any pod not showing istio-proxy in its container list will fail to connect immediately:

kubectl get pods -n payments -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}' | grep -v istio-proxy

Identity-Based AuthorizationPolicy

PeerAuthentication STRICT mode establishes that all connections are mTLS authenticated. AuthorizationPolicy determines which authenticated identities are allowed to reach which services. Policy is written against SPIFFE identity, not IP address — a compromised pod presenting the correct service account certificate is still bound by that identity’s policy, and a pod running under the wrong service account presents a different SPIFFE SAN and is denied.

Allow only the checkout service account in the frontend namespace to reach the payments service:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payments-allow-checkout
  namespace: payments
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/frontend/sa/checkout"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/charge", "/v1/refund"]

Add a default-deny in the same namespace — an empty AuthorizationPolicy with no action field acts as a deny-all baseline; all ALLOW rules are additive on top of it:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: payments
spec:
  {}

This implements least-privilege for east-west traffic. For the broader zero-trust pattern, see Zero Trust Architecture Principles and Microsegmentation with Cilium.

Detecting mTLS Bypass Attempts

The iptables redirect rules installed by the initContainer are the foundation of transparent proxy interception. If those rules can be bypassed, an attacker can communicate directly with the application port on a pod without going through Envoy — meaning no mTLS, no certificate validation, no policy enforcement.

Three bypass vectors exist:

1. Privileged containers with NET_ADMIN. A container that can modify iptables can flush the redirect rules and communicate directly with the pod network. Any pod running with NET_ADMIN capability and not running the mesh proxy is a bypass candidate. Audit for this:

kubectl get pods -A -o json | jq -r '
  .items[] |
  select(.spec.containers[].securityContext.capabilities.add[]? == "NET_ADMIN") |
  [.metadata.namespace, .metadata.name] | @tsv
'

2. Host-network pods. Pods running with hostNetwork: true bypass the pod-level iptables namespace entirely. Their traffic originates from the node’s network namespace, and the mesh intercept rules do not apply. These pods appear in Envoy access logs as connections from the node IP without a peer certificate.

3. Direct pod IP connections. An attacker with network access to pod IPs — from a compromised node or via a host-network pod — can connect to the application’s listening port directly, bypassing Envoy. The application itself does not require client certificates in most mesh deployments; the proxy is the only enforcement point.

Detection via Envoy access logs: when an inbound connection arrives without a client certificate, Envoy logs %DOWNSTREAM_PEER_CERT% as empty, and %DOWNSTREAM_MUTUAL_TLS% as false. In STRICT mode, Envoy rejects the connection before it reaches the application, so the application never receives it. In PERMISSIVE mode, the connection proceeds and the access log is the only evidence of the unauthenticated connection.

Enable Envoy access logging with peer certificate fields using an Istio Telemetry resource in istio-system that adds %DOWNSTREAM_MUTUAL_TLS% and %DOWNSTREAM_PEER_CERT_V_END% to the access log format. A DOWNSTREAM_MUTUAL_TLS value of false on an inbound connection where STRICT mode is enforced indicates a bypass attempt or sidecar injection failure. Alert on this pattern. For detecting iptables tampering, audit the pod’s network namespace with nsenter -t <pid> -n iptables -t nat -L ISTIO_REDIRECT — an empty or missing ISTIO_REDIRECT chain on a meshed pod means the redirect rules have been flushed.

Debugging Certificate Issues

Istio: istioctl proxy-config secret

The istioctl proxy-config secret command shows the certificates currently held by a pod’s Envoy proxy:

istioctl proxy-config secret <pod-name>.<namespace>

Output includes the certificate serial, expiry, and whether the root CA bundle is loaded. To see the full certificate chain:

istioctl proxy-config secret <pod-name>.<namespace> -o json | \
  jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | \
  base64 -d | openssl x509 -noout -text

Check the SAN field to verify the SPIFFE URI matches the expected service account:

X509v3 Subject Alternative Name:
    URI:spiffe://cluster.local/ns/payments/sa/payments-backend

A mismatch here (wrong namespace or service account in the SAN) means the pod is running under a different service account than expected, which will cause AuthorizationPolicy failures.

If the certificate is expired or Envoy cannot reach istiod to renew it, connections fail with TLS handshake errors. Verify istiod connectivity from the pod’s proxy container using curl -sk https://istiod.istio-system:15012/debug/ndsz.

Linkerd: linkerd check --proxy

linkerd check --proxy --namespace payments

This verifies that each meshed pod’s certificate is valid, not expired, and issued by the current trust anchor. linkerd check --pre validates the full trust anchor → issuer → leaf chain before a Linkerd upgrade — useful for catching rotation issues before they cause an outage. For live traffic inspection, linkerd viz tap deploy/payments -n payments --to deploy/checkout -n frontend | grep tls shows tls=true for mTLS-protected connections and tls=false for plaintext; the latter between two meshed workloads indicates a proxy interception failure.

Trust Hierarchy Hardening Checklist

  • Root CA private key is never stored in the cluster. Use Vault PKI, AWS PCA, or an offline HSM-backed CA.
  • Workload certificate TTL is 24 hours or shorter. Shorter TTLs reduce the window of a stolen certificate.
  • Issuer certificate TTL is managed by cert-manager with automated renewal. Manual issuer rotation is error-prone and frequently missed.
  • STRICT mode PeerAuthentication is applied at the root namespace (istio-system) or via the Linkerd equivalent.
  • Default-deny AuthorizationPolicy exists in every namespace. ALLOW rules are additive and explicitly reference SPIFFE principals.
  • NET_ADMIN capability is not granted to any application container. Audit this in CI via OPA/Gatekeeper or Kyverno.
  • hostNetwork: true pods are documented, minimized, and excluded from STRICT mode coverage maps with compensating controls.
  • Envoy access logs include DOWNSTREAM_MUTUAL_TLS and trigger alerts for any false value on inbound connections.
  • Certificate expiry is monitored independently of the mesh (Prometheus x509_cert_expiry metric from the x509-certificate-exporter).
  • The trust anchor certificate expiry is calendared manually. Linkerd and Istio do not alert on trust anchor expiry proactively; discovering it at expiry is a full mesh outage.