Network & API Security advanced 13 min read

Microsegmentation with Cilium: L7-Aware Network Policy for Zero Trust Kubernetes

cilium microsegmentation zero-trust ebpf network-policy kubernetes

Microsegmentation with Cilium: L7-Aware Network Policy for Zero Trust Kubernetes

Why L3/L4 NetworkPolicy Is Not Enough

Standard Kubernetes NetworkPolicy controls which pods can talk to which other pods on which TCP or UDP ports. That is the entirety of its expressive power. It cannot distinguish between an HTTP GET and an HTTP DELETE on the same port. It cannot restrict a client to a specific URL path. It cannot say “this pod may resolve api.stripe.com but not api.pastebin.com.” It cannot authenticate the identity of the caller — it trusts IP addresses, which are ephemeral in Kubernetes and can be spoofed by any workload that controls its own network namespace.

The gap becomes concrete with a typical internal API. A payment-processor pod and an audit-logger pod both legitimately call account-service:8080. Standard NetworkPolicy allows both callers on port 8080 — that is all it can do. audit-logger should only call GET /api/v1/accounts/{id} to read account metadata. If it calls DELETE /api/v1/accounts/{id} or POST /api/v1/accounts/{id}/withdraw, those requests reach account-service because NetworkPolicy passed them at L4. The application may or may not enforce its own RBAC. If it does not, or if the workload is compromised and bypasses its own client code, the policy boundary does not exist.

The same problem appears at egress. A workload should be able to reach api.github.com but not arbitrary internet hosts. A port-based egress rule allowing outbound 443 permits connections to every HTTPS destination in existence. DNS-based filtering — limiting which FQDNs a pod may resolve — requires DNS interception that NetworkPolicy does not provide.

Zero trust architecture assumes every workload is untrusted until identity and intent are verified at each request. L3/L4 policy verifies neither. It verifies IP reachability. That is a necessary control but not a sufficient one in a threat model where lateral movement within a cluster is the primary risk.

Cilium Architecture

Cilium is a CNI plugin built around eBPF — extended Berkeley Packet Filter — programs that run in the Linux kernel. When a packet arrives at a network interface inside a pod or node, Cilium’s eBPF programs intercept it before it reaches iptables or the userspace network stack. Policy decisions happen in the kernel data path, which gives Cilium several properties that matter for security:

  • No iptables dependency. Cilium can operate in fully eBPF-native mode (bpf-masquerade, bpf-host-routing) with no iptables rules at all. This eliminates an entire class of iptables ordering bugs and race conditions that have historically produced policy bypass vulnerabilities.
  • Identity-based enforcement. Cilium assigns each pod a numeric identity derived from its Kubernetes labels. Policy rules reference identities, not IPs. When a pod restarts with a new IP, its identity is stable and policy continues to apply correctly.
  • L7 via Envoy. When a CiliumNetworkPolicy contains L7 rules (HTTP, gRPC, Kafka), Cilium transparently redirects matching traffic to an Envoy proxy running on the node. Envoy evaluates the L7 rule, enforces it, and forwards or drops the request. There are no sidecars — one Envoy process runs per node, managed by the Cilium agent.

The Cilium agent (cilium-agent) runs as a DaemonSet. It watches Kubernetes API resources — CiliumNetworkPolicy, CiliumClusterwideNetworkPolicy, CiliumEndpoint, endpoints — and translates them into eBPF maps and programs loaded into the kernel. The agent also manages the per-node Envoy process, feeding it xDS configuration derived from L7 policy rules.

Hubble is Cilium’s observability layer. The Hubble server component runs inside cilium-agent and exports flow records from the eBPF data path: every connection attempt, its policy verdict (ALLOWED or DROPPED), and for L7 traffic, the HTTP method, path, response code. Hubble relay aggregates records from all nodes and exposes them via a gRPC API. Hubble UI and the hubble CLI consume that API.

CiliumNetworkPolicy: L7 HTTP Rules

CiliumNetworkPolicy is a Cilium-specific CRD that extends Kubernetes NetworkPolicy with L7 matching. HTTP rules live under toPorts[].rules.http.

A minimal policy allowing payment-processor to call only GET /api/v1/accounts on account-service:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: account-service-ingress
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app: account-service
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: payment-processor
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: /api/v1/accounts
              - method: GET
                path: /api/v1/accounts/[0-9a-f-]+

Path values are regular expressions. The second rule matches UUIDs. Any other method or path from payment-processor to account-service:8080 is dropped by Envoy with an HTTP 403, logged as a policy violation in Hubble, and never reaches the application.

You can also match on headers. Restricting access to a specific internal service account token header:

rules:
  http:
    - method: POST
      path: /internal/v1/transfer
      headers:
        - X-Internal-Service: payment-processor

Header matching is case-insensitive. Requests missing X-Internal-Service: payment-processor are dropped regardless of method and path match.

For gRPC, the rules.http block matches on the gRPC framing that maps to HTTP/2:

rules:
  http:
    - method: POST
      path: /api.AccountService/GetAccount
    - method: POST
      path: /api.AccountService/ListAccounts

gRPC uses POST for all methods, and the path is the fully qualified service and method name. Any call to DeleteAccount or CreateWithdrawal from a pod that only has these rules applied is blocked.

DNS-Based Egress with toFQDNs

toFQDNs controls which DNS names a pod’s egress traffic may reach. Cilium implements this by intercepting DNS responses at the pod: when a pod resolves a permitted FQDN, Cilium captures the returned IP addresses and dynamically updates its eBPF maps to allow TCP connections to those IPs. When the DNS TTL expires and the IPs change, the maps are updated automatically.

Restricting a pod to specific external APIs:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payment-processor-egress
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app: payment-processor
  egress:
    - toFQDNs:
        - matchName: api.stripe.com
        - matchName: api.braintreegateway.com
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP

The toEndpoints rule for kube-dns is required: toFQDNs rules need DNS resolution to work, and Cilium enforces egress to DNS as a separate rule. Without the DNS egress rule, the pod cannot resolve any FQDNs.

matchPattern supports glob syntax for wildcard matching:

toFQDNs:
  - matchPattern: "*.stripe.com"
  - matchPattern: "*.amazonaws.com"

matchPattern wildcards match only one label component — *.stripe.com matches api.stripe.com but not api.internal.stripe.com. Use multiple patterns for multi-level subdomains.

For a default-deny egress posture, the policy must explicitly allow any internal Kubernetes service DNS names as well. All cluster.local resolution must be permitted or internal service discovery breaks:

- toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s:app: kube-dns
  toPorts:
    - ports:
        - port: "53"
          protocol: UDP
- toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: payments
  toPorts:
    - ports:
        - port: "8080"
          protocol: TCP

Mutual Auth with SPIFFE Identity

Cilium integrates with SPIFFE/SPIRE workload identity to provide cryptographic identity verification as a policy primitive. Standard CiliumNetworkPolicy relies on Kubernetes label selectors — any workload that can set matching labels could impersonate a legitimate peer. SPIFFE-based mutual authentication adds a cryptographic assertion: the caller must present a valid SPIFFE SVID (an X.509 certificate with a SPIFFE URI SAN) matching the policy.

Cilium’s mutual authentication uses SPIRE to issue SVIDs to each endpoint and performs mTLS verification in the eBPF data path using kernel TLS. The policy references SPIFFE identities in the authentication block:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: account-service-mtls-ingress
  namespace: payments
spec:
  endpointSelector:
    matchLabels:
      app: account-service
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: payment-processor
      authentication:
        mode: required
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: /api/v1/accounts/[0-9a-f-]+

With authentication.mode: required, Cilium enforces that the connection carries a valid SPIFFE SVID before evaluating L7 rules. A connection without a valid SVID — even from a pod with matching labels — is dropped at the eBPF level before reaching Envoy.

This closes the impersonation gap: a compromised pod that spoofs labels still cannot satisfy the SPIFFE challenge without a valid cryptographic credential issued by SPIRE. The combination of identity-based L7 policy with SPIFFE mutual authentication is the minimal implementation of identity-aware proxy security without a separate proxy tier.

The SPIRE agent and SPIRE server must be deployed and the Cilium-SPIRE integration enabled via Helm values:

authentication:
  mutual:
    spire:
      enabled: true
      install:
        enabled: false
      adminSocketPath: /run/spire/sockets/admin.sock
      agentSocketPath: /run/spire/sockets/agent.sock

Setting install.enabled: false assumes SPIRE is already deployed in the cluster. Set to true for Cilium to deploy a minimal SPIRE installation automatically.

Hubble: Flow Observability and Policy Verdicts

Hubble records every network flow processed by Cilium and annotates it with policy verdicts. This is the operational feedback loop that makes L7 microsegmentation maintainable: without it, you cannot distinguish “policy is working” from “policy has no matching traffic.”

Install the Hubble CLI and connect to the relay:

cilium hubble port-forward &
hubble observe --namespace payments --last 100

Filter to dropped flows only, which surfaces policy violations:

hubble observe \
  --namespace payments \
  --verdict DROPPED \
  --last 500 \
  --output json | jq '{
    time: .time,
    src: .source.pod_name,
    dst: .destination.pod_name,
    method: .l7.http.method,
    url: .l7.http.url,
    verdict: .verdict
  }'

Watch live policy violations for a specific destination:

hubble observe \
  --to-pod payments/account-service \
  --verdict DROPPED \
  --follow

The output for an L7 HTTP drop includes the method, path, and response code that Envoy returned:

payments/payment-processor → payments/account-service:8080
  HTTP DELETE /api/v1/accounts/abc123 → 403 Forbidden
  Policy verdict: DROPPED (L7 rule)

Hubble also exposes flow data as Prometheus metrics via hubble-metrics. The metric hubble_drop_total with labels direction, reason, and protocol is the primary signal for policy violation rate alerting:

- alert: CiliumL7PolicyDropRateHigh
  expr: |
    rate(hubble_drop_total{reason="POLICY_DENIED"}[5m]) > 0.1
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "High L7 policy drop rate in {{ $labels.namespace }}"

Hubble UI (hubble-ui) provides a service map overlay showing allowed and dropped flows between services in a namespace, rendered as a directed graph with policy verdict colour coding. It is useful for initial policy development but not a substitute for metric-based alerting in production.

Default-Deny Posture with CiliumClusterwideNetworkPolicy

CiliumClusterwideNetworkPolicy applies policy cluster-wide without a namespace scope. A deny-all baseline at the cluster level forces every team to write explicit allow rules before their workloads can communicate:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: default-deny-all
spec:
  endpointSelector: {}
  ingress:
    - fromEntities:
        - host
  egress:
    - toEntities:
        - host
        - kube-apiserver

endpointSelector: {} matches every endpoint in the cluster. The ingress and egress rules permit communication with the host (required for node-local traffic including health checks and Kubelet probes) and kube-apiserver (required for workloads that call the Kubernetes API). All other ingress and egress is denied by default.

With this in place, deploying a new workload without a corresponding CiliumNetworkPolicy means it receives no traffic and sends no traffic to anything other than the node and the API server. The default-deny is not advisory — it is enforced at the kernel level.

Teams then add namespace-scoped CiliumNetworkPolicy resources for their workloads. A namespace policy allowing intra-namespace communication while maintaining external isolation:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-intra-namespace
  namespace: payments
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: payments
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: payments

This stacks with the cluster-wide deny-all: traffic from payments pods to payments pods is allowed by the namespace policy, while traffic crossing namespace boundaries is still denied unless an explicit policy exists for it.

Verify the effective policy for a specific endpoint:

kubectl exec -n kube-system ds/cilium -- \
  cilium endpoint list | grep payment-processor

kubectl exec -n kube-system ds/cilium -- \
  cilium endpoint get <ENDPOINT_ID> | jq '.[].status.policy'

Migrating from Calico to Cilium Without Downtime

The standard approach for migrating from Calico is to run both CNIs in parallel using a migration CNI (typically cilium-migration or the upstream multus-based approach), then shift traffic workload by workload.

Cilium’s recommended production migration path uses the --set operator.unmanagedPodWatcher.restart=false Helm value and node-by-node migration:

Step 1: Install Cilium in parallel CNI mode. Add Cilium to the cluster without replacing Calico. On EKS and GKE this means installing Cilium with CNI chaining disabled and waiting for the operator:

helm install cilium cilium/cilium \
  --version 1.15.6 \
  --namespace kube-system \
  --set cni.exclusive=false \
  --set cni.chainingMode=none \
  --set operator.replicas=1 \
  --set ipam.mode=kubernetes

Step 2: Cordon and drain nodes one at a time. For each node:

kubectl cordon NODE_NAME
kubectl drain NODE_NAME \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --pod-selector 'k8s-app!=calico-node'

After draining, remove the Calico CNI plugin binary from the node and restart the Kubelet. New pods scheduled on the node will use Cilium. Validate with:

kubectl exec -n kube-system ds/cilium -- \
  cilium node list | grep NODE_NAME

Confirm Cilium endpoints are healthy:

kubectl exec -n kube-system ds/cilium -- \
  cilium endpoint list | grep -v 'ready'

Any endpoint not in ready state indicates a policy or IP allocation problem. Resolve before draining the next node.

Step 3: Translate Calico NetworkPolicy to CiliumNetworkPolicy. Standard Kubernetes NetworkPolicy objects are understood by Cilium natively — Cilium installs a NetworkPolicy controller that reads them. You do not need to convert standard NetworkPolicy resources immediately. Calico-specific GlobalNetworkPolicy and NetworkPolicy CRDs (in the projectcalico.org/v3 API group) have no automatic translation. Audit these manually:

kubectl get globalnetworkpolicies.projectcalico.org -o yaml
kubectl get networkpolicies.projectcalico.org -A -o yaml

Map each Calico-specific rule to an equivalent CiliumNetworkPolicy or CiliumClusterwideNetworkPolicy. Calico GlobalNetworkPolicy with selector: all() maps to CiliumClusterwideNetworkPolicy with endpointSelector: {}.

Step 4: Validate policy parity with Hubble. Before removing Calico entirely, run both CNIs on separate nodes and use Hubble to confirm that the expected traffic is flowing and no unexpected drops are occurring on Cilium-managed nodes:

hubble observe \
  --verdict DROPPED \
  --namespace production \
  --last 1000 | grep -v 'health\|kube-system'

Any drops that were not present under Calico indicate a missing CiliumNetworkPolicy rule.

Step 5: Remove Calico. Once all nodes are migrated and Hubble shows no unexpected drops:

helm uninstall calico -n kube-system
kubectl delete crd \
  felixconfigurations.crd.projectcalico.org \
  bgpconfigurations.crd.projectcalico.org \
  ippools.crd.projectcalico.org \
  hostendpoints.crd.projectcalico.org \
  globalnetworkpolicies.projectcalico.org \
  globalnetworksets.crd.projectcalico.org \
  networkpolicies.crd.projectcalico.org \
  clusterinformations.crd.projectcalico.org

Verify no iptables rules from cali-* chains remain:

iptables-save | grep cali

Any remaining Calico chains indicate incomplete cleanup. Remove them with iptables -F CALI-* per chain or reboot the node.

Operational Checklist

After deploying Cilium with L7 microsegmentation:

  • Confirm cilium status --verbose shows Endpoint health monitoring: OK and no degraded endpoints.
  • Verify bpf-host-routing is enabled if you need L7 policy enforcement on same-node traffic: cilium config get bpf-host-routing.
  • Run hubble observe --verdict DROPPED --follow for 30 minutes after initial rollout and resolve all unexpected drops before declaring the migration complete.
  • Alert on hubble_drop_total{reason="POLICY_DENIED"} with a low threshold for the first week to catch policy gaps.
  • For SPIFFE integration, verify the SPIRE agent is healthy on every node: kubectl logs -n spire ds/spire-agent | grep -i error.
  • Test L7 rule enforcement explicitly: send an HTTP DELETE from a pod that only has GET permission and confirm it is blocked with a 403 and appears in Hubble as DROPPED.

L7-aware network policy is not a replacement for application-layer authorization, but it eliminates an entire class of lateral movement where a compromised workload calls APIs it should never reach. Combined with SPIFFE workload identity and a cluster-wide default-deny posture, it is the most effective single control for containing east-west blast radius in a Kubernetes cluster.