CI/CD & Supply Chain intermediate 11 min read

Golden Path Security: Building Security In from Day Zero with Paved Road Templates

platform-engineering golden-path paved-road security-by-default shift-left kubernetes

Golden Path Security: Building Security In from Day Zero with Paved Road Templates

What “Golden Path” Actually Means for Security

Platform engineering teams talk about golden paths as productivity tools: one scaffold command, and a developer has a runnable service with CI/CD, observability, and a Helm chart. That framing leaves security as a separate concern. The security perspective is different: the golden path is the only realistic point at which you can make secure defaults cheaper than insecure ones.

Every hardening control you add after a service ships has a cost: developer attention, regression risk, migration effort, exception handling. The same control embedded in a template has near-zero marginal cost per service. A Semgrep scan configured in the scaffold-generated CI workflow costs nothing to the 200th team that uses the template.

The goal is not to prevent developers from doing insecure things — it is to make the secure thing the path of least resistance. If the template generates a Helm chart with securityContext.runAsNonRoot: true and a minimal distroless base image, developers running the golden path get those controls by default. Deviating from them requires deliberate effort and triggers policy enforcement.

This also defines where the responsibility boundary sits. The platform team owns the golden path security controls. Individual teams own the application-layer decisions (business logic, data validation, authentication design). The division is clean and auditable.

For broader context on how this fits into an internal developer platform security strategy, particularly around RBAC, namespace isolation, and secrets management within the platform itself, read that companion article first.

Designing Secure Service Templates in Backstage

Backstage’s scaffolder is the standard mechanism for implementing golden paths. A scaffolder template is a YAML-defined wizard that collects parameters from the developer and uses them to render files from a template directory, register the service in the catalog, and trigger optional actions (create repo, provision namespace, configure CI secrets).

Security controls belong in the template rendering layer, not in documentation.

A minimal secure Backstage scaffolder template for a Kubernetes service looks like this:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: secure-go-service
  title: Secure Go Microservice
  description: Go service with security controls pre-configured
  tags:
    - go
    - kubernetes
    - golden-path
spec:
  owner: platform-team
  type: service
  parameters:
    - title: Service Details
      required:
        - name
        - system
      properties:
        name:
          title: Service Name
          type: string
          pattern: '^[a-z][a-z0-9-]{2,62}$'
          description: Lowercase, alphanumeric, hyphens only
        system:
          title: Owning System
          type: string
          ui:field: EntityPicker
          ui:options:
            catalogFilter:
              kind: System
        imageRegistry:
          title: Container Registry
          type: string
          default: registry.internal.example.com
          enum:
            - registry.internal.example.com
          description: Only approved internal registries

  steps:
    - id: fetch-template
      name: Fetch Template
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          system: ${{ parameters.system }}
          imageRegistry: ${{ parameters.imageRegistry }}
          sbomEnabled: true
          slsaEnabled: true
          networkPolicyEnabled: true

    - id: publish
      name: Publish to GitHub
      action: publish:github
      input:
        repoUrl: github.com?repo=${{ parameters.name }}&owner=my-org
        defaultBranch: main
        repoVisibility: internal
        requireCodeOwner: true
        dismissStaleReviews: true
        requiredApprovingReviewCount: 1

    - id: register
      name: Register in Catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['publish'].output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml

The imageRegistry field is constrained to an enum with a single approved value. A developer cannot scaffold a service that references Docker Hub as the base image registry. The sbomEnabled, slsaEnabled, and networkPolicyEnabled values are hardcoded to true in the step input — they are not parameters the developer can toggle.

The skeleton directory contains the actual rendered files. Key security-relevant files in the skeleton:

  • .github/workflows/ci.yml — the CI workflow with security steps pre-wired
  • Dockerfile — distroless base, non-root user, pinned digest
  • helm/templates/deployment.yaml — with securityContext blocks
  • helm/templates/networkpolicy.yaml — default-deny ingress/egress with minimal allow rules
  • catalog-info.yaml — Backstage catalog entry with security metadata

Secure Base Images: Distroless, Pinned Digests, Non-Root

The skeleton Dockerfile must enforce three things: a minimal base image, a pinned digest (not a floating tag), and a non-root runtime user.

FROM golang:1.22.3-alpine3.19@sha256:cdc86d9f363e8786845bea2f1b49b3b8e0c01f7f87fc73e7a5d80c49e9a3de3f AS builder
WORKDIR /build
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /app ./cmd/server

FROM gcr.io/distroless/static-debian12:nonroot@sha256:6706c73aae2afaa8201d63cc3dda48753c09bcd6c300762251065c0f7e602b2e
COPY --from=builder /app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]

Using @sha256: digest pins mean a dependency update to the base image requires an explicit digest change, which creates a diff in version control, which can be reviewed and scanned. Floating tags (golang:1.22 or latest) allow base image content to change silently between builds.

Distroless images contain only the application and its runtime dependencies — no shell, no package manager, no coreutils. The attack surface for a compromised container is substantially smaller: an attacker who achieves code execution has no curl, no bash, no wget to stage follow-on tooling.

If a team’s service genuinely requires a shell for debugging, the golden path should offer a debug variant that uses a Red Hat UBI minimal or Alpine image, with the distroless variant as the default. The Backstage template can present this as a parameter, but the default must be the more restrictive option.

The digest pinning process should be automated. A Renovate or Dependabot configuration in the skeleton keeps digests current and generates PRs with changelogs:

{
  "extends": ["config:base"],
  "docker": {
    "pinDigests": true
  },
  "packageRules": [
    {
      "matchDatasources": ["docker"],
      "allowedVersions": "/^(golang|gcr.io\\/distroless)/",
      "registryUrls": ["registry.internal.example.com"]
    }
  ]
}

Mandatory Security Controls in Golden Path CI

The skeleton CI workflow pre-configures four security steps that run on every pull request and main branch push. These are not optional jobs that teams enable when they feel like it — they are the CI workflow, and disabling them requires modifying a file that platform teams own via CODEOWNERS.

name: CI

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read
  packages: write
  id-token: write

jobs:
  sast:
    name: SAST (Semgrep)
    runs-on: ubuntu-latest
    container:
      image: registry.internal.example.com/semgrep:1.72.0@sha256:abc123...
    steps:
      - uses: actions/checkout@v4
      - name: Run Semgrep
        run: semgrep scan --config=auto --config=p/security-audit --sarif -o semgrep.sarif
        env:
          SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}
      - uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: semgrep.sarif

  build-and-scan:
    name: Build, Scan, SBOM, Provenance
    runs-on: ubuntu-latest
    needs: sast
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: false
          tags: ${{ env.IMAGE_REF }}
          load: true

      - name: Scan image (Trivy)
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.IMAGE_REF }}
          format: sarif
          output: trivy.sarif
          exit-code: '1'
          severity: CRITICAL,HIGH
          ignore-unfixed: true

      - name: Generate SBOM (Syft)
        uses: anchore/sbom-action@v0
        with:
          image: ${{ env.IMAGE_REF }}
          format: spdx-json
          output-file: sbom.spdx.json

      - name: Attest SBOM
        uses: actions/attest-sbom@v1
        with:
          subject-name: ${{ env.IMAGE_NAME }}
          subject-digest: ${{ steps.build.outputs.digest }}
          sbom-path: sbom.spdx.json
          push-to-registry: true

      - name: Generate SLSA provenance
        uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0
        with:
          image: ${{ env.IMAGE_NAME }}
          digest: ${{ steps.build.outputs.digest }}
          registry-username: ${{ secrets.REGISTRY_USER }}
        secrets:
          registry-password: ${{ secrets.REGISTRY_PASSWORD }}

Semgrep runs in a pre-approved internal image, not the public Docker Hub image. SBOM generation uses Syft via the anchore/sbom-action and attaches the result to the image in the registry as a signed attestation. SLSA provenance at level 3 is generated by the slsa-github-generator reusable workflow, which runs in an ephemeral, isolated environment and produces a provenance document that cannot be forged by code running in the main build job.

The SBOM generation step connects to the practices described in the SBOM generation and consumption article. The SLSA provenance step is covered in depth in SLSA build provenance.

NetworkPolicy and PodSecurityContext by Default in Helm Templates

The skeleton Helm chart generates secure Kubernetes manifests. Security contexts and network policies appear in every chart by default, not as optional values.

helm/templates/deployment.yaml skeleton fragment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "app.fullname" . }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "app.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "app.selectorLabels" . | nindent 8 }}
    spec:
      serviceAccountName: {{ include "app.serviceAccountName" . }}
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        fsGroup: 65534
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}@{{ .Values.image.digest }}"
          imagePullPolicy: IfNotPresent
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
          resources:
            limits:
              cpu: {{ .Values.resources.limits.cpu }}
              memory: {{ .Values.resources.limits.memory }}
            requests:
              cpu: {{ .Values.resources.requests.cpu }}
              memory: {{ .Values.resources.requests.memory }}

helm/templates/networkpolicy.yaml:

{{- if .Values.networkPolicy.enabled }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: {{ include "app.fullname" . }}
spec:
  podSelector:
    matchLabels:
      {{- include "app.selectorLabels" . | nindent 6 }}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
      ports:
        - protocol: TCP
          port: {{ .Values.service.port }}
    {{- range .Values.networkPolicy.additionalIngress }}
    - {{ toYaml . | nindent 6 }}
    {{- end }}
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
    {{- range .Values.networkPolicy.additionalEgress }}
    - {{ toYaml . | nindent 6 }}
    {{- end }}
{{- end }}

The default values.yaml sets networkPolicy.enabled: true. Ingress is limited to the ingress controller namespace. Egress allows only DNS resolution by default. Teams that need to reach a database or downstream service add entries to networkPolicy.additionalEgress. This forces explicit, reviewable declarations of all allowed traffic rather than silent allow-all defaults.

The image.digest field in values.yaml is required and has no default. A Helm install without a digest value fails. This enforces the same digest pinning at deployment time that the Dockerfile enforces at build time.

Enforcing Golden Path Adoption: Admission Control

Templates solve the creation problem. They do not solve the drift problem: teams that start on the golden path but later modify their manifests to disable security controls, or teams that deploy without using the template at all.

Admission control is the enforcement layer. Kyverno policies running as ValidatingWebhookConfiguration resources reject manifests that violate the security baseline. These policies apply to all workloads in application namespaces, regardless of how the manifest was produced.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-non-root
  annotations:
    policies.kyverno.io/title: Require Non-Root Containers
    policies.kyverno.io/category: Golden Path Enforcement
    policies.kyverno.io/severity: high
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-runAsNonRoot
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaceSelector:
                matchLabels:
                  environment: production
      validate:
        message: "Pods must set securityContext.runAsNonRoot: true at pod or container level."
        anyPattern:
          - spec:
              securityContext:
                runAsNonRoot: true
          - spec:
              containers:
                - securityContext:
                    runAsNonRoot: true
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-approved-base-image
  annotations:
    policies.kyverno.io/title: Require Approved Base Image Registry
    policies.kyverno.io/category: Golden Path Enforcement
    policies.kyverno.io/severity: critical
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-registry
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaceSelector:
                matchLabels:
                  environment: production
      validate:
        message: "Images must be pulled from registry.internal.example.com"
        foreach:
          - list: "request.object.spec.containers"
            deny:
              conditions:
                any:
                  - key: "{{ element.image }}"
                    operator: NotIn
                    value:
                      - "registry.internal.example.com/*"
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-network-policy
  annotations:
    policies.kyverno.io/title: Require NetworkPolicy Coverage
    policies.kyverno.io/category: Golden Path Enforcement
    policies.kyverno.io/severity: medium
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: check-networkpolicy-exists
      match:
        any:
          - resources:
              kinds:
                - Namespace
              namespaceSelector:
                matchLabels:
                  environment: production
      validate:
        message: "Each namespace must have a default-deny NetworkPolicy."
        deny:
          conditions:
            any:
              - key: "{{ request.object.metadata.name }}"
                operator: NotIn
                value: "{{ request.object.metadata.annotations.\"platform.example.com/networkpolicy-verified\" || '' }}"

For deeper coverage of Kyverno policy authoring, the admission webhook architecture, and generating policy reports, see Kyverno controller security.

These policies also apply retroactively via background scanning. Kyverno generates PolicyReport resources with violations for existing workloads. This surfaces drift without requiring a deployment event to trigger the check.

Metrics for Golden Path Adoption

Enforcement via admission control tells you what is blocked. Metrics tell you the current state of the fleet and whether security posture is improving over time.

Four metrics provide meaningful signal:

% of production workloads using approved base images. Query the Kyverno PolicyReport API or scan the container registry for image digests and match them against the approved image list. A Prometheus scraper on the Kyverno policy report endpoint makes this a dashboard metric:

sum(kyverno_policy_results_total{policy="require-approved-base-image", result="pass"})
/
sum(kyverno_policy_results_total{policy="require-approved-base-image"})

% of images with an attached SBOM attestation. Query the registry using cosign verify-attestation against each image digest in the registry. A nightly job that iterates over all images in production namespaces and checks for a valid SBOM attestation (predicate type https://spdx.dev/Document) produces a count that can be exported as a gauge metric.

% of services with an active NetworkPolicy. Count namespaces in application environments that have at least one NetworkPolicy with a non-empty pod selector, divided by total application namespaces. kubectl get networkpolicies --all-namespaces -o json | jq queries can populate this; a kube-state-metrics custom resource metric exporter is cleaner at scale.

Golden path adoption rate. The Backstage catalog tags services created from golden path templates with a label (backstage.io/template: secure-go-service). Services without this label were created outside the golden path. The adoption rate is the fraction of catalog-registered services with the template label. This is the leading indicator — other metrics lag it by the time it takes teams to deploy.

Track these metrics per team and per business unit, not only as fleet aggregates. A fleet-level 95% adoption rate can mask a single team with 0% adoption that happens to run the highest-risk service.

Handling Exceptions

No golden path enforcement survives contact with a sufficiently large organization without a formal exception process. Teams will have legitimate reasons to deviate: a legacy service being migrated incrementally, a third-party operator that deploys its own images, a compliance requirement that mandates a specific base image from a vendor.

The exception process must have four properties to avoid becoming a rubber-stamp:

Time-bounded. Every exception has an expiry date, stored as an annotation on the exempted namespace or workload. A controller watches these annotations and alerts or auto-revokes when the expiry passes:

metadata:
  annotations:
    platform.example.com/policy-exception: "require-approved-base-image"
    platform.example.com/exception-expiry: "2026-08-01"
    platform.example.com/exception-reason: "vendor image: Datadog operator requires registry.hub.docker.com/datadog/agent"
    platform.example.com/exception-ticket: "SEC-4821"
    platform.example.com/exception-approver: "security-team"

A Kyverno policy exception resource provides the actual enforcement bypass. The annotation is metadata for the governance record; the Kyverno PolicyException resource is what tells the webhook to pass the workload:

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: datadog-agent-base-image
  namespace: monitoring
  annotations:
    platform.example.com/exception-expiry: "2026-08-01"
    platform.example.com/exception-ticket: "SEC-4821"
spec:
  exceptions:
    - policyName: require-approved-base-image
      ruleNames:
        - check-registry
  match:
    any:
      - resources:
          kinds:
            - Pod
          namespaces:
            - monitoring
          selector:
            matchLabels:
              app: datadog-agent

Scoped to a specific workload. Exceptions cover named deployments or namespaces, not entire clusters or environments. A namespace-level exception for the monitoring namespace does not exempt unrelated workloads in that namespace.

Auditable. Every exception appears in a central registry (a Git repository of exception YAML files works well). New exceptions require a pull request with approval from the security team. The history of exceptions — granted, expired, revoked — is in the commit log.

Actively expired. A controller (a simple CronJob that runs kubectl get policyexceptions --all-namespaces -o json and checks expiry annotations against the current date) deletes or patches expired exceptions to remove the bypass. Teams receive advance warning via Slack or ticket automation before expiry, giving them time to complete the migration or renew the exception with updated justification.

The combination of admission control enforcement and a governed exception process means the default state is compliance, and non-compliance is visible, time-limited, and attributed to a specific team and decision.

Closing

The golden path is only as valuable as its adoption and enforcement. A Backstage template that developers ignore, with no admission control backing, produces documentation artifacts rather than security outcomes. The model that works is: make the golden path easy enough that using it is the default choice, and enforce it at admission time so that bypassing it requires deliberate action that creates an audit trail.

The investment concentrates in three places: the template itself (written once, maintained by the platform team), the admission control policies (written once, applied to every workload), and the exception process (overhead per exception, not per service). The per-service cost of security drops toward zero as adoption grows. That is the compounding return on the paved road.