Reproducible Builds: Eliminating Build Environment as a Supply Chain Attack Surface

Why This Matters

The XZ Utils compromise (CVE-2024-3094) was not a vulnerability in source code visible to a code reviewer — it was a backdoor injected into the distribution tarballs. The attacker, operating as “Jia Tan,” modified release artifacts that diverged from what git contained. A reproducible build system would have caught this immediately: the tarballs produced by an independent rebuild of the tagged commit would not match the published artifacts, and any automated comparison would have failed loudly.

SolarWinds was structurally different — the attacker compromised the build pipeline itself — but the diagnostic principle is the same. If you can independently reproduce every artifact from source and verify it matches what consumers receive, then either your build environment is also compromised (now you have a much harder problem with a much smaller attack surface), or you have a guarantee the artifact is clean.

Reproducibility is a verification primitive, not a feature. It converts “trust the publisher” into “verify independently.” Without it, every binary you ship is an assertion that nothing went wrong in your CI — an assertion your downstream users cannot check. With it, they can.

See also: SLSA build provenance for how provenance attestations complement reproducibility, and dependency confusion defence for the adjacent supply chain risk.

Root Causes of Non-Reproducibility

Understanding why builds are not reproducible is necessary before you can fix them. The failure modes are consistent across ecosystems.

Embedded Timestamps

The most common source of divergence. Compilers, archivers, and documentation generators routinely embed the current time into their output.

# .a static archives embed file modification times
$ ar t libfoo.a
foo.o  # header contains: mtime=1715289600

# .deb packages embed build time in ar headers
$ ar p foo.deb control.tar.gz | tar -tvz
-rw-r--r-- builder/builder  0 2026-05-09 14:32 ./

# Python .pyc files embed source mtime
$ python3 -c "import py_compile; py_compile.compile('foo.py')"
$ xxd __pycache__/foo.cpython-312.pyc | head -2
00000000: 0d0d 0d0a 00000000 8b3c 0d67 ...

Non-Deterministic Linking Order

Linkers produce output whose section layout depends on the order files are passed to them. File system traversal order — find, ls, glob expansion — is not guaranteed to be stable across invocations, kernels, or filesystems.

# Non-deterministic: glob order is filesystem-dependent
gcc -o mybinary $(ls *.o)

# Deterministic: sort explicitly
gcc -o mybinary $(ls *.o | sort)

DWARF debug info is especially prone to this. Section offsets change when link order changes, which changes the .debug_line and .debug_info sections, which changes the binary hash even if the executable code is identical.

Build Path Embedding

__FILE__ macros, DWARF DW_AT_comp_dir, Python __file__, and Go’s runtime.Callers all embed absolute build paths into compiled output. If your CI builds in /home/runner/work/project and a colleague builds in /home/alice/src/project, the binaries will differ.

// This embeds the full path at compile time
printf("Error at %s:%d\n", __FILE__, __LINE__);

Locale and Environment Leakage

Tools that sort, format, or filter output based on LC_ALL, LC_COLLATE, or LANG produce different results across environments. Makefiles that invoke sort without LC_ALL=C produce locale-dependent ordering. Man page formatters embed locale-specific hyphenation. Perl scripts reading /usr/lib/locale produce different output on different distros.

Nondeterministic Parallelism

Build systems that dispatch jobs to a thread pool and aggregate results in completion order will interleave output differently on each run. Archive members, object file tables, and linker map files all reflect this.

ASLR and PIE Artifacts

Position-independent executables built with certain toolchains embed relocation tables whose layout can vary if the build process itself invokes the binary during compilation (code generators, reflection-based tools). ASLR affects address-space layout at runtime, not at link time, so this is rare — but profiling-guided optimization (PGO) that uses sampled addresses does produce non-reproducible output if samples differ between builds.

SOURCE_DATE_EPOCH

SOURCE_DATE_EPOCH is a POSIX epoch timestamp (integer seconds since 1970-01-01 00:00:00 UTC) that instructs build tools to use a fixed, source-controlled timestamp instead of the current time. It is the primary mechanism for eliminating timestamp-based divergence.

Set it to the timestamp of the last source commit:

export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)

Propagation Through Build Tools

Make: $(shell date ...) calls in Makefiles do not automatically respect SOURCE_DATE_EPOCH. You must replace them:

# Before
BUILD_DATE := $(shell date -u +%Y-%m-%dT%H:%M:%SZ)

# After
BUILD_DATE := $(shell date -u -d @$(SOURCE_DATE_EPOCH) +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
              || date -u -r $(SOURCE_DATE_EPOCH) +%Y-%m-%dT%H:%M:%SZ)

CMake: CMake 3.8+ respects SOURCE_DATE_EPOCH for its own timestamps. For embedded __DATE__ and __TIME__ macros in C/C++ sources, pass -DCMAKE_C_FLAGS="-ffile-prefix-map=$(pwd)=." and handle timestamps explicitly:

string(TIMESTAMP BUILD_DATE "%Y-%m-%d" UTC)
# Replace with:
if(DEFINED ENV{SOURCE_DATE_EPOCH})
  execute_process(
    COMMAND date -u -d "@$ENV{SOURCE_DATE_EPOCH}" +%Y-%m-%d
    OUTPUT_VARIABLE BUILD_DATE OUTPUT_STRIP_TRAILING_WHITESPACE
  )
endif()

Cargo (Rust): The SOURCE_DATE_EPOCH environment variable is passed through to build scripts (build.rs). The vergen crate, commonly used to embed build metadata, respects it since version 7:

[build-dependencies]
vergen = { version = "8", features = ["build", "git", "rustc"] }

// build.rs
fn main() {
    vergen::EmitBuilder::builder()
        .build_timestamp()
        .emit()
        .unwrap();
}

With SOURCE_DATE_EPOCH set, vergen uses that value instead of SystemTime::now().

Go: Go’s toolchain does not embed build timestamps by default. The -trimpath flag eliminates embedded build paths:

go build -trimpath -o mybinary ./cmd/server

Set via GOFLAGS to apply globally:

export GOFLAGS="-trimpath"

Go’s Built-In Reproducibility

Go is unusual in shipping reproducible builds as a first-class property.

-trimpath removes the module cache path and working directory from all recorded file paths. Without it, a binary built from /home/alice/go/pkg/mod/github.com/foo/bar@v1.2.3 will differ from one built from /home/runner/go/pkg/mod/github.com/foo/bar@v1.2.3 in DWARF data.

GOPROXY pinning ensures dependency resolution is deterministic:

export GOPROXY="https://proxy.golang.org,direct"
export GONOSUMCHECK=""
export GOFLAGS="-trimpath"

The go.sum file provides cryptographic pinning of every transitive dependency. A build that cannot satisfy go.sum verification fails. Combined with GOFLAGS=-trimpath and GONOSUMDB="" (requiring all modules to appear in the sum database), Go builds are highly reproducible without additional tooling.

CGo breaks this. A binary with CGo links against system libraries whose versions may differ. Disable CGo for reproducible builds when feasible:

CGO_ENABLED=0 go build -trimpath -o mybinary ./...

When CGo is required, pin the libc version via a container with a fixed base image and verify the base image by digest, not tag.

Nix: Derivation Hashing and Content-Addressed Stores

Nix takes a fundamentally different approach. Rather than retrofitting reproducibility onto an existing build system, Nix’s build model makes reproducibility structurally mandatory.

Every build in Nix is a derivation — a pure function from a set of inputs (source, compiler, flags, environment variables) to a set of outputs. The derivation is hashed; the output path in /nix/store encodes that hash:

/nix/store/ywyvs8yq1klyb7yzrwpfrfvkfd1kn9dv-openssl-3.3.2/
           ^-- hash of all inputs to the derivation

The build environment is hermetically sandboxed: no network access (by default), no access to /home, no ambient environment variables, fixed uid/gid. If two derivations have the same input hash, they produce the same output.

Content-Addressed Store

Nix 2.4+ added content-addressed derivations (experimental, increasingly production-ready) where the store path is determined by the hash of the output rather than the inputs. This means two derivations that produce identical output share the same store path even if their input hashes differ — a stronger reproducibility guarantee.

Enable with:

{
  description = "reproducible build";
  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.default = nixpkgs.legacyPackages.x86_64-linux.stdenv.mkDerivation {
      name = "myapp";
      src = self;
      __contentAddressed = true;
      outputHashAlgo = "sha256";
      outputHashMode = "recursive";
    };
  };
}

Binary Cache Substituters and Trust Model

Nix’s binary cache (cache.nixos.org) stores pre-built outputs keyed by store path. When you nix build, Nix first checks if the expected output path exists in the cache — if so, it substitutes the cached binary instead of rebuilding.

The trust model: the cache is trusted based on its signing key. Any binary fetched from a substituter is verified against the public key before being added to the local store:

nix.settings = {
  substituters = [ "https://cache.nixos.org" "https://your-cache.example.com" ];
  trusted-public-keys = [
    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    "your-cache.example.com-1:<key>"
  ];
};

An independent rebuilder can rebuild a derivation, compute the output hash, and compare it to what the cache served. A mismatch is evidence of cache tampering or non-reproducibility.

Debian Reproducible Builds

The Debian reproducible builds project has been systematically making the Debian package archive reproducible since 2013. As of 2026, over 95% of Debian packages in trixie build reproducibly.

rebuilderd

rebuilderd is the infrastructure that continuously rebuilds Debian (and Arch, Fedora) packages and publishes verdicts. A verdict of GOOD means the rebuild matched the published binary; BAD means it did not.

# Install rebuilderd worker
apt install rebuilderd

# Add a Debian suite for monitoring
rebuildctl pkgs sync-suite --suite bookworm --architecture amd64

# Query a package verdict
rebuildctl pkgs ls --name openssl

The public instance at rb-pkg.debian.net shows per-package reproducibility status. Build maintainers receive automated bug reports when their packages regress.

.buildinfo Files

Debian .buildinfo files record the complete build environment: exact package versions of every build dependency, the build host architecture, and checksums of all output files. They are signed by the build daemon and uploaded alongside .deb files.

Format: 1.0
Build-Architecture: amd64
Source: openssl
Version: 3.3.2-1
Checksums-Sha256:
 abc123... 1234 openssl_3.3.2-1_amd64.deb
Build-Depends: gcc (= 13.3.0-2), ...

An independent rebuilder uses the .buildinfo to reproduce the exact build environment, then compares output checksums.

diffoscope: Diagnosing Unreproducible Builds

When a build is not reproducible, you need to find out why. diffoscope is the standard tool. It recursively unpacks archives, decompiles binaries, and produces a human-readable diff showing exactly where two artifacts diverge.

pip install diffoscope
# or
apt install diffoscope

Compare two builds of the same package:

diffoscope build1/mypackage_1.0_amd64.deb build2/mypackage_1.0_amd64.deb

Output for a timestamp issue:

├── control.tar.gz
│   └── ./control
│     @@ -5,3 +5,3 @@
│     -Date: Sat, 09 May 2026 10:00:00 +0000
│     +Date: Sat, 09 May 2026 10:05:00 +0000
└── data.tar.gz
    └── ./usr/bin/myapp
      ├── readelf --wide --debug-dump=info {}
      │   @@ -102,3 +102,3 @@
      │   -    (DW_AT_comp_dir  : /home/runner/work/myapp)
      │   +    (DW_AT_comp_dir  : /home/alice/src/myapp)

This immediately identifies two problems: an embedded build timestamp and a non-stripped build path. Fix with SOURCE_DATE_EPOCH and -ffile-prefix-map (GCC/Clang) or -trimpath (Go).

For ELF binaries, diffoscope invokes readelf, objdump, and strings to surface differences that would be invisible in a raw binary diff:

# Standalone ELF comparison
diffoscope --text-color always build1/mybinary build2/mybinary | less -R

# HTML report for CI artifact upload
diffoscope --html diff-report.html build1/mybinary build2/mybinary

GCC’s -ffile-prefix-map flag remaps embedded paths:

gcc -ffile-prefix-map=$(pwd)=. -o mybinary main.c

This replaces the absolute build path with . in all embedded debug and source path information.

Hermetic Build Sandboxing with Bazel

Bazel’s core design principle is hermeticity: every build action runs in a sandbox with an explicit, declared dependency set. No action can read files that are not declared inputs; no action can write files that are not declared outputs.

This directly enables reproducibility. Because every action’s inputs are known and fixed, and the action graph is deterministic, the same source produces the same output on any machine with the same toolchain.

# BUILD file
cc_binary(
    name = "server",
    srcs = ["main.cc"],
    deps = [
        "//lib:crypto",
        "@openssl//:ssl",
    ],
)

Bazel’s remote build execution (RBE) extends this to distributed builds. Actions are dispatched to remote workers; because actions are hermetic, any worker can execute any action and produce the same output. Results are cached by action hash:

build --remote_executor=grpcs://remotebuildexecution.googleapis.com
build --remote_instance_name=projects/myproject/instances/default
build --remote_download_minimal

--remote_download_minimal downloads only final outputs, not intermediate artifacts — the cache serves all intermediates from the action cache. Two developers building the same commit get identical outputs from cache, never rebuilding.

For Bazel + Docker, rules_docker pins base image digests:

container_pull(
    name = "ubuntu_base",
    registry = "index.docker.io",
    repository = "library/ubuntu",
    digest = "sha256:a1b2c3...",  # pinned, not a tag
)

Integrating Reproducibility Checks in CI

The canonical CI pattern is: build twice, compare, fail on mismatch.

# GitHub Actions example
jobs:
  reproducible-build:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      
      - name: Set SOURCE_DATE_EPOCH
        run: echo "SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)" >> $GITHUB_ENV
      
      - name: First build
        run: |
          make -j$(nproc) GOFLAGS="-trimpath"
          cp dist/mybinary dist/mybinary.build1
      
      - name: Clean build artifacts (keep source)
        run: make clean
      
      - name: Second build
        run: |
          make -j$(nproc) GOFLAGS="-trimpath"
          cp dist/mybinary dist/mybinary.build2
      
      - name: Compare builds
        run: |
          sha256sum dist/mybinary.build1 dist/mybinary.build2
          if ! diff -q dist/mybinary.build1 dist/mybinary.build2; then
            echo "BUILD IS NOT REPRODUCIBLE"
            diffoscope dist/mybinary.build1 dist/mybinary.build2 --html diffoscope.html || true
            exit 1
          fi
          echo "Builds match: reproducible"
      
      - name: Upload diffoscope report on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: diffoscope-report
          path: diffoscope.html

For Debian packages, use the reprotest tool which goes further — it varies build path, username, hostname, time zone, filesystem ordering, and CPU count between the two builds to stress-test reproducibility:

apt install reprotest
reprotest 'dpkg-buildpackage -b' '../mypackage_*.deb'

reprotest applies a configurable set of variations. A package that survives all variations is genuinely reproducible, not just accidentally identical in a controlled environment.

For container images, compare layer digests:

#!/usr/bin/env bash
set -euo pipefail

IMAGE="myapp"
TAG="$(git rev-parse --short HEAD)"

docker build -t "${IMAGE}:${TAG}-build1" .
docker build -t "${IMAGE}:${TAG}-build2" --no-cache .

DIGEST1=$(docker inspect --format='{{index .RepoDigests 0}}' "${IMAGE}:${TAG}-build1" 2>/dev/null \
          || docker inspect --format='{{.Id}}' "${IMAGE}:${TAG}-build1")
DIGEST2=$(docker inspect --format='{{index .RepoDigests 0}}' "${IMAGE}:${TAG}-build2" 2>/dev/null \
          || docker inspect --format='{{.Id}}' "${IMAGE}:${TAG}-build2")

if [ "$DIGEST1" != "$DIGEST2" ]; then
  echo "Container image is not reproducible"
  # Compare layer by layer
  docker save "${IMAGE}:${TAG}-build1" | tar -tv > layers1.txt
  docker save "${IMAGE}:${TAG}-build2" | tar -tv > layers2.txt
  diff layers1.txt layers2.txt
  exit 1
fi

For OCI images, crane provides content-addressable comparison:

crane digest myapp:build1
crane digest myapp:build2

Summary

Reproducible builds do not prevent a compromised build machine from producing malicious output — but they make that compromise detectable. Any independent rebuilder can verify the artifact. That shifts the attacker’s requirement from “compromise one build pipeline” to “compromise every independent rebuild infrastructure simultaneously,” which is a significantly harder problem.

The minimum viable reproducibility stack:

1. Set SOURCE_DATE_EPOCH in CI, derived from git log -1 --pretty=%ct.
2. Use -trimpath (Go) or -ffile-prefix-map=$(pwd)=. (C/C++) to strip build paths.
3. Sort all file lists before passing them to linkers and archivers.
4. Build twice per CI run, compare with sha256sum, surface diffs with diffoscope.
5. For packages: generate and publish .buildinfo-equivalent metadata.
6. For infrastructure: adopt Nix or Bazel where hermeticity is a hard requirement.

This is not a complete supply chain solution — pair it with SLSA build provenance for signed attestations and dependency confusion defence for dependency integrity. But without reproducibility, provenance is unverifiable: you can prove who built the artifact, not that the artifact matches the source.