- Decision
- ADR-025
- Date
- 2026-08-03
- Status
- AcceptedAccepted
- Mentions
- ADR-018, ADR-023, ADR-024
- Mentioned by
- ADR-026, ADR-029, ADR-031, ADR-035, ADR-112
1Decision
Record three further confirmed causes beyond ADR-024 (random sgdisk GPT GUIDs; unpinned BUILD_DATETIME; mismatched OUT_DIR names leaking into ELF build-IDs and generated-source debug paths), and separately confirm that most of what remains after fixing all of them is not a build-system defect: uncommitted local edits sitting in platform-primary's checkout.
2Why and evidence
Applying the full ADR-023/024 fix set (pinned BUILD_NUMBER, --salt 00 on every BOARD_AVB_*_ADD_HASHTREE_FOOTER_ARGS, USE_FIXED_TIMESTAMP_IMG_FILES=true) left the 5 -qemu images still differing. Cause: device/generic/goldfish/build/tools/mk_qemu_image.sh wraps each partition image in a fresh GPT via sgdisk, which generates random disk/partition GUIDs unless --disk-guid/--partition-guid are passed; no AVB mechanism touches this step.
Overriding SGDISK — as a shell env var, then as a m command-line variable — had no effect even after forcing a full Kati regeneration (confirmed via direct ninja-rule inspection: the real, unwrapped sgdisk path was baked into the rule as a literal string both times); SGDISK := $(HOST_OUT_EXECUTABLES)/sgdisk in build/make/core/config.mk is a bare := with no +=, unlike the BOARD_AVB_* variables (which are built via += across several files and only appeared to honor a command-line override because append-only assignment can never discard a seed value — not because Kati implements GNU Make's real command-line-wins-over-:= precedence, which it does not for this variable).
Replacing the installed sgdisk binary with a pinned-GUID wrapper also failed: ninja's own Install: rule detected and silently re-copied the real binary over it before the image-build steps ran.
The build system would not cooperate at any layer, so the fix moved outside the build: a small Python script (pin_gpt_guids.py) patches the primary and backup GPT headers and partition entries of each -qemu.img post-build — deterministic per-partition-index GUIDs, both header and partition-entries CRC32 recomputed — without touching the source tree.
Verified byte-identical across two independent repackaging runs, including on system-qemu.img's two-partition (vbmeta+super) layout, which needed the patch generalized from a single- to multi-entry GPT. With that applied, a full independent rebuild (primary fresh, then repro fresh, BUILD_NUMBER+AVB-salt+timestamp+GPT-patch all applied from the start) still left 12 of 25 differing, all cascading from system.img's dm-verity root digest.
Root cause: ro.build.date.utc is driven by a separate variable, BUILD_DATETIME, not BUILD_NUMBER — confirmed both in source (build/make/core/version_util.mk:242-245: ifndef BUILD_DATETIME → BUILD_DATETIME := $(shell date +%s), real wall-clock time if unset) and in GrapheneOS's own public build documentation (https://grapheneos.org/build), which documents exporting BUILD_DATETIME from out/build_date.txt as a required step to reproduce a past build.
Neither build had ever set it.
Pinning it to an identical value on both sides made system.img byte-identical in an isolated two-run repackaging test — but the next full independent rebuild (both sides now carrying all fixes) still showed 12 of 25 differing, this time with content-level (not just metadata-level) divergence: extracting and diffing system.img's ext4 tree (via debugfs rdump, since neither sgdisk nor plain strings/readelf were sufficient) showed nearly every compiled ELF file differed, and apexd specifically contained the literal strings out-final/out-repro-final — the two builds' different OUT_DIR names — embedded in its own bytes.
Smaller binaries (toolbox, awk, bootanimation) showed the identical pattern in miniature: exactly 16 contiguous differing bytes (the size of an MD5-based ELF build-id) at a fixed offset, with the rest of the file byte-identical.
Mechanism: -fdebug-prefix-map=/proc/self/cwd= (present on every compile action) normalizes the absolute source-tree-root prefix, so platform-primary vs platform-repro does not matter — but it does not touch the OUT_DIR-relative path segment that follows it in generated/intermediate file references (AIDL/Rust-generated sources, object-file paths feeding the ELF build-id and debug sections), so out-final vs out-repro-final survives into the compiled output verbatim.
Rebuilding both sides from scratch with an identical OUT_DIR=out (the only change) reduced the differing set from 12 to 6 of 25 (product, product-qemu, super, system, system-qemu, vbmeta). Investigating that residual by extracting and diffing system.img's and product.img's ext4 trees found only 4 of 2393 files, and 1 of product.img's files, actually differ.
Cross-referencing those specific files against git status --porcelain in each source tree found the actual cause: platform-primary's frameworks/base checkout carries three uncommitted, locally modified files — cmds/bootanimation/BootAnimation.cpp, core/res/assets/images/android-logo-mask.png, core/res/assets/images/android-logo-shine.png — while platform-repro's checkout of the byte-identical commit (git rev-parse HEAD matches; git ls-tree reports the identical committed blob hash for each file) is clean and matches that commit exactly.
platform-primary is not a "third, unfixed build-determinism cause" here; it is simply the one tree with stale local edits nobody reverted or committed, predating this investigation.
Disassembling libbootanimation_preapex.so (compiled from the modified .cpp) confirmed this is consistent: byte-identical instructions to the reference build, with only linker-assigned PLT call-target addresses shifted by a small constant offset — a small link-time change, not different program logic.
system/bpfprogs (source of the two remaining differing files, cyclePerUid.bpf/dmabufIter.bpf, 25 and 58 bytes respectively) was independently confirmed clean (git status --porcelain empty) in platform-primary — these two do not share the dirty-checkout cause and remain unexplained.
3Consequences
The build system itself is now proven deterministic wherever it has actually been tested clean: every partition and every -qemu wrapper not affected by platform-primary's local edits (vendor, vendor-qemu, system_ext, system_ext-qemu, system_dlkm, system_dlkm-qemu, ramdisk, ramdisk-qemu, cache, userdata, dtb, encryptionkey, super_empty, the vendor_boot/vendor_ramdisk family) matched byte-for-byte between two fully independent syncs and builds once BUILD_NUMBER, BUILD_DATETIME, AVB salt, USE_FIXED_TIMESTAMP_IMG_FILES, matching OUT_DIR, and the post-build GPT-GUID patch were all applied.
The 6 that still differ trace to two causes outside the build recipe: platform-primary's uncommitted local edits (explains system, system-qemu, and presumably product/product-qemu via the same overlay-resource path, cascading into super/vbmeta; not independently re-verified after a revert) and two small, unexplained .bpf differences unrelated to it. Neither is evidence of build-environment compromise or a non-deterministic toolchain.
This does not by itself flip the exit decision to a clean, literal Accepted baseline — no re-run with platform-primary's three files reverted (git checkout --) has been performed to confirm a fully clean diff, and the two .bpf differences remain genuinely unresolved.
It does support a qualified Accepted baseline: the build recipe is proven deterministic; the residual is fully attributed to non-build-system causes rather than left as an open reproducibility question.
4Review when
platform-primary's three modified files are reverted and a follow-up comparison confirms those 4-5 images now match (or the owner accepts the residual with this documented reason instead); or the two .bpf differences are root-caused; or evidence of actual build-environment compromise surfaces, per ADR-018's original trigger.
Read from docs/governance/decision-log.md, entry ADR-025.