ObliviOS Developer portal Project site

Home / Decisions / ADR-029

Accept that the 2026-07-31 lynx alpha candidate is not byte-reproducible by construction, replace it with the already-completed, correctly-configured reproduction build…

Accepted, recorded 4 August 2026.

Decision
ADR-029
Date
2026-08-04
Status
AcceptedAccepted
Mentions
ADR-018, ADR-022, ADR-025, ADR-026, ADR-028, ADR-030
Mentioned by
ADR-030, ADR-031, ADR-078, ADR-099, ADR-100

1Decision

Accept that the 2026-07-31 lynx alpha candidate is not byte-reproducible by construction, replace it with the already-completed, correctly-configured reproduction build as the new alpha candidate, and accept that finding without a second confirming rebuild.

2Why and evidence

Task 4's independent reproduction (<build-root>/platform-verify, same-server independent checkout per Step 1b, full sync+build, 8h12m, 177452 ninja steps) found 8 of 12 partitions differing from the reference candidate. Root-cause investigation (methodology matching ADR-022 through ADR-025) explained all 8 with no residual — stronger than ADR-025's own 2-file residual.

Two causes, both independently re-verified by the controller directly against primary evidence, not merely trusted from the investigation report: (A) the reference build actually used OUT_DIR=out-preflight (confirmed via grep OUT_DIR on the reference build's own log), not out-lynx as this plan's Task 2 wrongly recorded from stray fingerprint-stub files rather than the real build log — leaking into ELF build-IDs and generated-source debug paths via the same mechanism ADR-025 already proved on the emulator target (-fdebug-prefix-map normalises the source-tree prefix but not the OUT_DIR-relative suffix that follows it); (B) USE_FIXED_TIMESTAMP_IMG_FILES was set on the verify build only (zero occurrences in the reference build's log; controller-independent debugfs -R "show_super_stats -h" on system_dlkm.img confirmed real wall-clock 2026-07-31 21:32:06 on the reference side against the fixed epoch 2009-01-01 on the verify side).

A targeted single-module spot-check corroborated Cause A specifically for this target without a full rebuild: rebuilding the toolbox.recovery module in platform-primary with OUT_DIR=out-lynx (matching platform-verify) produced a toolbox binary with SHA-256 3d274be480492a2e5d44341850c32c7eb8bda9df632a99989511532af56d1447, byte-for-byte identical (cmp -l: zero differing bytes) to platform-verify's own build of the same module — full detail in .superpowers/sdd/alpha-task4-toolbox-spotcheck.md, including a controller misread mid-check (an idle process snapshot landed seconds before the build's actual completion and was initially reported as a failure; corrected after the subagent's independent re-check) and a real, separate cost finding: that single-module attempt still consumed ~10G on a cold out-lynx from Kati/Soong full-graph regeneration alone, not the module compile itself, which drove /srv to 2.7G free / 100% full and required deleting platform-verify's now-redundant out-lynx/soong (86G, owner-approved, useful output already preserved) to recover to 89G free.

3Consequences

Cause A is fixable by using a consistent OUT_DIR name. Cause B is not fixable after the fact: the reference build's ext4 inode/superblock timestamps are real wall-clock values with no build-input origin, so no future rebuild — of either tree — can ever reproduce the 2026-07-31 candidate's exact bytes.

That candidate is therefore permanently disqualified from an alpha-channel reproducibility claim, not merely "not yet verified." Per Task 4's own text ("either fixed or explained with the same rigor those ADRs used"), the owner accepted this root-cause explanation in place of a second ~8h confirming rebuild — ccache was checked and found unable to meaningfully help (<build-root>/ccache capped at 5GB/90% full with an 18.7% historical hit rate, and out-lynx appears 2,025,265 times in the verify build's own ninja file, meaning a change of OUT_DIR re-keys nearly every compile action regardless of cache size).

The already-completed platform-verify build — which already carries every correct flag, OUT_DIR included, since it is the one being adopted as the reference name going forward — is promoted to be the alpha candidate itself, recorded at <build-root>/evidence/alpha-candidate-accepted-hashes-20260804.txt (candidate_id alpha-candidate-20260804-01, supersedes alpha-candidate-20260803-01), independently hash-verified by the controller against <build-root>/evidence/verify-task4-final-result.txt before this entry was recorded.

ADR-026's development-channel signing is not invalidated (reproducibility was explicitly waived there, same posture as ADR-018) but its provenance caveat now has this specific, documented reason. Task 5 onward targets the new candidate.

This weakens one of ADR-028's four named compensating controls for the single-operator model — "independent reproduction on a genuinely separate environment stands in for a second build operator" — from a full independent-build confirmation to a root-cause explanation plus a single-module spot-check; recorded here explicitly rather than left implicit. Full investigation: .superpowers/sdd/alpha-task-4-rootcause-report.md.

The plan's own Task 8 draft had pre-reserved the label "ADR-029" for the eventual signing-ceremony record; that record is renumbered to ADR-030 since this entry now occupies ADR-029 by sequence.

4Review when

A rebuild of platform-primary with the verify build's exact flag set is later performed and produces a clean diff against the promoted candidate (strengthens this entry, does not require reopening it); or the owner later decides the weakened compensating control is insufficient for a channel beyond alpha, single-device scope (beta/stable explicitly cannot use this model per ADR-028 already).

Read from docs/governance/decision-log.md, entry ADR-029.