- Decision
- ADR-046
- Date
- 2026-08-24
- Status
- AcceptedAccepted
- Mentions
- ADR-030, ADR-041, ADR-045
- Mentioned by
- ADR-047, ADR-048, ADR-078, ADR-079
1Decision
Accept the root-cause resolution of ADR-045's Task 7 flash failure (lynx AVB red-state "corrupt" boot, docs/devices/lynx-ota-server-flash-rollback-corruption-incident.md): two independent, stacked bugs in the signed vbmeta, neither in the device or in system.img.
2Why and evidence
Following this project's own systematic-debugging discipline (root cause before fix, hypothesis testing over assumption): the initial leading hypothesis (device hardware rollback counter drifted above the reused 1783209600) was investigated and found unsupported — direct inspection instead showed the flashed vbmeta itself carried Rollback Index: 0, the exact make_vbmeta_image-without---rollback_index slip ADR-030 already hit once, confirmed via the signing host's own .bash_history rather than reconstructed.
Re-signing with the correct index alone was insufficient: the corrected file still failed identically on real hardware, proving a second bug.
Root-caused by direct sha256sum comparison: the 11 partitions Task 7 treated as "unchanged" were sourced from the 2026-08-04 alpha candidate, but the device's real last-flashed state (from the duress dummy-cost primitive's Task 3, docs/superpowers/plans/2026-08-15-duress-dummy-cost-primitive.md, independently confirmed booting 2026-08-20) used Task 3's own full-build copies instead — a full droid build run without this project's usual BUILD_NUMBER/BUILD_DATETIME/USE_FIXED_TIMESTAMP_IMG_FILES discipline had silently produced different real bytes for 10 of those 11 functionally-unchanged partitions.
Before trusting Task 3's files as the correct reference, cross-verified that Task 3's own vbmeta independently verifies 13/13 against them in an isolated directory (avbtool's verify_image was found to implicitly and silently resolve partition files by name in the vbmeta's own working directory with no ambiguity warning — a real, separate footgun hit and isolated with a symlink-only verification directory, not the cause of this incident but discovered investigating it).
The corrected vbmeta-ota-server-signed-v2.img (today's system-ota-server-signed.img + Task 3's real 11 files + --rollback_index 1783209600) was independently verified 13/13 on two separate machines (the signing host and a separate workstation, different avbtool builds), diffed descriptor-by-descriptor against both the insufficient first fix and Task 3's own proven-good vbmeta before flashing, then flashed and confirmed via real device evidence, not just static verification: sys.boot_completed=1, ro.build.fingerprint incremental (2026081401) matching the value this investigation's own descriptor analysis predicted from vendor's embedded Prop: entry, zero Dropbox crash/ANR/tombstone entries, verifiedbootstate=orange/vbmeta.device_state=unlocked unchanged from this device's existing posture.
Separately traced (not left unexplained): today's signing used the signing host's vault-generated key (cd7b74b3fb5c84cfc518ac961cc066d6473a4722) rather than the alpha key this device has booted with since ADR-030 (265ade399a0f3ee65003e56daef26a8f4d70e6a4) — confirmed via key-file mtimes (a .orig-encrypted-backup written 19 seconds before the "current" avb.pem) that this is ADR-041's documented deliberate key-bundle generation, not drift or an accidental swap; confirmed the two candidate fingerprints via AVB's native .avbpubkey struct format after an initial generic-OpenSSL-DER comparison produced a third, meaningless value.
3Consequences
The physical lynx device boots clean on vbmeta-ota-server-signed-v2.img, now signed with the signing host's vault key rather than the alpha key — the first time that key has been used for a direct device flash rather than the OTA pipeline ADR-041 generated it for.
This is confirmed non-blocking only because the device remains unlocked (verifiedbootstate=orange); AVB does not enforce embedded-key identity on an unlocked device, only signature and descriptor self-consistency.
Five new durable findings recorded in docs/research/build-and-signing-engineering-notes.md, independent of this specific incident: a full (not just targeted) build's non-reproducibility blast radius across functionally-unrelated partitions; sourcing a partial re-sign's "unchanged" partitions from a device's actual last-flashed state rather than the most recent or best-documented named candidate; an unopened the signing host vault's misleading ML-DSA is not supported by the system openssl error signature (confirmed twice, not a one-off); avbtool verify_image's undocumented same-directory partition-name resolution; and computing AVB key fingerprints only via the native .avbpubkey format.
4Review when
Before any future ad-hoc vbmeta re-signing that reuses "unchanged" partitions from a named candidate: re-derive the reference from the device's actual last real flash, not the most recent or best-documented candidate. Before lynx bootloader relock or avb_custom_key enrollment (support-policy step 6 / LYNX-001) is proposed: resolve which key (alpha or the signing host's vault key) is meant to be the device's actual enrolled root of trust going forward — today's non-blocking result relies entirely on the device staying unlocked, and a locked device would enforce exactly the key-identity check that was found to differ here.
Read from docs/governance/decision-log.md, entry ADR-046.