1Decision
Deploy the private lynx OTA update server (docs/superpowers/plans/2026-08-23-lynx-ota-update-server.md, docs/superpowers/specs/2026-08-23-lynx-ota-update-server-design.md, Tasks 1-5 of 9), with two real deviations from the approved design: manual certbot --manual TLS issuance instead of Caddy's native tls { dns vercel ... } ACME automation, and a broadened a private /16 range firewall allow-range instead of the design's single /24.
2Why and evidence
Caddy's built-in DNS-01 automation could not be made to work end to end. Root causes, in order found: (1) github.com/caddy-dns/vercel has had exactly one release ever (v0.0.2, 2021-08-08) and does not compile against current libdns/libdns (confirmed real compile errors, and confirmed this is why caddyserver.com/api/download's own build-from-source also 400s for this plugin specifically — the bare download endpoint with no plugin returns 200).
Forcing libdns/vercel@v0.1.0 fixes the compile. (2) Neither library ever sends the teamId/slug query parameter Vercel's own REST API documents as required for a Team-owned zone (https://vercel.com/docs/rest-api/dns/create-a-dns-record) — every write from a Team-scoped token failed with a content-free Forbidden (403) (the library also silently discards Vercel's real JSON error body, compounding the diagnosis difficulty).
A local patch adding a TeamID field/team_id Caddyfile subdirective, tracked in this repo at tools/ota-server/vercel-dns-patch/ (see docs/research/ota-server-caddy-vercel-notes.md for the full investigation), fixes both the missing parameter and the swallowed error detail. (3) Even with teamId correctly wired (confirmed: the error changed from a bare 403 to a real authorization denial naming the exact team/domain/action, i.e.
Vercel resolved the request correctly and then still refused it), a Team-scope token held by that team's Owner, with no granular permission step ever offered at token creation, was still denied — {"message":"You don't have permission to create the domain record."} — while the same record type succeeded immediately via the Vercel dashboard UI under an identical login.
This API-vs-dashboard gap remains unexplained; ruled out during investigation: wrong team scope, domain ownership, account role, and (separately) a Full Account-scope token, which fails differently ("invalidToken":true) rather than better.
Given the UI path is proven reliable, the server's real TLS certificate was instead obtained via one manual certbot certonly --manual --preferred-challenges dns run against real production Let's Encrypt, with the operator pasting the _acme-challenge TXT record into the Vercel dashboard by hand when certbot paused (confirmed working).
Separately, the firewall CIDR was broadened after discovering the design's assumed single /24 (the build host's own internal VM-bridge subnet) is not actually where client devices live: a workstation VPN session got an address in another /24 and the physical lynx device (over the same home OpenVPN server, different pool) got an address in yet another pool — both would have been silently dropped by the original single-/24 rule.
a private /16 range was chosen because the design's own stated real security boundary is the home router's absence of port-forwarding (DNS-01 needs no inbound reachability at all), making this nftables rule "defense in depth" rather than the primary control, and all private RFC1918 space is equally non-routable from the public internet regardless of which specific /24 a client happens to land in.
3Consequences
Certificate expires 2026-11-22 with no auto-renewal hook (--manual mode requires --manual-auth-hook for that, not configured); re-running the same manual certbot command before then is a required recurring operational task, not a one-time step, until/unless the Vercel API issue is resolved and native DNS-01 automation is retried. The patched Vercel Go modules are preserved in-repo specifically so that retry doesn't require redoing this investigation.
Two unrelated real bugs were also found and fixed in the same session: caddy run --environ in the systemd unit leaked VERCEL_API_TOKEN into the journal in plaintext on every start (removed; the leaked token was rotated); and file_server browse=false is not valid Caddyfile syntax (browse is a bare flag, present-or-absent, not =value) and was a hard parse error, fixed to plain file_server.
4Review when
Revisit the certbot/Caddy split before 2026-11-22 (cert renewal due) or sooner if Vercel support clarifies the API-vs-dashboard permission gap, at which point retrying native tls { dns vercel { team_id ... } } with the tracked patch would restore real automation. Revisit the a private /16 range range if a legitimate client ever needs to reach the server from outside that block (e.g. a different VPN pool), or narrow it back if the home network's actual subnet layout becomes fully known and stable (see the open question raised in this same session about the physical device's real at-home WiFi subnet, never confirmed live).
Read from docs/governance/decision-log.md, entry ADR-045.