The deploy artefact that had never once built

task backend:build:linux had never worked. Not once, since the day it was written.

Nothing noticed, because nobody had run it. Every developer build is a darwin build, and a darwin build takes a different path that succeeds. The task that produces the thing we would actually deploy had been quietly broken for as long as it had existed.

The failure

Building the tenant with GOOS=linux CGO_ENABLED=0 failed on undefined symbols.

The cause was three layers down. Litestream, which replicates the sqlite database to object storage, registered a driver in its init() against a cgo sqlite implementation. So the whole binary needed cgo, and CGO_ENABLED=0 could not link.

Why it was invisible

A darwin build defaults CGO_ENABLED=1. So on every machine anybody develops on, the dependency’s cgo requirement is satisfied, the link succeeds, the tests pass, and the problem does not exist.

Only cross-compiling exposes it. And the only task that cross-compiles was the one nobody ran until it was time to build a container.

This is the shape of failure worth naming: two halves that are each individually correct, producing a broken whole that no single test observes. The dependency is not wrong to want cgo. The Mac is not wrong to enable it. The task is not wrong to disable it. Every component behaves as designed, and the artefact does not exist.

The quieter problem underneath

The build failure was a morning. What it had been hiding was worse.

The design chose a pure-Go sqlite driver specifically so the tenant binary would be CGO-free and cross-compile cleanly. That is a written decision with a written rationale. And a transitive dependency had silently voided it — the build had not been CGO-free since the day litestream was added, and the decision that justified the driver choice had been false the entire time.

A design decision can be undone by a dependency’s init() function. Nothing warns you. The document still says the thing it said.

The fix

Litestream had moved to the same pure-Go sqlite implementation in a later major version, so the fix was an upgrade rather than a workaround. The adapter needed rewriting — one replica per database rather than a slice, Close taking a context, and no more hand-rolled restore-if-absent branch because upstream now has one.

Then the round trip that matters: replicate, destroy the disk, restore, and prove the data came back. Green. And the binary cross-compiles CGO-free, which is what the design said it would from the beginning.

Two things we changed about how we work

Cross-compile in CI, always. Not because we deploy from CI — because the local build takes a different path and will therefore never tell you the truth about the artefact.

Verify a build produced what you asked for, not just that it exited zero. The related guard we added for the frontend bundles reads the finished output back and checks it, because a build-time override is invisible in source. The first version of that guard was itself broken: a glob matching nothing makes grep -q fail, which reads as “clean”. It now proves it found something to check before it reports success.

A check that cannot fail is not a check. A check that reports success when it found nothing to look at is worse, because it also tells you it did its job.