We fixed checkout in production. The generator was still holding the bug.
A guest-checkout fix that never reached the build engine would have re-broken four storefronts on the next rebuild. Configuration drift, and fixes that repair themselves.
In June we found that guests couldn't buy. The add-to-cart call on a set of storefronts used an API path whose cart isn't bridged to the guest session, so anyone who hadn't logged in was sent to an empty basket with a straight face. We patched it live, in two dozen places, across every market. Verified, closed, moved on.
The handover note had one line left open at the bottom: update the build engine so a rebuild can't reintroduce this. Nobody did it. In August I rebuilt those shop pages — and the generator handed me the broken call back, unchanged, ready to ship.
The fix existed in exactly one place, and it wasn't the source
Had I not read that line before deploying, the rebuild would have re-broken guest checkout on four storefronts at once. Silently, and only for guests — the segment least likely to write in and tell you, because they haven't got an account, a relationship, or a reason to try twice.
This failure has a name, and it's older than any of the tooling involved. Martin Fowler called it the snowflake server: a production system made unreproducible by accumulated manual changes, where the reasons behind each change have quietly left the building. The general condition is configuration drift, and the whole point of describing infrastructure as code is that the description and the reality can't diverge without somebody noticing.
Everyone accepts this for servers. Almost nobody applies it to a marketing site, because the site doesn't feel like infrastructure. It feels like pages. But the moment any part of it is generated, you have two versions of the truth — the one that's running, and the one that regenerates it — and the second one wins on the day you rebuild.
Every manual fix in production is a loan. The rebuild is when it comes due, and you don't get to choose the date.
The same week, the same shape, from the opposite direction
A day later I found every product category page listing the entire catalogue instead of that category. All four markets, for months.
The cause was not a bug anybody wrote. Someone had deliberately swapped the category loop for a generic product widget, and at the time that was correct: the old shop page filtered products in the browser, so it genuinely needed every product present in the page at once. Perfectly good decision.
Then the shop page got rebuilt to render server-side, and the reason evaporated. The workaround didn't. It sat there, outliving its own justification, doing damage to every category page and to the crawlable structure of the site. A deliberate exception nobody revisited is indistinguishable from a defect — it just has a better origin story.
So don't fix it. Make it repair itself.
Both fixes went in the same way, and it's the part I'd argue for hardest.
The build script now forces the correct loop widget back on every single run, preserving the element id and styling, and prints a warning if it had drifted since last time. It isn't a one-off correction; it's a rule that reasserts itself, so the only way to reintroduce the bug is to remove the rule on purpose. The checkout path moved into the generator the same way. This is idempotency doing what it's for: run it as often as you like, land in the same correct state every time.
Then a separate verifier, because a build script marking its own homework isn't evidence. It fetches every non-empty category anonymously and compares the products actually rendered against the count the commerce API reports, and it checks that the shop page still has its grid. Anonymously matters — logged in as an administrator, sessions and caches behave differently and you see a site your customers don't. I confirmed the checkout fix the same way, with a fresh cookie jar: session created, cart populated, totals correct. Test as the customer, not as yourself, or you're reading a green light that's measuring the wrong thing.
The uncomfortable part
None of this is exotic knowledge. It's the discipline of asking, every time something gets fixed live: what generates this, and does the fix exist there too? The answer is usually no, and the gap is usually invisible for months — until a redesign, a migration, or a new agency triggers the rebuild that collects the debt. Which is really the same question as knowing which machine you're standing on: it's cheap to fix the thing in front of you and expensive to be wrong about which thing that is.
A production fix that doesn't reach the code that produces production isn't a fix. It's a delay with good paperwork.
Sources & further reading
External: Snowflake Server, Martin Fowler · Phoenix Server, Martin Fowler · Infrastructure as Code, Martin Fowler
Related posts: The same task three days running · The biggest leak in your mobile checkout · Everything returned 200 · The hard part isn't Git