Lessons Learned
A real build has real bugs. Here are the ones worth telling — not because they were dramatic, but because of what they kept teaching the same lesson about, over and over: the gap between believing something works and actually verifying it does.
A uniqueness constraint that never enforced uniqueness
Early on, the prediction ledger had a standard SQL UNIQUE constraint meant to stop the same prediction from ever being written twice. It looked correct in every code review. It wasn't. SQL treats NULL as never equal to anything — including another NULL. Since every market has at least one nullable subject column (a team-level prediction has no player, a player-level prediction has no team), the constraint silently never deduplicated anything, for any market, from day one. The fix was an expression-based index using COALESCE instead. The real lesson wasn't about SQL syntax — it was that a constraint existing in the schema isn't the same thing as a constraint actually working.
Petey reads a percentage as a body count
The first time Petey answered a real question about a market with a 0% hit rate, it said: “we didn't score any goals in the tournament.” Nothing was scored, literally — it read a stated confidence number as a count of real-world events. The fix was a clearer prompt explaining what “hit rate” actually means. That held for exactly one more bug: even with careful instructions not to editorialize, Petey kept describing neutral numbers as “struggled,” “moderate,” “steady” — none of them grounded in anything it was actually given. Asking the model more politely didn't hold. What did: a structural filter that throws away any answer containing an unsupported judgment word and falls back to a plain, deterministic sentence instead. Safety through architecture, not through a better-worded request.
A hostname that only existed inside the cluster
Petey worked perfectly in every local test. It failed completely the moment it went live. The reason: the frontend's API calls were pointed at Kubernetes' internal service name for the backend — a hostname that only resolves inside the cluster's own network. A real browser, out on the public internet, has no way to look that up. Every server-rendered page worked fine, because those requests run from inside a pod. Only the browser-side requests broke. The fix was routing those calls through small Next.js API routes that run server-side and proxy to the real backend — the same pattern every other working page already used, just not yet applied to the one page built differently.
Fixed, tested, shipped — except it wasn't
This is the one worth sitting with. Two separate fixes were built, verified in isolation, and genuinely believed to be live: a corners/shots statement rewrite so predictions read clearly per team instead of ambiguously, and an out-of-fold isotonic calibration layer built specifically to correct a confirmed overconfidence problem in the corners model. Both were reported, at the time, as done. Neither one had actually been wired into the script the real nightly pipeline runs. The calibration code sat in a validated, working training script for weeks while live predictions kept shipping on raw, uncorrected probabilities. Nobody caught it because nothing was wrong in an obviously visible way — the predictions still looked reasonable, the site still worked, the commit messages read like a finished feature. It took directly grepping the actual file the CronJob executes, rather than trusting memory or a past summary, to find out. That's the real habit this project reinforced: the only thing that confirms a fix works is checking the code that actually runs in production — not the code that was written, not what a commit message claims, not what was believed the last time someone looked.
None of these were caught by getting smarter about writing bugs. They were caught by building the discipline to keep checking — against real data, real running code, real production behavior — instead of trusting that something already confirmed once will still be true later.
How it works →