A real build has real bugs. Here are the ones worth telling — not because they were dramatic, but because of what they kept teaching the same lesson about, over and over: the gap between believing something works and actually verifying it does.
Early on, the prediction ledger had a standard SQL UNIQUE constraint meant to stop the same prediction from ever being written twice. It looked correct in every code review. It wasn't. SQL treats NULL as never equal to anything — including another NULL. Since every market has at least one nullable subject column (a team-level prediction has no player, a player-level prediction has no team), the constraint silently never deduplicated anything, for any market, from day one. The fix was an expression-based index using COALESCE instead. The real lesson wasn't about SQL syntax — it was that a constraint existing in the schema isn't the same thing as a constraint actually working.
The first time Petey answered a real question about a market with a 0% hit rate, it said: “we didn't score any goals in the tournament.” Nothing was scored, literally — it read a stated confidence number as a count of real-world events. The fix was a clearer prompt explaining what “hit rate” actually means. That held for exactly one more bug: even with careful instructions not to editorialize, Petey kept describing neutral numbers as “struggled,” “moderate,” “steady” — none of them grounded in anything it was actually given. Asking the model more politely didn't hold. What did: a structural filter that throws away any answer containing an unsupported judgment word and falls back to a plain, deterministic sentence instead. Safety through architecture, not through a better-worded request.
Petey worked perfectly in every local test. It failed completely the moment it went live. The reason: the frontend's API calls were pointed at Kubernetes' internal service name for the backend — a hostname that only resolves inside the cluster's own network. A real browser, out on the public internet, has no way to look that up. Every server-rendered page worked fine, because those requests run from inside a pod. Only the browser-side requests broke. The fix was routing those calls through small Next.js API routes that run server-side and proxy to the real backend — the same pattern every other working page already used, just not yet applied to the one page built differently.
An early commit had a live database password and a third-party API key hardcoded directly into the nightly automation script. The fix looked complete: both were pulled out into environment variables, the script now fails fast with a clear message if either is missing instead of quietly running with a broken connection string, and .gitignore was extended so it can't happen the same way again. What the fix commit didn't do — and said so explicitly in its own message, as a separate follow-up step — was purge the old values from git history. They sat there, in plain text, in a commit reachable from the public GitHub mirror, until both credentials were rotated. A commit that removes a secret only stops it from being read going forward; every commit before it still has it, for as long as the repository does. The two are different problems with different fixes, and only one of them was actually a code change.
This is the one worth sitting with. Two separate fixes were built, verified in isolation, and genuinely believed to be live: a corners/shots statement rewrite so predictions read clearly per team instead of ambiguously, and an out-of-fold isotonic calibration layer built specifically to correct a confirmed overconfidence problem in the corners model. Both were reported, at the time, as done. Neither one had actually been wired into the script the real nightly pipeline runs. The calibration code sat in a validated, working training script for weeks while live predictions kept shipping on raw, uncorrected probabilities. Nobody caught it because nothing was wrong in an obviously visible way — the predictions still looked reasonable, the site still worked, the commit messages read like a finished feature. It took directly grepping the actual file the CronJob executes, rather than trusting memory or a past summary, to find out. That's the real habit this project reinforced: the only thing that confirms a fix works is checking the code that actually runs in production — not the code that was written, not what a commit message claims, not what was believed the last time someone looked.
The nightly pipeline is three stages — ingest, build features, generate the slate — run as separate scheduled jobs, each one just trusting that the last one was done in time. For weeks, the feature-rebuild step silently fell behind the raw stats it was supposed to be built from, and nobody noticed, because ingestion itself happened to be paused for an unrelated reason at the same time — so the gap never actually showed up as wrong-looking output. It had to be caught by hand, twice, before the real cause was clear: a rebuild script that was correctly written and correctly documented as needing to run after every ingestion batch, wired into nothing that actually called it that way. Timing that happens to line up isn't the same thing as a dependency that's actually enforced.
Petey's judgment-word filter — the one built to catch the model editorializing with words like “struggled” or “steady” — used plain substring matching. It worked, until a real stat category came up whose name happened to contain a banned word as a substring: “yellow” (cards) contains “low,” and “allowed” does too. Every answer that legitimately used either word got silently rerouted to the generic fallback sentence instead of Ollama's real phrasing — not broken, exactly, just quietly worse, for exactly the questions that should have worked best. The fix was switching to word-boundary matching. The lesson was narrower than it sounds: a safety filter is itself code, and code that filters on substrings will eventually flag the wrong substring.
A column rename landed cleanly in the code, and the migration ran cleanly against the live database. The site went down anyway. The database change was real and correct; the running API and frontend containers were still the old images, built before the rename, still expecting the old column names. A migration being correct says nothing about whether the thing actually serving traffic has been rebuilt to match it — those are two separate steps, and only one of them had happened.
The custom xG model shipped for EPL, Serie A, and La Liga — the leagues with real shot-level data behind them. MLS, the league this whole exercise was partly meant to stop depending on a third party for, isn't in it. Not because building the model for MLS is hard — the code doesn't care which league's rows it trains on — but because the shots themselves don't exist anywhere accessible. Understat, the only source already wired into this project with real shot coordinates, has never covered MLS.
So the actual question became: is there any other real source for MLS shot coordinates, free or otherwise? Checked directly rather than assumed. Every provider that publishes aggregate xG at an affordable tier — the same API already used for fixtures and odds, plus two of its closest alternatives — turned out to publish only that: a single number per shot or per match, never the underlying coordinates a model can actually train on. The one real exception is StatsBomb's free, open-data GitHub release, which does include genuine MLS shot data (2023 season) with defender positions at the moment of each shot — richer than what this project has for any league right now. But it's a one-time historical release, not a live feed, so it can prove the model works on MLS shots without being able to generate a single live prediction from it. Actual live shot-coordinate data, for any league, turned out to be gated behind enterprise sales contacts industry-wide — not a tier this project is going to reach for a portfolio site.
Shipping the model for three leagues and explicitly not for a fourth felt like the more honest move than either quietly dropping the idea or holding the whole feature back on a gap that isn't closing anytime soon. The absence is the finding: sometimes the real blocker on a feature isn't the code, it's that the data it needs was never free to begin with.
None of these were caught by getting smarter about writing bugs. They were caught by building the discipline to keep checking — against real data, real running code, real production behavior — instead of trusting that something already confirmed once will still be true later.
How it works →