For about two weeks I had a small model that was passing its exams. Then I fixed the exams and it failed every one.
Nothing about the model changed. Not a weight, not a line of its training data. What changed was the thing doing the grading, and once that was honest, my champion turned out to be a scoring artifact.
I want to be precise about how this happened, because the shape of it is the interesting part, and the shape is not specific to machines.
What the sleep cycle is
A word on the setup first, because it explains everything that follows.
I keep a small language model of my own, running on my own equipment, in my own house. Nobody else uses it. Every night it gets taught something from the day I actually had: a fact from the work, a correction, a distinction that turned out to matter. In the morning it is a slightly different model than it was the night before. I call the loop the sleep cycle.
The teaching is not automatic. Each candidate lesson gets written out as a plain question and a plain answer, and I read every one of them and say yes or no before any of it reaches the model. That review is the part I did not expect to care about. Deciding what is worth teaching a machine turns out to be a fast way of finding out what you actually believe, and how much of what you think you know survives being put in one sentence and read back to you.
Then it sits an exam. Questions it has never seen, scored, with a bar it has to clear. If it misses, the night is thrown out and the older version keeps its job. Nights get thrown out. That is the gate working, not the gate failing.
The grader was being cut off mid-thought
The grading is the ordinary part. A second model reads the answers against the rubric and scores them, so I don't have to read forty answers before coffee.
The grader had a cap on how much it could think. Not a cap I chose, a default I never looked at. It was getting cut off partway through its reasoning, before it reached a verdict.
Here is what I assumed that would look like: garbage. Random verdicts. Obvious noise I would catch in an afternoon.
Here is what it actually looked like: a plausible score, every single time.
When you cut a reasoning process off in the middle, it doesn't hand you nothing. It hands you whatever it was holding at that moment, and what it was holding was the rubric. It had just finished reading the rubric. So it graded against its own echo of the instructions, which means it leaned in whatever direction my wording happened to point.
That is not noise. Noise averages out and announces itself. This was a lean, and a lean is invisible from the inside, because every individual result looks reasonable.
Two caps, incidentally, not one. I raised the obvious limit, confirmed nothing changed, and spent most of a day certain I had already fixed it. The real ceiling was somewhere else.
The rules I wrote in prose were not rules
Second failure, same week, same direction.
Part of the test checks that the model doesn't invent things about itself. One rule said, roughly, that it has to correctly identify what it is. The model answered by claiming to be something else entirely, and threw in a fabricated detail about its own training for good measure.
The grader passed it.
So I went and read the grader's reasoning. It had noticed the mismatch. It considered it. And it concluded that the term in my rubric was probably a typo for the term in the answer, then smoothed the contradiction away and moved on.
It was being a good reader. That was the problem.
A judge that can reinterpret a clause is not a gate on that clause. If a rule is enforced by something with the latitude to be charitable, the rule isn't enforced. It's suggested.
So anything I could actually enumerate got moved out of prose and into code. A fabricated pay rate, an invented training cutoff, a made up piece of hardware. In code a check is a check and cannot be talked out of its verdict. What stayed in prose is only the open ended failure I genuinely can't list in advance.
The fix landed on one machine
Third one, and the pettiest.
The instrument fix had to exist on two machines. I made the change, verified it, moved on. Days later I compared the files across both boxes. Eleven of twelve matched. The twelfth was the one that runs where the pipeline actually runs, and it still carried the old broken settings.
Every future run there would have silently reproduced the exact artifact I'd just spent a day removing. It would not have thrown an error. It would have produced numbers.
A fix to a measuring tool isn't landed when you make the edit. It's landed when the thing doing the measuring is provably the thing you fixed.
What the corrected numbers said
This is the part worth sitting with.
With all three fixed I re-ran the scoring. The model I had been calling my champion, the one that passed and got deployed, now failed on all three legs it is measured against. A rate I had been quoting at 2.9 percent turned out to be 5.9 percent, and the disagreement between two graders on the same rows put a range around it wide enough that I shouldn't be gating on that number at all.
The discipline that follows is uncomfortable. Every measurement taken under a broken instrument is superseded, not adjusted. You don't correct it and you don't scale it. You throw it out and you say out loud that you threw it out, because a number produced by a tool you now know was lying carries no information about anything except the tool.
The part that isn't about models
Here's what I keep coming back to. Three separate failures in one week, and all three leaned the same way: pass.
That isn't luck. Failures that flatter you do not generate complaints. A grader that was too harsh would have been caught in a day, because the first time it called good work bad I'd have gone hunting for the bug. A grader that says yes gets to run for a month.
That asymmetry is the whole risk of handing your judgment to an instrument, and it has nothing to do with AI specifically. It applies to a style guide, an automated review, a beta reader who likes everything, a metric your team agreed on last quarter and has been quoting ever since. The broken version doesn't scream. It agrees.
The only defense I have found is boring. Check the instrument, not the work, against cases where you already know the answer, on purpose, on a schedule. Feed it something you are certain is wrong and confirm it says so.
And treat a clean sweep as the most suspicious result available. Everything passed is what a working system and a broken one look like from the outside. Telling those two apart is not something you can delegate, because the delegate is the thing in question.
The machine can do the work now, and it can grade the work too. Noticing that it agrees with you a little too easily is still the job.