Your benchmark is a training set
I read a post this week about how cheap performance work has become now that you can hand the grinding parts to a coding agent. The demonstration was a regex engine: the author had an agent add an ahead-of-time compilation path, then measured what it bought. What caught me wasn’t the speedup. It was that he reported two of them. Two to four times faster on the long queries he’d been optimising against, and about seven percent on a separate set of representative queries he’d deliberately held back. He put both numbers on the page and moved on, as if reporting two was the obvious thing to do.
It stopped me because I know exactly which number I would have quoted. The two-to-four. It’s the one I made happen, the one I watched go green, the one that would go in the commit message. The seven percent is the quieter, more embarrassing figure, and for most of my career I wouldn’t have generated it at all — not out of dishonesty, but because it never occurred to me that the queries I’d tuned against and the queries I actually cared about were two different populations, and that only one of them was telling me the truth.
The optimisation that flew on the benchmark
I have shipped a change that was wonderful on the benchmark and did almost nothing in production. More than once. You pick a workload to optimise against — a set of representative inputs, a corpus of queries, whatever you can get to run repeatably — and you tune against it, because you need something stable to measure. And the number climbs, and you feel the satisfaction of the number climbing, and then the change goes out and the graphs that matter barely move. The tuning latched onto something specific to the inputs I’d chosen: an access pattern the cache happened to love, a branch the real traffic takes far less often, a shape of data that my sample over-represented. I’d optimised the benchmark. I had not, it turned out, optimised the thing the benchmark was standing in for.
Every engineer who has done this knows the feeling and most of us know the guard against it, at least in principle: don’t tune against your production traffic, load-test against a sample of it and keep the real distribution in view, because a system tuned too closely to one workload gets brittle on the others. I knew that. I just filed it as a fact about caches and query planners, a piece of performance folklore, rather than an instance of anything more general.
The queries you tuned are a training set
Here is the general thing, and it took me longer than I’d like to admit to see that it was the same thing. The set of queries you optimise against is a training set. Your performance on it is in-sample performance — how well you do on the exact examples you got to look at while you were tuning. The held-back queries are a test set, and your performance on them is the only figure that estimates what happens on inputs you haven’t seen. A model that scores brilliantly on its training data and mediocrely on fresh data isn’t doing anything exotic; it has memorised the particulars of what it was shown instead of learning the pattern underneath. The name for the gap between those two scores is overfitting, and I resisted the word for a while because it sounded like a machine-learning problem, something that happened to other people’s neural networks. It is not a machine-learning problem. It is what my brittle cache optimisation had been the whole time.
I put the author’s two numbers next to each other and sat with the arithmetic longer than I expected to. Two-to-four on the training queries, seven percent on the held-back ones. I had been reading numbers like the first as the result and the effort as finished. What the pairing made unavoidable is that the first number was never an estimate of anything I cared about. It was a measure of how well the change fit the examples I’d chosen to fit. The seven percent was the result. The other figure was the sound of the optimisation learning my benchmark by heart.
The gap is the measurement
The reflex, when the holdout number comes in lower, is to treat the drop as loss — the disappointing haircut you take off the good number, noise eating into the real gain. It’s the opposite. The distance between in-sample and out-of-sample is not noise; it is the most informative quantity in the whole exercise. A small gap says the speedup is general, that it travels to inputs you didn’t tune on, that you found something real about the workload. A large gap says most of what you measured was specific to the queries in front of you and won’t survive contact with the rest. You cannot read either of those things off the training number alone. You can only read them off the gap, which is why reporting one number and not two isn’t just incomplete — it’s withholding the single figure that tells you whether you learned anything.
This is a different discipline from the one I’ve written about before, and I want to be careful not to blur them, because I blurred them for years. That earlier instinct — run it more than once, watch the spread, don’t trust a single measurement — is about variance, about a number being unstable across repeats of the same workload. This one is about a number being unrepresentative across different workloads, however stable it is on each. You can have a speedup that’s rock-steady on ten reruns of your benchmark and still evaporates on real traffic. Steadiness on the training set is no defence against overfitting to it. I had internalised the first lesson and mistaken it for the whole of the second, and they are not the same lesson. One says run the benchmark again. The other says the benchmark was never the point.
The holdout only works once
The awkward part, the part I haven’t made peace with, is that a held-back set is only honest until you use it. The moment the seven percent comes in low and I go back, look at which held-back queries regressed, and tune to recover them, those queries have quietly joined the training set. I’ve looked at the answers; I can’t unlook. The next measurement on them is in-sample again, dressed as a test, and the reassuring number it produces is measuring my ability to teach to the test rather than anything that generalises. A clean holdout is a thing you get to spend exactly once, and every glance you take at it to decide what to try next spends a little more of it.
So the discipline isn’t really the split. It’s the restraint not to peek — to keep a set of inputs I have genuinely never optimised against, and to accept that the only trustworthy number is the one I read off it the first time, before I knew what it would say. I am not good at that yet. I catch myself wanting to consult the holdout the way you’d sneak a look at the back of the book, to steer by it, and every time I do the honest number I was protecting gets a little less honest. I’ve started keeping a sample I refuse to look at until the end. I have not yet managed to look at it only once.
Part of an occasional series reframing everyday engineering through a data scientist’s eyes. The ideas here are developed properly in Thinking in Uncertainty and Building with Certainty.