Bryan O’Neill, CTO of FormAssembly, wrote a piece tracing three decades of the same mistake:
– In the 1990s, some companies paid engineers per line of code, and got bloated, unmaintainable software for it.
– In the 2000s, agile teams adopted story points, and learned to inflate estimates so the burndown chart looked healthy.
– Now it is tokens: engineers competing on how much AI they can consume, as though volume of generation were the same thing as work worth doing.
O’Neill is right about all of that. Effort keeps getting measured instead of value, under a new name each decade. I’ve made a version of this argument before, about the internal leaderboards where engineers compete on token spend the same way they used to compete on lines shipped.
Where I think his piece goes wrong is the suggested fix.
The upstream move
O’Neill’s answer is spec-driven development: write a detailed specification first, then let AI generate the code against it. The logic is that if intent is nailed down before anything gets generated, nobody can tokenmax their way to a bad outcome. It sounds like progress: it relocates the point of measurement from output back to intent.
But intent, the moment you decide to measure it, becomes exactly as gameable as output was. A team under pressure to demonstrate spec-driven development will write specs that look thorough: exhaustive edge cases, generous acceptance criteria, more words per requirement. None of that guarantees the spec describes a problem worth solving. It only guarantees the spec is long.
The pattern repeats
This is the part O’Neill doesn’t seem to notice he’s repeating. Every fix for a bad proxy metric is itself a new proxy metric. Lines of code stood in for did work happen?. Story points stood in for how much work happened?. Tokens stand in for how much effort went in?. A spec stands in for did we think it through?. All four are answerable without ever touching the only question that matters: was this worth building at all?.
Goodhart’s Law covers exactly this, and I’ve applied it to token spend already. When a measure becomes a target, it stops being a good measure. O’Neill’s fix assumes the law applies to the metric he’s replacing and not the one he’s proposing. It applies to whichever one a team currently trusts, which is precisely the one nobody thinks to question.
David Pereira makes a related argument about estimation itself: story points don’t make plans more accurate, they just give Parkinson’s Law a number to expand into. He puts it plainly: teams keep sizing work because the ritual feels like rigour, not because the number improves the plan. The estimate becomes the artefact everyone manages toward, and the value it was supposed to protect becomes an afterthought.
John Cutler makes the wider point, in a piece about treating software as a portfolio of assets and liabilities rather than a single throughput number: AI does not eliminate complexity, coordination, or uncertainty, it changes the cost of responding to them. A better proxy doesn’t eliminate the gap between effort and value either. It changes how expensive the gap is to notice, and a longer spec makes that gap more expensive to notice than a token count did, because a spec looks like diligence.
I’ve seen a smaller version of this play out in product teams that were shipping too fast without validating anything. The fix was a definition-of-done checklist: user research done, success metric named, rollback plan written. Sensible, on paper. Within two quarters, the checklist was getting completed on every ticket, and the tickets were exactly as unvalidated as before. Reviewers had learned to write a plausible success metric the same afternoon they wrote the code, which satisfied the checklist without ever being checked against reality. The checklist was a spec by another name, and it failed for the same reason O’Neill’s fix will.
What a spec is actually good for
None of this makes specs useless. A clear brief beats a vague one, whatever writes the code afterward. That is not my argument.
My argument is that writing one, however detailed, doesn’t exempt a team from the question that actually matters: whether what they are about to build is worth building, for anyone who might pay for it. A beautifully specified feature nobody adopts fails exactly the way ten thousand tokens used to build it do. Specification quality and token count both measure effort a team can point to. Neither measures whether that effort was worth spending.
The check that has no leaderboard
You do not get out from under Goodhart’s Law by inventing a smarter target. You get out from under it by checking, directly and often, whether the thing you shipped changed anything for the person who needed it: did they adopt it, did their behaviour change, did it produce value someone would pay for again.
That check does not fit on a leaderboard. It cannot be inflated by writing more words into a document or spending more tokens on a generation run. It is also the only one of the four proxies in this piece’s history that has ever actually worked, and the reason it keeps getting replaced by something easier to measure is that it is, by a wide margin, the hardest one to fake.
Leave a Reply