You would not run an A/B test where you choose the success metric after seeing which variant won. The result would be meaningless and everyone would know it. But somehow, this seems to be the standard practice for AI features.
An AI evaluation harness bolted on after the demo is not a safeguard. It is a rationalisation with a dashboard.
The sequence usually goes like this. A team builds an agent. It demos well. Someone asks how it will be measured, and an evaluation set gets assembled afterwards, mostly from examples the system already handles because those are the ones people have in front of them. The dashboard comes back green. It ships. Nobody set out to game anything; the order of operations did it for them.
Why the order matters
Mark Ajzenstadt put the rule in one line: you cannot trust a measurement you build after seeing the answers. Hindsight makes every borderline case look like it should obviously pass. You write the evaluation to match the behaviour you have, and the behaviour you have becomes the definition of correct. The eval stops being a test and becomes a mirror.
Build it first and it does the opposite. Before the agent exists, you write down the workflow it serves, the outcome that actually matters to the business, the cases it must get right, the cases it must refuse, and the cost and latency you are willing to pay. Then you build toward that bar. The measurement constrains the system instead of flattering it.
A common case: the support-triage assistant. The team builds it, points it at a few weeks of resolved tickets, and reports high accuracy. The tickets it was tested on are the ones a human already found easy enough to close. The cases that actually cost the support team time, the ambiguous ones and the ones that should have been escalated, were underrepresented in the test set because they were underrepresented in the pile of things marked resolved. The eval measured the assistant on the work that did not need help.
What building it first buys you
A read on whether the thing works that you can actually rely on, because the bar was set by someone who could not see the answers yet.
A way to compare model swaps without reopening the argument every time. When a new model ships, you run it against the same harness and you get a number. Without that, every upgrade turns into a fresh round of vibes and demos.
A definition of done that is not “the demo was convincing.” Convincing demos are cheap now. They were always a poor signal and they are a worse one when a plausible-looking result takes an afternoon to produce.
The Asana detail people skipped
When Asana cleared its Enzyme migration in two weeks with Codex, the prompt was five sentences. That part got the attention. The load-bearing artefact was the way the team could tell, continuously, whether the migration was actually complete and correct across thousands of files. Instructing the agents was the easy half. Knowing when they were done, and being able to trust that judgement, was the work.
That generalises. For most agentic projects the prompt is not where the difficulty lives. The difficulty lives in having a trustworthy signal for whether the output is right, produced by something other than a person eyeballing a sample.
“We don’t know what good looks like yet”
This is the usual objection, and it is often true. You do not yet know exactly how to judge the output.
That is the first piece of work, not a reason to defer it. Building the system before you have defined good does not teach you what good is. It gives you a plausible artefact and a strong incentive to rationalise whatever it produces. Sit with the workflow, talk to the people who do it today, and write down what a correct result looks like and what an unacceptable one looks like. If you cannot do that, you are not ready to build, and no amount of building will get you readier.
A short checklist
- Write the evaluation before the build starts, not after the first demo.
- Include refusal cases: the inputs where the correct behaviour is to decline or escalate.
- Set the cost and latency ceiling up front, and treat breaching it as a failure, not a footnote.
- Keep a holdout set the builders never see, so you always have an uncontaminated read.
- Give ownership of the evaluation to someone other than the team building the system, so the bar is not quietly renegotiated every time the build falls short of it.
None of this is elaborate. It is a page of writing and a bit of discipline about sequence. The reason it is rare is not difficulty. It is that writing the evaluation first means confronting, before anyone has built anything, that you might not actually know what a good outcome looks like. Skipping that step feels like progress. It is deferral with a demo attached.
If you can only tell whether it worked after you have seen what it did, you did not measure anything. You reacted.
Leave a Reply