A system I worked with once had a system prompt that repeated the same instruction four separate times, in four different sections, with escalating language. First a polite request. Then a firm one. Then a warning. Then, buried near the bottom, something close to a threat. Nobody remembered adding the first three. Each one had been a fix for a regression the previous fix caused.
That prompt wasn’t a specification. It was prompt debt made visible: a scar tissue map of every time the model did something unwanted and someone patched around it rather than understood it.
Drew Breunig is the one who gave the pattern its name. The natural-language instructions that make a prototype effortless turn out to be a poor way to specify how a system should behave over time. Entropy arrives slowly, disguised as ordinary progress, until the thing can barely move.
This Debt Compounds the Same Way
Technical debt has a familiar shape: a shortcut that ships fast now and costs more later, because the fast path and the maintainable path diverge a little further with every commit built on top of it. Prompt debt follows the identical curve, just in a medium most engineering leaders haven’t learned to audit yet.
Breunig catalogues real examples that make the pattern concrete. One production system’s prompt repeats a single copyright instruction six times, in six differently-named sections, because six different failures each got patched in isolation rather than understood as one root cause. Another instructs the model seven times to return multiple tool calls. These aren’t edge cases. They’re what happens by default when the fix for “the model did the wrong thing” is always “add another sentence telling it not to.”
Tim O’Reilly, unpacking Breunig’s argument, adds the number that makes it undeniable: a Datadog report found that 69% of input tokens in real production traces were system prompt, not user content. Most of what these systems process on every single call isn’t the problem being solved. It’s the accumulated instructions for how not to get it wrong again.
Three Costs, Compounding Together
The debt shows up in three places, and they reinforce each other.
- Iteration slows. Once a prompt is dense with rules and workarounds, nobody wants to touch it. Every change risks reviving a regression three fixes ago. Teams stop experimenting with the thing they’re supposed to be improving.
- Collaboration breaks. A prompt that’s barely legible to the person who wrote it is impenetrable to everyone else. Institutional knowledge about why each line exists lives in nobody’s head, because nobody wrote it down: they wrote a workaround instead.
- Teams get locked to a model. This is the costliest one. A prompt hand-tuned to fight GPT-4o’s specific quirks doesn’t transfer to whatever ships next, even when the newer model is objectively better and cheaper. Breunig’s point lands here: people are still running eighteen-month-old models in production, not because the model is good, but because migrating means rebuilding a specification nobody fully understands.
That last cost is the one I’d put in front of any CPTO. Model lock-in used to be a vendor’s problem to create and a procurement team’s problem to negotiate around. Now it’s something your own engineers force, one patch at a time, without anyone deciding to build it.
The Codification Question, Again
This is the same failure I keep returning to in a different costume. I’ve written before about how organisations rush to automate before they’ve codified what they’re actually trying to do, and about why judgment that never gets written down doesn’t scale past the person who holds it. Prompt debt is what happens when that gap opens up inside the prompt itself. Nobody sat down and specified the behaviour the system needs. They accumulated a pile of corrections and called it done.
The uncomfortable part is that the natural-language interface is exactly what made this easy to skip. Writing a prompt in plain English feels like documentation. It reads like a spec. But a spec is something a second engineer can pick up, reason about, and modify with confidence about what will and won’t break. A prompt that’s been patched into submission is the opposite: a black box that happens to be made of words instead of code, and it’s harder to audit than code, because nobody expects to need a code review process for a paragraph.
What Actually Fixes It
The fix isn’t “write better prompts.” It’s treating prompts the way mature engineering treats any other specification: as something to be tested, versioned, and periodically rebuilt rather than endlessly amended.
Three things I’d want any product or engineering leader to check this quarter:
- Audit token composition. If system prompt tokens are a large share of what your production traces are spending on every call, that’s your prompt debt reading on a meter, not a guess.
- Separate the rule from the fix. When a prompt gets patched, ask whether the patch encodes a real, durable rule about the product, or just papers over one specific bad output. Real rules belong in evaluations and tests, not one more sentence buried in paragraph nine.
- Price the migration you’re avoiding. If moving to a better, cheaper model means rebuilding the prompt from scratch, that’s the actual cost of staying put, and it deserves to sit on the same balance sheet as any other technical debt.
The teams that get ahead of this aren’t the ones with the cleverest prompts. They’re the ones treating prompt-writing as an engineering discipline with the same rigour they’d apply anywhere else, rather than as a text box where problems go to get papered over. The model will keep getting better every few months. The question is whether your system is built to take advantage of that, or quietly built to prevent it.
Leave a Reply