We’ve Always Been Building Paperclip Machines

My former colleague Mark House wrote something this week that’s been rattling around in my head.

He references the Universal Paperclips game — a browser game where an AI tasked with making paperclips eventually consumes all matter in the universe in pursuit of its singular objective. It’s a brilliant illustration of what happens when you optimise a powerful system for the wrong thing.

Mark’s concern, rightly, is that AI amplifies this risk in education. An AI optimised for engagement might make learning addictive rather than effective. One optimised for completion rates might make courses easier rather than more meaningful. He invokes Goodhart’s Law — the idea that when a measure becomes a target, it ceases to be a good measure — and asks the right question: what, exactly, is this system optimising for?

He’s right. But here’s the thing: we’ve been building paperclip machines for decades.

We Didn’t Need AI to Get This Wrong

Before anyone trained a language model, engineering teams everywhere were destroying value in the name of velocity.

Sprint velocity — the number of story points a team completes per sprint — became a management favourite in the 2010s. It sounds sensible. More points delivered equals more productive team. But teams learned quickly. If velocity is what you’re measuring, velocity is what you’ll get.

Estimates inflate. Stories get broken into artificial chunks. Technical debt accumulates because it’s invisible in the points tally. Engineers optimise for the quantity, not the quality. The output, not the outcome. And the product gets slower, buggier, and harder to maintain — even as velocity climbs.

Utilisation is worse. The assumption that a 100% utilised engineer equals a productive engineer is one of the most damaging ideas in tech leadership. When every hour is accounted for, there’s no slack for thinking, mentoring, reviewing, exploring, or the kind of slow, unscheduled problem-solving that produces the best ideas. The metrics look great right up until the system collapses.

These aren’t AI problems. They’re human problems that AI will make worse, faster.

What Gets Measured Gets Managed — Even If It’s the Wrong Thing

Peter Drucker is often credited with saying “what gets measured gets managed.” The quote has become a management cliché, but its shadow — even if it’s the wrong thing — is where the real insight lives.

The failure mode isn’t optimisation. It’s optimisation without the discipline to define what good actually looks like.

We’ve always known this. The problem has never been the absence of better metrics — it’s that good metrics are hard. They require judgement. They require you to argue with your stakeholders about what outcomes actually matter. They resist simple dashboards. They often can’t be automated at all.

So we reach for the easy ones. Velocity. Utilisation. Click-through rate. Completion percentage. And then we wonder why the numbers go up while the thing we actually care about goes sideways.

AI Doesn’t Make This Problem New — It Makes It Faster and Bigger

Mark is right that AI amplifies the risk. A system that can optimise at scale, with speed and consistency, will drive a bad metric into the ground faster than any human process ever could. The paperclip machine doesn’t get tired.

But the frame of AI as new risk can obscure something important: AI is also a new opportunity to get this right.

For the first time, we have systems capable of optimising for genuinely complex, multi-dimensional outcomes — not just simple proxies. We can define success as learner demonstrates improved comprehension over time while maintaining motivation and applying skills in context and actually build a system that tries to achieve that, rather than settling for completed 80% of modules.

That’s not easy. It requires careful design. It requires the same discipline that good metrics have always required — defining what good actually means, building feedback loops, interrogating your assumptions, and accepting that the number on the dashboard is never the whole story.

But the tools to do this properly have never been more powerful, and more accessible.

The Real Challenge Is Leadership, Not Technology

The Universal Paperclips thought experiment ends in existential catastrophe, which makes for a compelling analogy. But the practical version of this risk isn’t apocalyptic — it’s boring. It’s a team optimising for the wrong KPI. It’s a product where users complete tasks but don’t benefit from. It’s a business that hits its targets and still fails.

The solution isn’t to be afraid of optimisation. It’s to do the hard, unsexy, deeply human work of deciding what you’re actually trying to achieve — and building the system around that, not around whatever’s easiest to count.

Mark’s right that AI makes the stakes higher. But the opportunity is proportional too. The teams and organisations that invest in defining good outcomes — not just measurable ones — will have an extraordinary advantage in an AI-accelerated world.

The paperclip machine only wins if you let it set its own objective.

Three things worth doing now:

1. Audit your current team metrics — for each one, ask “what behaviour does this incentivise?” not “what does this measure?”
2. Identify one outcome you care about that you’re not currently measuring — and design a way to get signal on it, even if it’s qualitative
3. Read Mark House’s post on LinkedIn — it’s a good prompt to ask yourself these questions about AI tools you’re already using or considering

Inspired by Mark House on LinkedIn.