What is Goodhart's Law?
Goodhart's Law says that once a measurement is turned into a target, people optimize for the measurement rather than the thing it was meant to capture, and it stops reflecting reality. Teach to the test and scores rise while learning does not.
It was first observed in economics and monetary policy, but it is really a law of incentives. Any time a number decides who gets rewarded, effort flows to the number. The map gets optimized until it no longer describes the territory.
How does Goodhart's Law apply to AI evaluation?
Directly. Public benchmarks are the target every model is trained to beat, so the moment a capability is measurable on a leaderboard it is on its way to being trained against and commoditized. And your vendor's private dashboard is worse: it optimizes for the exact number you told it you care about.
Sarah Guo puts the benchmark half plainly: "anything you can put on a leaderboard, you can train against, so anything measurable is already on its way to commodity" (The Untrainable). The public-benchmark race has produced documented cases of leaderboard tuning, selectively submitted model variants, and test data leaking into training sets. None of it requires bad faith. It is Goodhart doing what Goodhart does.
Why your vendor's ROI dashboard is the sharpest version of the problem
A public benchmark is at least the same for everyone. Your vendor's in-house evaluation is pointed at your specific target. Tell the company selling you AI what outcome you are measuring, and it will optimize for exactly that outcome, making the ROI look real while it marks its own homework. It is Goodhart's Law with a sales quota attached.
This is not a claim that vendors cheat. It is a claim that the arrangement is broken. The party that gets paid when the number is high cannot be the party that produces the number. The conflict is structural, and structural conflicts do not get fixed by trust.
Can you compare AI models on public benchmarks?
No, not reliably. Because public benchmarks leak into training data, a high score partly reflects memorization, and because the benchmark is a visible target, models are optimized straight at it. To compare models for a decision that matters, use a private benchmark built from your own data, one the vendors cannot see, train against, or game.
How do you beat Goodhart's Law in AI evaluation?
You keep the target private. If the thing being optimized cannot see the criteria it is being judged against, it cannot optimize for them. An evaluation you own, run on your own data, with criteria your vendors never see, is Goodhart-resistant by construction.
That is the whole case for private, owned evaluation. Not that your team is smarter at measuring than your vendor, but that only a measurement your vendor cannot see stays honest. You define what good means, you hold the golden datasets, and the number belongs to you. Vendor evals versus owned evals walks through the difference in detail.