Benchmarks Are a Social Contract
A leaderboard is only as honest as the incentives around it. What breaks when everyone optimizes the measure — and how to design evaluation frameworks that remain meaningful over time.
A benchmark is not a measurement instrument in the scientific sense. It is more accurately described as a social contract: a negotiated agreement between the parties who build systems and the parties who compare them about what the word "better" is permitted to mean. Like all contracts, it holds only as long as both sides believe the other is not gaming it — and the history of machine learning evaluation suggests that belief erodes with remarkable speed once the stakes are high enough.
This essay argues that the current evaluation ecosystem is structurally broken in ways that cannot be fixed by adding more benchmarks, and that the path forward requires rethinking evaluation as an ongoing audited process rather than a static artifact.
The Saturation Problem
Every widely adopted benchmark follows a predictable arc. In its early phase, scores track genuine capability improvement with reasonable fidelity. The tasks are novel, the training data does not yet contain the test set, and the community has not yet learned the specific idiosyncrasies that can be exploited. In this phase, the benchmark serves its function.
Then the benchmark becomes a target in Goodhart's sense: once a measure becomes a target, it ceases to be a good measure. Contamination leaks test items into training corpora, either through deliberate data collection or through the diffuse way in which widely discussed benchmarks propagate across the internet. Ensembles are tuned to overfit to the idiosyncrasies of specific evaluation splits. Prompting strategies are optimized against known rubrics. The leaderboard decouples from the underlying capability it was intended to measure, and the numbers continue to rise while genuine progress slows or stagnates.
In a systematic audit of 40 popular evaluation suites conducted over the past year, roughly one third showed measurable contamination in at least one major model's training data. The tools to detect this contamination exist — n-gram overlap analysis, membership inference attacks, held-out partition analysis — and they are not technically demanding. They are simply not part of anyone's default workflow, because detecting contamination would require lowering one's own reported scores.
The structural problem is one of incentives. The parties responsible for reporting benchmark performance are the same parties whose reputations depend on that performance being high. Independent evaluation infrastructure, analogous to financial auditing, does not exist at scale in machine learning.
What a Durable Evaluation Framework Looks Like
Sustained engagement with the evaluation literature suggests three properties that consistently distinguish evaluation frameworks that remain informative over time from those that collapse under optimization pressure.
Refreshability. A durable evaluation framework can generate or collect new test items on a schedule that outpaces contamination. This requires either a procedural generator — a formal system that can produce novel instances of the target task — or an ongoing data collection operation with access to genuinely new human-produced material. Static benchmark files that are released once and never updated have a finite lifespan proportional to their prominence.
Legibility. A failing case should be readable and understandable by a domain expert within a few minutes, without access to model internals. This constraint rules out evaluation paradigms that rely on aggregate statistics over tasks whose individual items are opaque or require specialized tooling to inspect. Legibility is what allows the community to distinguish genuine capability failure from benchmark artifact.
Cost honesty. Reported scores should include the compute budget, number of attempts, and degree of prompt optimization that produced them. A score achieved with one forward pass under a standardized prompt is categorically different from a score achieved after extensive prompt engineering across dozens of variants. Conflating these is not merely imprecise; it is misleading in ways that distort resource allocation across the field.
None of these properties are technically difficult to implement. All of them are socially difficult, because they constrain the practices that produce impressive numbers. An evaluation framework with refreshability, legibility, and cost honesty will systematically produce lower scores than an equivalent framework without those constraints. The organizations that adopt these constraints voluntarily will appear to perform worse relative to those that do not, at least in the short term.
The Deeper Issue: Static Tests in a Dynamic System
There is a more fundamental problem with the evaluation paradigm that the three properties above do not fully address. A static benchmark, however well-designed, tests a system at a point in time against a fixed distribution of tasks. The systems being evaluated are not static: they are retrained on expanding corpora, fine-tuned on new data, and updated in response to feedback. The gap between a system's benchmark performance and its in-deployment behavior is therefore not merely a function of contamination — it is a function of the mismatch between the evaluation distribution and the deployment distribution, which grows over time as deployment contexts evolve.
Addressing this requires a shift in how the field conceptualizes evaluation: not as a test that a system passes or fails at a moment in time, but as an ongoing measurement process that tracks system behavior across the distribution of real-world tasks. This is more expensive and more difficult to standardize, but it is the only approach that scales to systems deployed in genuinely diverse real-world contexts.
Where the Field Is Likely to Go
The trajectory of the evaluation ecosystem, viewed over a five-year horizon, points toward something resembling audited measurement processes rather than benchmark leaderboards. The analogy to financial reporting is instructive: financial statements are not self-reported without constraint; they are produced under standards, audited by independent parties, and subject to legal liability for material misrepresentation. Machine learning evaluation currently lacks all three of these structures.
The organizations that build this infrastructure early — independent evaluation labs, standardized reporting protocols, auditing mechanisms with genuine teeth — will produce results that remain citable and trustworthy as the field evolves. The organizations that continue to publish self-reported benchmark scores without independent verification will find those scores increasingly discounted by a community that has learned to read the incentive structure behind them.
The social contract of evaluation is not broken beyond repair. But repairing it requires treating evaluation as an institution, not a leaderboard.