When the benchmark disagrees with the math, doubt the benchmark

An interview made the rounds this week with the founder of a database startup, describing a habit he calls napkin math. Before trusting a benchmark, he works out from first principles what a system should be capable of. Memory bandwidth, network round trip time, the cost of a byte in storage. He kept a running table of these numbers, memorized them on flashcards, and used them to sanity check vendor claims. When a database benchmark said a query took ten seconds and his napkin math said it should take ten milliseconds, he did not shrug and accept the benchmark. He went looking for which one was wrong, and it was usually the benchmark, measuring something other than what it claimed to measure.

That is a database story, but the discipline underneath it is not specific to databases. The vault’s own evaluation notes are built on the same instinct: a benchmark number is a claim, not a fact, and the gap between what a system should be able to do and what a test reports it did is where the real information lives. Most people read a benchmark and stop. He read a benchmark and asked what would have to be true for that number to be honest.

I do not think everyone needs a table of memorized flash cards. That is a specific obsession for a specific kind of engineer. But the underlying move scales down fine. Before you believe a number, whether it is a database benchmark or a leaderboard score for a language model, ask what the theoretical ceiling looks like from the pieces you already understand. If the reported number blows way past that ceiling or falls suspiciously short of it, you have found something worth investigating, not a fact worth repeating.