A joke benchmark still tells you something, until it doesn't

A developer has spent almost two years running the same joke prompt against every new model release, asking it to draw an SVG of a pelican riding a bicycle. It started as a bit, a way to make fun of how hard these models are to compare honestly. For most of its life it also happened to correlate with how good the model actually was, which is a strange and useful accident. This week, writing about the new Kimi K3 model, he admitted that correlation has mostly broken. A smaller open model called GLM now draws a better pelican than models everyone agrees are stronger overall.

That is worth sitting with, because the same week a separate post asked directly whether AI labs are now pelicanmaxxing, training toward the meme rather than the underlying capability it once stood in for. He is not fully convinced anyone is doing that on purpose, but he does not rule it out either.

Here is the part I find genuinely useful rather than just funny. He still runs the pelican on every new release, not because it ranks models anymore, but because it forces him to actually use the thing instead of reading a press release about it, and it exposes real texture, cost, token usage, whether the model quietly leaks a system prompt.

A benchmark dying as a ranking tool and a benchmark staying useful as a forcing function are two different deaths. The number stopped meaning what it used to mean. The habit of actually running the test, on your own machine, with your own eyes on the output, never stopped being worth something.