The corner where more tokens is not on the menu

Nearly everything in this week’s capture is a release. Weights dropping tomorrow, a quantization comparison, a megathread, then a second megathread about the megathreads. Sitting next to all of it is a Latent Space piece on Anima Anandkumar’s work, and it lands in the one corner where the usual move is not available.

Her area is continuous physical systems. Weather, fusion, fluid and heat flow. The piece reports that open datasets in many of these domains run to tens or hundreds of thousands of examples, nowhere near what a token-hungry transformer wants, and quotes her on the resolution problem: at a few hundred grid points per dimension, “we’re talking hundreds of billions to even a trillion context length. So forget ever having a transformer for anything of this scale, all of the world’s compute will not be enough.”

The bitter lesson deserves its say here. Structure hand-built by clever people has lost this argument many times, and there is a version of this post from a decade ago that aged badly.

Still, the vault’s model research notes give a rule that settles which case you are in. Map any architecture claim to the bottleneck it tries to move, and treat a scaling extrapolation as useful only when the fitted regime matches the run you are about to do. That is the disagreement in one line. Scaling arguments assume the data and the regime carry over. In these domains the piece says they do not, and not for want of budget.

What replaces it, per the piece, is neural operators: learning a function that evolves across scales instead of a grid, with the spherical version powering FourCastNet 3 and staying stable long enough to roll forward months rather than days. It reports that in fusion, a few thousand samples predict plasma disruptions a million times faster than traditional simulation.

The earth being round is not a hyperparameter. Somebody had to know it and build it in, and that is the step the scaling story keeps assuming it will get to skip.