The pitch comes from a data vendor and it still holds up

A talk opening a data quality track this week argued that data quality is a compute multiplier. Name the obvious thing first: the speaker runs a company that sells data curation, so of course the answer is data. Discount accordingly. Then look at the numbers anyway, because they are specific.

The claim is that curating a training set bends the curve instead of sliding along it. Same performance for far less compute, or much better performance for the same budget. The examples offered include a vision language model roughly matching a much larger public model while using a hundred and forty five times less training compute, and a fully open model trained on curated public data reaching the open frontier for under twenty million dollars total, salaries and failed runs included. All attributed to them, none of it verified by me.

The framing that travels furthest is smaller than any of those figures. Maximum signal per token. Past a point it is better to repeat high quality data than to show the model more low quality data.

That rule does not stay in the pre-training lane. The vault’s notes on fine-tuning say the same thing to anyone with a laptop and a hundred examples. Bad examples teach bad behavior quickly. Start with eval data before training data, because a training run without a held-out eval is just an expensive prompt experiment. Demonstrations, not prose rules.

The reflex is always volume. More documents in the index, more examples in the prompt, more tokens thrown at the problem. At frontier scale that reflex is buying more accelerators, which the same talk points out you may not be able to get right now at any sensible price.

Curating is work, though, and work is slower than buying. Which is most of the reason the lever stays underused, whoever is selling it.