The saving happened before the training run

On r/LocalLLaMA this week, in a post written up as an end to end fine-tuning guide, somebody says they tuned a Mistral 7B to outperform their costly foundation model and saved $300k, and that they had previously thought fine-tuning was pointless. The day before, the same feed carried a meme captioned “me to the model I spent all weekend fine-tuning.” Both posts are the genre working exactly as intended.

Concede the fair point. The “just prompt it, a fine-tune is tech debt” line got repeated until it stopped being an argument and turned into a reflex. There are narrow, repeated, measurable jobs where a small tuned open model wins on cost and latency and gives up nothing anyone would miss.

The vault’s fine-tuning notes are stricter about the order of operations. Tuning is not what you do when prompting gets annoying. It is what you do after repeated failures, representative examples, and an eval show the model needs learned behavior rather than clearer instructions, retrieved evidence, or better tool design. Build the measured prompt and retrieval baseline first. Label the actual failure mode: missing knowledge, instruction following, format drift, preference, reasoning. Then take the lightest method that matches. The line worth keeping is blunter than that: a fine-tuning run without a held-out eval is just an expensive prompt experiment.

So read the savings claim backwards. To make it, somebody had to know what the old system cost, know what a good output looked like, and be able to tell whether the new one matched. That is the transferable part, and it existed before any weights moved.

Training scripts are public and free. The eval set is the part nobody can hand you, and it is the whole reason the number means anything.