The old eval suite still passes, and that is the problem

A talk from Braintrust’s field CTO walks through five generations of agent architecture in about fifteen minutes, and the throughline is that nobody rebuilt for fun. One prompt, one call. Then chains, once retrieval showed up. Then a model running in a loop, which was the right idea a little before the models could hold a plan. Then graphs and state machines, because when the model cannot be trusted with the orchestration you take the orchestration away from it. Then back to the loop, because newer models could suddenly do the part that had been taken from them.

His argument is that evals trail that whole parade. Architecture follows the models, evals follow the architecture, except in practice they do not. They go static.

That failure is quieter than a broken test. A suite written for the graph checks branch logic and contracts between nodes. Rebuild the same product as a loop and the suite still runs, still passes, and no longer touches what you actually shipped. He notes the loop brought a second measurement problem with it. The same input takes wildly different paths on different runs, so one run stopped being a unit of evidence, and metrics that run the same case k times started to make sense.

The vault’s observability notes draw the boundary hard. Observability is not an eval. Traces from real runs are where the unanticipated failures come from, and the job is to turn the recurring ones into fixtures and checks rather than another dashboard.

A green suite is only ever a claim about the questions it asks. Change the shape of the system and that claim quietly stops being about the system.