The approval gate has an input nobody budgets for

381 points and 333 comments this week on a post titled AI handles incidents, engineers lose touch with their systems. I have the headline, so the argument is the author’s. A few entries down, somebody on r/LocalLLaMA writes up letting a local Qwen model unhack their PC, which is the same trade in miniature and reads as a happy story.

The trade looks good on its face. Incidents arrive at bad hours to tired people, a model reading logs at four in the morning does not get worse at reading logs as the night goes on, and nostalgia for pager suffering has never fixed an outage.

The vault’s operating principle here is approval-gated autonomy. Agents do the monitoring, drafting, testing, summarizing, and preparation. A person keeps lightweight approval over the consequential actions: publishing, merges, deletions, production writes, money, anything hard to undo. The same notes add that fast agents need logs, backups, and rollback paths before they touch production, and that no capable agent is safe by default.

I still hold that. It has one input that never makes it onto the diagram, though. An approval is worth something only if the person approving can tell a good plan from a plausible one, and that ability is not issued with the job title. It came from handling incidents. Guessing wrong at an hour when nothing makes sense and then finding out why is not a rite of passage, it is the curriculum. It is also the first thing anybody automates, because it is the most unpleasant.

So the gate keeps passing its own test while turning into ceremony. Same prompt, same click, less behind the click every quarter.

If a system plans to keep a human in the loop, somebody has to answer where that human’s judgment will come from once the loop has stopped manufacturing it.