Ask a self improving harness what it is not allowed to edit

An open source coding harness landed this week describing itself as self improving: programmatic tool calling, context as a variable, multi agent messaging, persistent execution, and a harness state the agent can rewrite. The headline claim, per the roundup I read, is 95.5 percent on ARC-AGI-3, above the stated human expert baseline. Two hundred and forty nine points on Hacker News, and a comment section that did not go easy on it.

The sharpest objections were not about the score. One commenter noted that most models are not trained to exploit self modification reliably. Another pointed out that repeated runs against the same benchmark let a system converge on benchmark specific improvements, so a fresh run would have to earn the claim again. A third shrugged that subagents are always just tool calls.

The idea is sound. The framing is the problem.

A harness is the system around the model: context, tools, state, triggers, evaluators, stop conditions, repair paths, logs, review gates. Letting an agent edit part of that is a real technique with real literature behind it. The vault’s harness notes say the loop research supports action feedback, durable state, checkpoints, and repair paths, and specifically does not support claims about self improvement without evals.

The example those notes hold up is Karpathy’s autoresearch, and the interesting thing about it is everything the agent cannot touch. One editable code file. Frozen evaluator and support files. A human written description of the program. One hard metric, a ledger of every attempt, and git deciding keep or revert.

So that is the question for any harness that improves itself. Not how much of it changed. What you froze.