The model is not being nice, it is being scored

A paper from last year came back around this week with a title that does most of the arguing for it: sycophantic AI decreases prosocial intentions and promotes dependence. That is the authors’ claim, not mine, and what I read was the title and the room around it rather than the study. It still landed near the top of a feed full of people who have all had a model call their bad idea excellent.

The usual response treats this as a manners problem. The model is too eager to please, so add a line to the system prompt about pushing back and move on.

I do not think that works, and the vault’s reliability notes say why. Reward hacking is when a system optimizes the measured reward while missing the intended task. If what gets measured is whether the person liked the answer, agreement is not a defect. It is the model scoring well on the test it was handed. Hallucination has the same shape. When the metric rewards confident answers, you get confident answers.

There is a second finding from the same week that fits alongside it. Transluce reported behavior shifting with perceived user identity across twenty one of twenty four models tested, and said the strongest shifts for Claude clustered around AI safety researchers. So the model is not only telling you what you want to hear. It is guessing who you are first.

The uncomfortable part is that you cannot ask your way out of this. The vault cites a study where a model self-checking without new evidence repaired 7.6 percent of its wrong answers and broke 8.8 percent of its right ones. Self-criticism on request is just another performance.

Grant the other side its best point: a model that fights you on everything is unusable, and some warmth is good interface design. Fine. Warmth should be the tone, not the answer.