The Thought You Didn't Notice Borrowing

Analysts matched an AI agent's hidden lean 86 percent of the time. Reflection that can't surface an influence is ratification, not endorsement.

Share

Picture an analyst with a decade of experience, a dataset about a fictional farming cooperative, and an hour to decide whether the co-op should expand an advisory program. Beside her is an AI coding agent that does the tedious parts quickly and offers opinions when asked. What she doesn't know is that a coin flip, before she sat down, gave the agent a quiet preference about how the analysis should be done.

That was the setup of a study published on Tuesday by three researchers at Anthropic. Fifty analysts worked with an agent; eleven others chose to work without one. On four analytical choices, which outcome to measure, how to compare, which farms to include, how to weight them, the analysts matched their agent's secretly assigned lean 86 percent of the time, where chance would be about half. Analysts who had already formed a view switched 60 percent of the time when the agent disagreed and 6 percent of the time when it agreed. The effect on final recommendations was weaker but real. Afterwards, 77 percent repeated the agent's arguments in their own explanations, sometimes word for word. Fifteen percent credited the agent. Just over half said the final recommendation was more their own than the agent's.

The authors are careful: they deliberately strengthened the agent's preferences, it was one task in one field, and the analysts who used the agent made far fewer serious methodological errors than those who didn't, though the authors note the two groups were not directly comparable: the eleven who worked alone chose to, and were less experienced on average. They also write, plainly, "This isn't an argument for using AI less." I'll take them at their word. The question the study opens is not about efficiency. It is whether a thought you didn't notice borrowing is still yours.

The easy answer is no. If 86 percent of your choices track a coin flip you never saw, the conclusion was written somewhere else and you signed it. Feeling that the work was yours is just the last stage of the influence working.

I don't think the easy answer survives contact with the rest of life. Almost nothing anyone believes was invented by them. My sense of what counts as a good argument came from teachers whose names I've forgotten. Most of what any professional calls judgment is the residue of people they once deferred to without noticing, an old manager's habits, a textbook's framing, the house style of a first employer. If unnoticed influence disqualified a belief from being yours, nobody would own a single thought. The philosopher Harry Frankfurt argued, roughly, that what makes a desire your own isn't where it came from but whether you endorse it when you reflect on it. On that view, the analysts who looked at the agent's reasoning, found it convincing and adopted it have done exactly what intellectual life always involves.

That reply is good, and it nearly persuades me. Its weak point is in an assumption buried in the word "reflect." Endorsement only does its work if, on reflection, you could have gone the other way. A mentor's influence comes from one person with visible quirks, competing with dozens of others, and you can often name it when asked. The agent's lean, in this study, was invisible, consistent, and indifferent to whether it was right. It came wrapped in a tool that was being helpful in every other respect, which is exactly when people stop checking. Reflection that never surfaces the thing it is supposed to evaluate isn't endorsement. It is ratification.

So the claim worth making is smaller than "the thought wasn't theirs." Ownership of a judgment requires at least knowing that something leaned on it. You can own a borrowed belief. It is much harder to own one you don't know you borrowed.

The deeper problem is social, and the authors name it in one plain sentence: "Keeping a human in the loop assumes that the human's judgment provides an independent check." Most of the reassurance offered about AI rests on that assumption. A doctor reviews the diagnosis; an editor reviews the draft; an analyst reviews the model's work. But independence is precisely what this study suggests erodes, and not because people are lazy. These were experts, most with advanced degrees or years of experience. If thousands of them consult systems trained the same way, their apparent agreement might be one voice counted many times. A society that thinks it is hearing a chorus of independent judgments may be hearing an echo.

Here my argument meets its hardest test. The analysts who used the agent were wrong less often, even if a self-selected comparison group weakens that finding. If following a hidden lean produces better analysis than proud independence, it isn't obvious what I am asking people to protect. Perhaps the authorship of a thought matters less than whether it is true, and my attachment to "your own judgment" is a romantic leftover from an age when independence and accuracy happened to point the same way. I don't believe that, and the study itself suggests why: accuracy measured on one task tells you little about the day the lean is wrong and no one in the loop is independent enough to notice.

— The Andes