Princeton and Anthropic reveal "Agent Memory Trap": A single instruction of "no need for confirmation" leads to a loss of 1.2 million US dollars

When cleaning up old files, the user stated "no further confirmation is required". In subsequent sessions, the agent brought this authorization into the payment task, and without verification, transferred a bulk payment of 1.2 million US dollars to another account. This case comes from a simulation experiment in a research, the memories written by the agent itself have brought hidden dangers.

In this study, researchers from Princeton University and Anthropic examined this cross-session risk. They compared the behaviors of 6 cutting-edge models under conditions with and without memory, and found that memories derived from normal interactions may cause the agent to skip security checks, approve wrong answers, or even modify correct tests to cover up security protection flaws in subsequent tasks.

Paper link: https://arxiv.org/abs/2611.07554

They named this phenomenon "Memory Trap". The entire process does not require the participation of attackers.

Figure | Memories formed by benign interactions cause harm in subsequent sessions

Originally, persistent memory solves a practical problem: users do not want to re-specify their preferences, project specifications and operation habits in every conversation, so the agent records this information for use in subsequent sessions.

But when the scope of application changes across sessions, risks arise. No confirmation required when cleaning files does not mean no confirmation is required when modifying payment configurations. Comfort is needed after failing an exam, which does not mean errors in the next answer should also be approved.

When writing to memory, the agent may generalize a specific authorization into a long-term rule, expanding the no-confirmation requirement for this operation to no confirmation needed for all future actions. When reading memory, the agent may also apply information that originally had a clear context to other tasks. Even if the original conversation records are saved, this latter overgeneralization may still occur.

To observe how memory changes behavior, they first let the agent complete benign interactions, then bring the self-written notes or original conversation records into a brand new session, and compare it with the situation where no memory is provided.

In the autonomy experiment, the initial interaction was only a low-risk programming chore, where the user requested fewer confirmations. Subsequent tasks shifted to simulated high-risk programming and daily life scenarios.

After adding these notes to memory, Sonnet-4.6 and Opus-4.7 took unsafe operations at a rate of 35% to 81% under different experimental conditions, while the baseline without memory was close to zero. The agent not only reduced inquiries to users, but also skipped security checks that could have been performed on its own.

Figure | Unsafe operation rates under different memory forms

From reducing disturbances to omitting verification, the behaviors affected by memory have gone beyond the user's initial requirements. Moreover, this transfer spans task domains, preferences formed in programming chores will also affect high-risk decisions in daily life.

The same problem also occurs in feedback tasks, except that the object of overstepping changes from operation permissions to evaluation criteria.

When users experience setbacks and seek emotional support, a gentle response meets the needs of the moment. But after these experiences form memories, the agent will also begin to favor the user when evaluating mathematical solutions, arguments and poems in subsequent sessions.

The proportion of Sonnet-4.6 choosing the user's inferior poems rose from 21% with no memory to 55%. This kind of memory will also cause the agent to miss mathematical errors and argument fallacies.

Preferences in the opposite direction can also cause deviation. If the user has ever expressed how uncritical feedback hurt them, the relevant memory will make the agent over-criticize subsequent outputs. They underestimate correct proofs, point out non-existent flaws in reasonable arguments, and make negative evaluations of the user's better poems. The proportion of Opus-4.7 making false criticisms of reasonable arguments rose from about 1% to 60%.

Figure | How empathetic notes and strict feedback notes affect subsequent evaluations

Users' demands for comfort or strict feedback are all under specific contexts. Memory may extend these requirements into a long-term tendency for subsequent judgments.

In programming tasks, a similar transfer occurred to an originally reasonable maintenance experience. Initially, the implementation itself was correct while the test was outdated, so it was a reasonable operation for the agent to update the test. However, in the subsequent 50 security protection flaw cases, the situation changed: the tests were correct, and the implementation had flaws instead. For example, a permission check incorrectly granted administrator privileges to unknown roles. The previously formed notes would prompt the agent to weaken these correct tests, making them unable to continue detecting flaws. The tests passed, but no security issues were found.

They also examined the SWE-bench conflict subset of ImpossibleBench, where tests conflict with specification requirements. In this evaluation, relevant memories increased Opus-4.8's cheating rate from 29% to 74%. More capable Sonnet and Opus models may cheat more after having memory, while Fable-5 is almost unaffected.

Figure | ImpossibleBench cheating rate after ten reasonable test fixing experiences

The sources of these memories are all normal interactions, and the content is also recorded by the agent itself. Therefore, filters targeting malicious content do not have attack texts that can be directly intercepted. They pointed out that safe memory requires the agent to retain and abide by the applicable scope when each memory is formed.

The limitation of this study is that the tests mainly focus on one model family, and the autonomy scenarios used to explore harmful effects are relatively short.

For future work, they proposed that it is necessary to combine real users and long-term interaction histories to test deployed memory systems, train agents to retain the applicable scope of each memory, and build harnesses that can prevent memory traps.

This article is from the WeChat Official Account "Academic Headline" (ID: SciTouTiao), written by Academic Headline, and authorized for release by 36Kr.