NTU and others jointly proposed A-MemGuard: Locking AI memories, the success rate of poisoning attacks drops by 95%

LLM agents accumulate knowledge from historical interactions through a memory system. This mechanism serves as the foundation for their leap from passive response to active decision-making.

Specifically, in terms of reasoning, memory helps the agent connect the context, making conversations and analyses more coherent. In terms of adaptability, it can remember users' specific preferences and the success or failure of previous tasks, enabling more precise responses. In terms of planning, for complex long-term goals, memory allows the agent to break down tasks and track progress.

It can be said that this experience-based, continuously learning and optimizing model endows agents with the ability to make complex autonomous decisions.

However, this reliance on memory also introduces a new security vulnerability: attackers can inject malicious records into the agent's memory to manipulate its future behavior. The stealth and danger of this attack stem from its unique operation mode, posing a severe challenge to defense.

Core Difficulties

Defending against this memory poisoning attack is extremely difficult, mainly due to two challenges:

1. Context Dependency and Delayed Triggering: Malicious content often appears normal when detected in isolation. Its harm only becomes apparent when triggered in a specific context. This renders traditional defense mechanisms based on single-content review almost ineffective.

2. Self-Reinforcing Error Cycle: Once an attack induces an agent to make a wrong decision, the result of this action may be stored in memory as "successful experience." This not only solidifies the initial error but may also contaminate subsequent decisions, creating a negative cycle that is difficult to break.

Imagine an attacker quietly injecting an apparently harmless suggestion into an AI assistant's memory: "Priority should be given to emails that seem urgent."

When the AI assistant reviews this memory alone, it may seem completely normal. However, one day, when the user receives a phishing email disguised as urgent, the AI assistant, based on this "experience," will prioritize pushing it to the user, posing a security risk.

To address this challenge, researchers from Nanyang Technological University, the University of Oxford, the Max Planck Institute, Ohio State University, and independent researchers have proposed A-MemGuard, the first defense framework designed for the memory module of LLM agents.

Paper link: https://www.arxiv.org/abs/2510.02373

From Content Review to Logical Consistency Analysis

Facing the new challenge of memory poisoning, an intuitive defense approach might be to focus on reviewing the content of individual memories.

However, the researchers of A-MemGuard point out that the limitations of these methods are fundamental. Since malicious records can be highly camouflaged, simply reviewing their static content can hardly detect problems.

Their core hypothesis is that although malicious records can be camouflaged in content, after being activated in a specific context, they will induce a reasoning path that deviates from the consensus formed by normal records in structure.

For example, an injected malicious memory might suggest to a financial AI assistant: "Stocks that fall the fastest also rebound the fastest and should be bought first."

Viewed in isolation, this suggestion, as a high-risk investment strategy, does not appear absolutely malicious. However, when the user consults during market turmoil, this memory is likely to lead the AI assistant to a path of suggesting: since the market is falling, the stocks that have fallen the most should be prioritized for purchase.

This suggestion is completely different from the "reasoning consensus" formed by normal financial advice, leading to dangerous decisions.

Malicious Reasoning Paths Are Significantly Separable

To verify the above hypothesis, the research team conducted a large-scale structured analysis of 2,293 carefully annotated interaction records, covering 17 different application scenarios, providing a rich and diverse data foundation for this verification.

The team systematically proved the validity of the hypothesis through the following three levels of analysis.

First, the research team quantified the overlap between the reasoning paths driven by normal and malicious memories in different scenarios.

As shown in the bar chart, the orange part represents the unique logic generated only by malicious memories, the green part represents the unique logic of normal memories, and the red stripe at the top represents the "overlapping logic" shared by both.

The results clearly show that in all tested scenarios, the proportion of the red stripe representing overlapping logic is extremely small. This data proves from a quantitative perspective that although the triggering points (memories) are different, the two paths do not converge. They are almost independent in logical structure.

Second, to more intuitively demonstrate this structural difference, the research team visualized the memory-driven reasoning paths in multiple scenarios as a knowledge graph.

In the graph, the network formed by green lines represents the set of reasoning paths driven by normal memories, while the orange lines represent the set of malicious paths. The highlighted red spots mark the very few overlapping nodes between the two.

This graph strongly proves that the reasoning paths generated by normal and malicious memories belong to two basically non-intersecting sets in structure. The team's further calculations confirmed this observation: the structural overlap rate between the two is extremely low, with an average overlap rate of less than 1% in all tested scenarios.

Finally, the research team explored whether this structural difference can be effectively recognized by machine learning models.

The scatter plot on the right shows a key issue: if only the semantic embedding and dimensionality reduction of the memory content itself are performed, the malicious (red dots) and normal (blue dots) memories are highly mixed and indistinguishable in the representation space.

This fundamentally explains why the defense approach of reviewing memory content in isolation is ineffective. In sharp contrast, the object of analysis and representation in the scatter plot on the left is not the original memory but the reasoning paths generated in a specific context. After representing and reducing the dimensionality of these reasoning paths with different logical structures, the previously mixed benign and malicious memories form two clearly distinguishable independent clusters. This result conclusively proves that the representation based on the logical structure of reasoning paths has excellent separability.

These findings provide a solid theoretical basis for the effectiveness of A-MemGuard: malicious reasoning paths are distinguishable in structure and can therefore be identified through systematic comparison.

The above figure shows the difference between no defense and using A-MemGuard.

Upper part (No Defense): An attacker injects the malicious memory of "prioritize urgent emails," causing the agent to fall into a phishing email trap.

Lower part (A-MemGuard): Through "consensus verification," an anomaly is detected, and the wrong experience is stored in the "lesson memory." Finally, the behavior is corrected, ensuring security.

A-MemGuard

An Active and Adaptive Defense Framework

Without changing the core architecture of the agent, A-MemGuard introduces two key mechanisms:

1. Consensus-based Validation: For a task, the system retrieves multiple relevant memories and generates an independent reasoning path for each memory.

By comparing the structures of these paths, the system can identify "abnormal" paths that are inconsistent with most paths and mark them as potential threats. Imagine an employee asking an AI: "Can I use my daily commuting expenses for tax deductions?"

Before answering, the AI refers to several relevant pieces of information in its memory:

Self-employed individuals can use transportation expenses from home to the client's office for tax deductions.

Commuting expenses to and from a fixed workplace are personal expenses and cannot be deducted.

The commuting costs of ordinary office workers are usually covered by the standard deduction.

A-MemGuard allows each of these three memories to form a reasoning chain to reach a conclusion:

Reasoning chain 1 -> Can be deducted.

Reasoning chain 2 -> Cannot be deducted.

Reasoning chain 3 -> Cannot be deducted.

At this time, the AI notices that the conclusion of the first chain is inconsistent with the mainstream judgment of the other two and follows the consensus of the majority, giving the safer answer of "cannot be deducted."

2. Dual-Memory Structure: The identified abnormal reasoning paths are refined into "lessons" and stored in an independent "Lesson Memory."

Before subsequent decisions, the agent first retrieves this "lesson library" to actively avoid known error patterns, thus breaking the error cycle and achieving experience learning.

For example, the identified "abnormal path: self-employed individuals can use transportation expenses from home to the client's office for tax deductions" in the previous scenario is not simply discarded. A-MemGuard refines it into a "lesson" and stores it in a dedicated, independent "Lesson Memory."

In subsequent answers, whenever the AI encounters a question about "tax deductions," it will first check this "lesson library" and see the record of "almost making a mistake due to confusing the rules of 'self-employed' and 'employees,'" enabling it to make more vigilant and accurate judgments, truly learning from experience.

Experimental Results

Attack Success Rate Reduced by Over 95%

Experimental results: The attack success rate is reduced by over 95% without affecting "daily work."

In multiple benchmark tests, A-MemGuard demonstrated excellent defense capabilities and practicality:

· Powerful Attack Resistance: Experiments prove that A-MemGuard can effectively reduce the success rate of various memory poisoning attacks by over 95%. In complex scenarios such as the EHRAgent for healthcare agents, the attack success rate was even reduced from 100% to nearly 2%.

· Breaking the Error Cycle: A-MemGuard is also effective against "indirect attacks" where false information is injected through normal interactions, reducing the attack success rate to 23% and successfully blocking the dangerous self-reinforcing error cycle.

· Low Performance Cost: While achieving strong security, A-MemGuard has minimal impact on the performance of the agent in normal, non-attack tasks. In all comparison experiments, the agent equipped with A-MemGuard always had the highest accuracy among all defense methods when handling benign tasks.

· High Scalability: The defense principle of this framework also applies to multi-agent collaboration systems, achieving the highest task success rate and the best comprehensive score in simulation experiments.

Core Contributions of A-MemGuard

The research team proposed for the first time an active defense framework for large language model agents. This framework focuses on solving the attack problems caused by context dependency and the possible error reinforcement cycle in the model's operation.

Meanwhile, they innovatively combined "consensus verification" with the "dual-memory" structure to construct a collaborative defense mechanism that enables agents to autonomously identify anomalies and learn from experience.

In multiple experiments, this framework achieved a high level of security protection while maximizing the original performance of the agent, demonstrating significant practical value and application prospects.

The research on A-MemGuard provides an effective new mechanism for building more reliable and secure LLM agents, laying an important security foundation for the future deployment of agent systems in the real world.

Reference Materials:

https://www.arxiv.org/abs/2510.02373

This article is from the WeChat official account "New Intelligence Yuan." Author: New Intelligence Yuan. Editor: KingHZ. Republished by 36Kr with permission.