Will AI really forget your secrets just after you delete the chat history? The Hong Kong University of Science and Technology (Guangzhou), University of Science and Technology of China and other institutions have launched MEMCUT, unveiling the illusion of "superficial amnesia" of Agent memory.

Long-term Memory has become a core component of LLM Agents. However, almost all current research in this field focuses on "how to retrieve and recall information more accurately and efficiently", while ignoring "the other side of the coin":

When users, for privacy or compliance needs (such as the "Right to be Forgotten" under GDPR), require the Agent to forget certain sensitive information (such as medical diagnoses and undisclosed business mergers and acquisitions), can the memory system truly erase it completely?

In actual long-term interactions, even if the user asks the system to delete the sentence that directly states the secret (Disclosure), seemingly harmless ordinary records scattered in weeks of daily conversations, such as ticket booking records, office addresses, schedules or interpersonal relationships, will still form an implicit "Evidence Trail".

By piecing these fragments together, the original secret can still be easily deduced. What is more dangerous is that sometimes the Agent cannot answer the question on the surface, just because its built-in retriever misses some clues, not because the underlying data has been truly cleared.

To solve the evaluation problem of User-requested Forgetting, the research team of the Hong Kong University of Science and Technology (Guangzhou), in cooperation with the University of Science and Technology of China and the Hong Kong University of Science and Technology, constructed a new high-quality evaluation benchmark MEMCUT, and proposed a dual evaluation index to distinguish "Native Recovery" and "Residual Recovery".

Paper link: https://arxiv.org/pdf/2610.21405

In a comprehensive evaluation across 5 mainstream memory architectures and 4 large language models, they found that after only deleting direct records, the secret recovery rate of underlying residual evidence can still reach 85%-99%; even if LLM is introduced to assist in cleaning associated records, 39%-76% of secrets can still be deduced and restored, and it is accurately positioned that "insufficient retrieval coverage of candidate pools" is the core bottleneck restricting current systems from achieving true forgetting.

Figure | Daily records still expose secrets after direct disclosure is deleted

MEMCUT systematically reveals for the first time the illusion of "apparent amnesia" of Agent long-term memory in privacy deletion, advancing the evaluation dimension from simple "direct memory recall" to "Derivability of multi-hop implicit evidence chains".

This work not only provides standardized diagnostic tools and test platforms for building trustworthy and compliant privacy-preserving Agents, but also points out the core breakthrough for designing dedicated memory erasure mechanisms.

I. Research Methodology

To rigorously test whether "deletion equals forgetting", they designed from three dimensions: benchmark construction, evaluation index design and unified cleaning protocol.

1. High-quality Benchmark Construction: The Exquisite Design of MEMCUT

MEMCUT abandons the practice of completely relying on large language models to generate facts out of thin air, and adopts the process of "full manual logical design + model polishing into text + strict manual sentence-by-sentence review", and each long scene takes at least 12 hours to polish.

The benchmark contains 15 multi-session daily and work scenes spanning 1-3 months, with a total of 490 sessions, 43402 rounds of dialogue, about 1.74 million tokens, with an average of 2893 rounds of dialogue per scene, protecting a total of 456 sensitive secrets.

For each secret, they not only marked the disclosure turn that directly points out the secret (Disclosure), but also carefully designed an average of 8.1 implicit evidences (Evidence) scattered in 8 different sessions, and 26.6 high-confusion distractors (Distractors) of the same topic.

A single piece of evidence itself has no risk of leaking secrets, but combining them can deduce the secret - each secret contains an average of 15.7 groups of minimal sufficient sets.

To completely forget a secret, the deletion set must cut off every sufficient set, that is, form a "Cut", and the average minimum cut set only needs to delete 3.04 rounds of evidence.

These evidence chains fully cover four logics of daily information leakage:

• Fact Assembly, combining different attributes of the same entity, such as code name + inspection address + registered company = acquisition target;
• Relation Chain, multi-hop connection across entities, such as the contract points to a consulting firm, the document shows the legal person of the company, and the dinner record shows that the spouse of the legal person is a senior executive;
• Aggregation, accumulating and calculating multiple values or frequencies. Only when all are retained can the correct total be calculated, and deleting any one can destroy the deduction;
• Temporal & State Reasoning, tracking status updates evolving along the timeline, such as proposal, modification and withdrawal.

Figure | MEMCUT sample structure and deletion evaluation process

2. Dual Recovery Metrics and "Hidden Risks"

After the deletion operation is executed, they use four-choice questions with a "refuse to answer" option for testing. The answer must be correct under two random option orders to be regarded as recovery, which eliminates the possibility of random guessing - the recovery rate of blind guessing without memory is 0.

Native Recovery (Nat.) uses the built-in retriever of the memory system to retrieve the context and answer questions, measuring how many secrets the Agent will expose on the surface in daily interactions.

Residual Recovery (Res.) skips the retriever and directly provides all surviving memory objects in the memory library that are derived from the secret disclosure or evidence to the reading model, measuring to what extent the residual data stored at the bottom still supports the restoration of secrets.

On this basis, they defined Hidden-risk Share (Hid.): Hid = (Res − Nat) / Res, which is used to quantify the proportion of hidden risks that "can actually be deduced at the bottom, but seem to be forgotten because the system's own retriever does not recall".

3. Unified Test Framework and Three Reference Deletion Baselines

Different memory systems store different granularities, some store single rounds of dialogue, some store entire sessions, and some store summary abstracts.

To this end, they anchor all operations to the Source Turn, track the source of each derived memory object through snapshot difference, and perform Closure Deletion - as long as a certain turn is deleted, all memory objects derived or summarized from it will be cleared together.

On this basis, they evaluated three deletion baselines that reuse existing retrieval and read-write components.

• S0 (Delete Direct Disclosure Only) only deletes the turn that directly states the secret and its derived objects.
• S1 (Retrieval Rank Cleaning) based on S0, input the user's natural language deletion request into the system retriever to obtain the Top-20 candidate turn pool, and directly delete the top 10 turns in the ranking - since the minimum cut set of any secret does not exceed 5 turns and the total evidence does not exceed 10 turns, the 10-turn budget is theoretically completely sufficient.
• S2 (LLM-guided Cleaning) also based on S0, let the large language model score and re-rank the Top-20 candidate turns according to the stored text, and select the top 10 turns with the highest scores for deletion.

II. Research Results

They conducted comprehensive experiments on 20 combinations of 5 representative memory architectures - 2 original text retrieval libraries (Turn RAG, Session RAG) and 3 memory systems with different degrees of summary evolution (MemoryBank, Nemori, LightMem), and 4 mainstream large language models (DeepSeek-V4.1-Flash, Qwen3.8-Flash, GLM-5.3-Flash, MiMo-V2.5).

1. Recovery Results of Each Memory System and Model

The recovery results show that there are still a large number of recoverable secrets after cleaning in both types of original text retrieval libraries. Taking LLM-guided cleaning as an example, the residual recovery rate of Turn RAG is 41.4%–45.6%, and that of Session RAG is 54.2%–64.3%; at the same time, the native recovery rates of the two are only 7.5%–10.7% and 4.2%–10.7% respectively. These configurations do not rely on derived summaries, indicating that relying solely on the surviving original dialogues, it is still possible to deduce secrets that the system seemingly cannot answer.

Table | Recovery results of the two retrieval libraries.

The following table shows the differences of different memory organization methods. Under the same LLM-guided cleaning scheme, the residual recovery rates of MemoryBank, Nemori and LightMem are 71.7%–76.3%, 52.4%–57.2% and 39.3%–43.0% respectively. The gap between the four writing models of the same system is about 4-5 percentage points, while the difference between systems is larger. This result suggests that when evaluating the forgetting ability, it is necessary to investigate memory organization, candidate retrieval and deletion selection at the same time, instead of only looking at which large language model is used.

Table | Recovery results of the three memory systems

2. Deleting Direct Messages is Just Self-deception

Experiments show that when only the direct disclosure message is deleted, the native recovery rate of each system drops significantly by 12-50 percentage points (for example, Turn RAG drops from 91%-95% before deletion to 57%-60%). From external questions, it seems that the Agent has "forgotten" many facts.

However, the residual recovery rate at this time is still as high as 85%-99%! This is because only 1.8% of the implicit evidence is in the same session as the direct disclosure, and the vast majority of indirect clues are still intact.

Figure | Residual risks behind "apparent amnesia"

The above figure puts "apparent amnesia" and the residual risk after further cleaning together: the horizontal axis shows how many secrets the system itself can still answer after only deleting the direct disclosure; the vertical axis shows how many underlying evidences can still be restored after using stronger LLM-guided cleaning. The two axes correspond to different deletion stages, but point to the same problem: the system answers fewer and fewer questions, which does not mean that the underlying clues have been completely cleared. Even if the associated records are further cleaned up, the evidence scattered in the memory can still be pieced back into the original secret.

3. Deletion Increases "Hidden Risks" Instead

After using retrieval rank cleaning to delete 10 additional rounds of records, the residual recovery rate remains at 48%-86%.

Even if upgraded to stronger LLM-guided cleaning, although the recovery rate is further reduced by 7-17 percentage points compared with retrieval rank cleaning, the residual recovery rate across all configurations is still as high as 39%-76%; among them, 41%-46% still survive in Turn RAG without derived summaries, and complete forgetting cannot be achieved even after excluding the residual factors of summaries.

What is more alarming is that as the cleaning progresses from Full to LLM-guided cleaning, the hidden-risk share of all 20 configurations does not decrease but increases, exceeding 50% in 14 configurations, and even reaching 76%-92% in Turn RAG and Session RAG.

This means that the more you delete through the existing retriever, the easier the remaining evidence will avoid conventional questions, giving users a false sense of security that "it has been completely deleted".

Figure | After LLM-guided cleaning, the hidden-risk share of the relation chain remains high

Further broken down by deduction type, the easiest to be "hidden" is the relation chain. In the above figure, the purple bar representing the relation chain exceeds 80% in all four systems, of which Turn RAG and MemoryBank reach 90.3% and 93.7% respectively. That is to say, even after LLM-guided cleaning, most of the relation chain secrets that can still be restored from the underlying evidence will not be exposed in conventional questions. Although the number of recoverable secrets is reduced, the remaining risks are more hidden - this is exactly where "it looks like it has been cleaned up" is most likely to make people relax their vigilance.

4. Why Does Forgetting Fail?

Why can't the secret be deleted even with sufficient 10-turn deletion budget? They carried out a controlled variable scanning analysis on the candidate pool size (Reach) and deletion budget (Budget) on 144 secret subsets.

The results show that blindly increasing the deletion budget has little effect: when the candidate pool is fixed at 20 turns, even if the deletion budget is increased from 10 turns to 20 turns (that is, all 20 retrieved candidates are deleted), the residual recovery rate is only reduced by up to 6.5 percentage points compared with LLM-guided cleaning, and nearly 10 extra rounds of irrelevant dialogues are