HKUST Team: Enabling Agents to Retrieve "Lost Memories" in Long Conversations

In long-term dialogues, personalized services and multi-round task collaboration, AI Agent often needs to answer a type of seemingly simple but actually very difficult question: two pieces of information are far apart, with almost no identical words on the surface, but both must be retrieved at the same time to get a complete answer.

For example, when the user asks "Where did I first meet the person who influenced my career choice?", the system needs to find two dialogues at the same time: "This teacher influenced my career choice" and "I met him at a certain event". The former is highly related to "career", while the latter may not have any similar words. If retrieved only by similarity, the AI Agent can easily know "who" it is, but cannot answer "where".

To address this, a research team from The Hong Kong University of Science and Technology (Guangzhou), Jilin University and The Chinese University of Hong Kong proposed HeLa-Mem (Hebbian Learning Associative Memory), trying to solve the association gap in long-term memory: to ensure that the experiences in the dialogue are not only saved separately, but also gradually build connections in the process of being recalled together again and again. The related paper has been accepted by ACL 2026 Main.

Paper link: https://aclanthology.org/2026.acl-long.625/ GitHub repository: https://github.com/ReinerBRO/HeLa-Mem

Experiments on LoCoMo show that HeLa-Mem uses significantly fewer context tokens while delivering excellent performance across all four question categories.

The value of this work lies in advancing the focus of AI Agent long-term memory from "how much history to retain" to "how relationships are formed between different pieces of history". When an AI Agent needs to accompany users, follow up on complex tasks or collaborate across sessions, the truly important information often lies in the associations gradually formed across multiple experiences.

Let memories build connections in the process of being used

At present, many AI Agent memory systems split historical dialogues into fragments, write them into vector databases, and then recall content according to semantic similarity. This method can efficiently retrieve content that is close to the wording of the question, but the key clues in long dialogues are often not the one that "looks the most similar".

Human memory does not rely entirely on word similarity. A person you meet at an event may later influence your career choice; a certain travel experience may also be related to a long-term preference. These pieces of content are far apart in text, but connected in real experiences. HeLa-Mem takes this kind of relationship as part of the long-term memory system: each round of dialogue is a memory node, and the edges between nodes record the degree to which they have been used together.

Figure | HeLa-Mem organizes dialogue history into a weighted association graph. Memories that are repeatedly evoked together will form stronger connections.

The following diagram puts the complete closed loop of HeLa-Mem together: each round of dialogue is first written into the episodic memory network; when facing a new question, the system searches for clues from both semantic similarity and existing relationships at the same time; the Reflection Agent then extracts knowledge and eliminates expired nodes according to the network status. The two retrieval results together with semantic memory form the answer context of the LLM, and the new usage process will continue to update the association strength. 

Figure | Overview of the HeLa-Mem method in the paper. Hebbian association builds the episodic memory network, the reflection integration module is responsible for memory refinement and forgetting, and the retrieval stage retrieves the context through two paths. 

The first step of HeLa-Mem is to encode each dialogue turn as an episodic memory node, storing the original text, time, role, keywords and vector representation. When several nodes appear together in one retrieval, the system will strengthen the association between them; weak connections that have not been used for a long time will gradually decay. As interactions accumulate, the memory bank will form a network with strong and weak relationships.

The system will also regularly review this network. For nodes with dense associations and their neighborhoods, the Reflection Agent will extract reusable information from them to form stable semantic memory, such as user preferences, event relationships or facts with absolute time. Nodes that are weakly associated, unused for a long time and not accessed recently will become forgetting candidates to control memory size and noise.

Figure | The Reflection Agent refines semantic knowledge around high-association nodes, and identifies isolated, stale and unaccessed forgetting candidates.

During retrieval, HeLa-Mem follows two paths: one path directly recalls candidate memories according to vector similarity, keywords and time factors; the other path starts from high-score nodes, diffuses activation along associated edges, and finds content that is not similar enough in text but has historical associations with the question. The results of the two paths are then merged with semantic memory and handed over to the LLM to generate answers.

In the aforementioned "career choice" case, semantic retrieval can first locate the turns related to "career advice"; the association path will retrieve the "first meeting location" along the existing connections. This enables the system to complete cross-fragment, multi-hop recollection.

Figure | The semantic path first locates career-related memories, and the association path then retrieves location information along the learned connections.

Performance on the long dialogue memory benchmark

The research team evaluated HeLa-Mem on the LoCoMo benchmark. This benchmark contains long dialogues with an average of about 300 turns and about 9K tokens, and the questions cover categories such as multi-hop, temporal, open-domain and single-hop.

As shown in the figure below, when using GPT-4o-mini as the base model, HeLa-Mem achieves F1 scores of 40.14, 47.29, 29.70 and 51.89 respectively in the four tasks of LoCoMo: multi-hop reasoning, temporal reasoning, open-domain and single-hop question answering.

At the same time, HeLa-Mem only requires an average of 1010 tokens of retrieval context; MemoryOS requires about 2000 tokens, and the baseline that directly inputs the complete LoCoMo dialogue requires about 16910 tokens. Across the three base models of GPT-4o-mini, GPT-4o and Qwen2.5-3B, HeLa-Mem has an average ranking of 1.25 across the four tasks (the lower the value, the better).

Figure | Performance vs. token overhead comparison reported in the paper. The horizontal axis is the token cost (logarithmic scale), and the vertical axis is the average F1; the figure summarizes the results of GPT-4o-mini and GPT-4o.

Of course, this study still has some limitations. For example, the association weights of HeLa-Mem need to be gradually accumulated in continuous interactions, so it still faces the cold start challenge when there is little interaction history. The semantic knowledge refined by reflective consolidation will also be affected by the capability of the underlying LLM.

This article is from the WeChat official account "Academic Headline" (ID: SciTouTiao), author: Academic Headline, published by 36Kr with authorization.