The biggest blind spot in world model evaluation has been broken through by this new benchmark.
Do world models really have memory?
If you're pouring a glass of water into a glass bottle, then turn your head to look out of the window and turn back, the water level in the bottle should have changed. This is common sense that any three - year - old child has: just because something is out of sight doesn't mean it has disappeared, let alone that it has stopped changing. Psychology calls this ability "object permanence", which is one of the earliest cognitive abilities established by humans.
However, for today's most advanced video - generation models, this task is not simple.
In the past year, "world models" have become one of the hottest keywords in the field of video generation. From Sora to many domestic large - scale video models, manufacturers have been vying to emphasize that their products are not just about "generating beautiful pictures", but about "simulating the operating laws of the real world". Among them, whether a world model has a cognitive ability similar to human "object permanence" is becoming the core criterion for measuring the modeling level: When an object temporarily leaves the field of view and continues to undergo physical changes during occlusion, can the model remember its identity, infer its state, and accurately restore it when it reappears?
For a long time, industry evaluations have failed to accurately test this ability: Most benchmarks only assess the consistency between frames when objects are continuously visible; the few tests involving occlusion mostly target static scenes (where the environment does not change at all while the object is absent), which cannot verify the model's memory and inference abilities for the dynamic world.
To address this issue, researchers from Harvard University, MIT, IBM, Boston University, Google, JHU, CMU, and the Kempner Institute have proposed a new diagnostic benchmark: MemoBench. This is the first "disappear - reappear" world - modeling evaluation benchmark for dynamic environments and has been accepted by the top computer vision conference ECCV 2026. Its first author, Haoyu Chen, is a first - year master's student in the field of computational science and engineering at Harvard University, under the supervision of Yilun Du, an assistant professor of computer science at Harvard University.
Paper title: MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
Paper address: https://arxiv.org/abs/2606.27537
Project homepage: https://memobench-team.github.io/
Code: https://github.com/MemoBench-Team/MemoBench
Dataset: https://huggingface.co/datasets/tonyc54/MemoBench
Based on 360 high - quality ground - truth videos, this benchmark, combined with an automated index and a semantic VQA evaluation system, systematically tested 10 mainstream world - generation models, clearly revealing the core shortcoming of current technology in memory consistency: None of them had an "object reappearance score" exceeding 0.6 (out of 1). That is to say, none of the models can stably make the disappeared object "be remembered, recognized, and change correctly".
This result also led the research team to make a very direct core judgment: Even if existing video - generation models can generate visually completely coherent pictures, they can hardly correctly restore the state changes that an object should have undergone during the "disappearance period" when it reappears. This conclusion clearly reveals the significant gap between "generating realistic pictures" and "truly understanding the world", and also provides a quantifiable diagnostic criterion for the research and development of the next - generation world models.
"Remembering" the world: A must - answer question for world models
In recent years, video - generation technology has been iterating rapidly. Models can now generate high - resolution, temporally coherent dynamic pictures and are increasingly regarded as "world simulators", providing environmental inference capabilities for scenarios such as autonomous driving, robot manipulation, and embodied intelligence.
However, the operation of the real world does not stop when the line of sight leaves: Ice cubes will continue to melt, flames will continue to burn, pedestrians will continue to walk, and paper touched by liquid will continue to be soaked...
A real - world data sample from the MemoBench dataset
A truly qualified world model should have a cognitive ability similar to human "object permanence": Even if an object temporarily leaves the field of view, it can maintain the representation of its identity, position, and state and correctly infer its changes during occlusion.
This is precisely the core blind spot of the current evaluation system. Most existing video - generation benchmarks only assess the consistency between frames when objects are visible throughout; the few tests involving occlusion mostly target static scenes (where the environment does not change at all while the object is absent). Therefore, they can pass the test by simply restoring the original appearance and cannot verify the model's memory and inference abilities for the dynamic world.
We still cannot confirm: When an object reappears in the picture, does the model really "remember and update the world state", or does it simply "regenerate a seemingly reasonable picture"?
MemoBench: A new benchmark to fill the gap in world - modeling evaluation
Core design: The "Visible - Disappear - Reappear" three - stage paradigm
Existing video - generation benchmarks (such as VBench, WorldScore, etc.) mainly evaluate visual quality, physical consistency, or camera motion control. They almost only evaluate objects that "continuously appear in the picture". If an object occasionally moves out of the picture, existing benchmarks mostly conduct evaluations in static scenes, lacking an investigation of dynamic changes during occlusion.
To answer this question, MemoBench has established a core evaluation paradigm of Visible–Disappear–Reappear (which can be abbreviated as V - D - R), using a simple three - stage structure to accurately detect whether the model maintains a persistent state representation of the object.
Each test sample strictly follows a unified physical logic:
Visible: The target object is completely within the field of view and is in the process of continuous physical change, such as a walking person, a melting ice cube, or pouring powder. The model can observe the initial state and change trend of the object.
Disappear: The camera completely shifts its line of sight through movements such as translation, turning, or a U - turn, and the target object completely leaves the field of view. During this period, the physical process does not stop and will continue to evolve according to natural laws.
Reappear: The camera turns back to the corresponding area, and the target object re - enters the picture. The model needs to accurately restore the updated state of the object, including its position, shape, and progress of physical changes, in alignment with the real evolution result.
Overview of MemoBench. Rows 1–2 show a synthetic "Visible–Disappear–Reappear" sequence and its camera trajectory; rows 3–4 show a real - world state - change sequence (powder pouring). MemoBench contains 196 synthetic clips and 164 real - world clips, which are evaluated through automated indices and VQA judged by LLM.
This seemingly simple paradigm tests the core ability of the world model: The model cannot simply generate pictures frame by frame but must maintain a continuously evolving world state outside the field of view.
It tests visual memory, state - updating, and dynamic - inference abilities simultaneously and is a direct test of the "real - world modeling level".
Data construction: Combining synthetic scenes and real materials
Based on this paradigm, MemoBench has constructed two parallel data pipelines for synthetic and real - world data, including a total of 360 high - quality ground - truth videos with a resolution of 1920×1080, along with complete geometric annotations and evaluation tools:
Synthetic subset (196 clips): Rendered and generated in Unreal Engine 5, covering 5 major categories of environments and 14 sub - scenes, including various complex camera trajectories such as U - turns, forward movements, head rotations, and vertical movements. Each video is accompanied by frame - by - frame RGB images, metric depth maps, camera intrinsics, and camera poses, which can support accurate geometric - level evaluations and focus on testing the spatial memory ability under large - angle changes.
Synthetic data: 14 scenes under 5 categories rendered by Unreal Engine 5.
Real - world subset (164 clips): Shot in a controllable indoor environment, covering 7 major categories and 30 physical state - change processes, including dissolution, combustion heat change, diffusion and absorption, chemical reactions, viscous flow, foam evolution, and mechanical deformation. These processes that rely on real material properties such as viscosity, elasticity, and heat conduction are difficult to accurately simulate by game engines and are specifically used to test the accuracy of the model's memory of real physical dynamics.
Real - world data: Shot in a controlled indoor environment, covering 30 physical state - change processes in 7 major categories
All samples have been manually annotated, accurately marking two key frames of the object "completely disappearing" and "completely reappearing", which are used to precisely divide the three stages of Visible (V), Disappear (D), and Reappear (R) as the basis for calculating all subsequent evaluation indices.
The data compilation process of MemoBench
How to score: Dual - line evaluation of automated indices + VQA
MemoBench has designed two complementary evaluation schemes: automated quantitative indices and VQA semantic evaluation, considering both low - level pixel fidelity and high - level semantic correctness to avoid the one - sidedness of a single index.
The automated indices cover three major dimensions, and all core scores are normalized to the range of 0 - 100, testing the world model's:
General video quality: Including visual quality, motion smoothness, object identity consistency, and 3D geometric consistency, which measure the basic generation ability;
Memory - specific indices: This is the core of the evaluation. The object reappearance score (ORS) uses the SAM - 3 text - driven segmentation model to detect the existence and confidence of the target object in the reappearance stage, directly measuring whether the model can make the target "come back correctly"; it also includes pixel - level fidelity (PSNR, SSIM, LPIPS) and camera controllability indices at different stages, which can accurately locate the specific links of performance degradation.
Prompt fidelity: Measures the matching degree between the generated content and the text prompt through ImageReward.
VQA semantic evaluation focuses on the high - level semantic rationality that is difficult to capture by automated indices, covering four diagnostic dimensions:
Instruction following: Whether the camera movement, object trajectory, and event sequence in the prompt are accurately executed;
Object - background consistency: Whether there are shape drifts, identity switches, or scene mutations in the foreground and background elements;
Memory continuity: Whether the model maintains the identity, trajectory, and state of the object during its disappearance, which is the dimension most directly corresponding to the "disappear - reappear" paradigm;
Physical rationality: Whether the movement, gravity, and light and shadow conform to physical laws.
The VQA evaluation process. The large - language model generates 24 polarity - balanced yes/no questions (6 for each dimension) based on the prompt and the first frame. The questions are screened through real - world annotations and failed - segment evaluations and then verified by manual review. The final question library is applied to each generated video to obtain the passing rates of the four diagnostic dimensions.
To ensure the reliability of the evaluation, all questions have undergone three rounds of screening: ground - truth filtering, failed - case filtering, and manual cross - verification. The overall consistency between manual and VLM judgments reaches 92.9%, and the Cohen's κ coefficient is 0.85, indicating a high degree of credibility of the evaluation results.
What is the real level of current world models?
The research team tested 10 current mainstream world - generation models on MemoBench, covering three technical routes: camera - controllable image - to - video (CI2V), explicit 3D perspective synthesis, and ordinary image - to - video (I2V). A series of key findings were obtained, clearly presenting the ability boundaries of current world - modeling technology.
The core conclusion given by MemoBench is: Currently, no model can reliably complete the "disappear - reappear" memory task.
The object reappearance scores (ORS) of all tested models did not exceed 0.6; among the models that truly execute the camera trajectory and make the object actually leave the picture, the semantic score in the "memory continuity" dimension is at most 55.6 points (out of 100. The score of LTX - Video, which is 57.0, is inflated because the camera does not move). That is to say, even the strongest model can only answer a little more than half of the memory - related questions correctly.
The automatic evaluation of 10 world - generation models on Mem