Solid research evidence proves that world models do not understand the laws of physics at all, and their "ChatGPT moment" is still far away.

In recent years, World Models built on video generation models have become one of the most high-profile research directions in the field of artificial intelligence. Video generation models including Sora, Genie, Cosmos and V-JEPA all point to a common hypothesis: if they are exposed to a sufficient volume of videos, they may gradually develop "physical intuition" and thus gain an understanding of the physical world.

However, the problem is that existing benchmarks can hardly verify whether this hypothesis holds true.

Do video generation models actually follow the laws of physics, or do they merely produce results that look correct? Can they think with physical laws like Newton, instead of only relying on "intuition" like Aristotle?

To address these issues, the S-Lab team from Nanyang Technological University and their collaborators have proposed Apple-π, the first video model evaluation benchmark centered on physical laws. The results show that current video generation models still lag far behind a reliable general world simulator, as even the SOTA model only scores 0.473.

Paper link: https://arxiv.org/abs/2607.16401

The research team stated that even if existing models can generate seemingly reasonable motions based on the labeled first frame, it does not mean that they truly understand object relationships, have mastered physical laws, or can consistently generate results in accordance with consistent motion laws.

This work aims to help future video generation models truly learn to understand and simulate the world in accordance with physical laws, but the "ChatGPT moment" that belongs to world models is still far away. To achieve this goal, researchers still need to endow video world models with stronger scene understanding capabilities, richer physical reasoning data, and specialized post-training oriented to reasoning tasks.

Apple-π: A Benchmark Anchored in Physical Laws

Apple-π is the first video model evaluation benchmark that takes physical laws as the core. It requires the model to perceive relevant physical quantities, state the governing laws, deduce subsequent motions that conform to the laws, and locate at which step the reasoning failure occurs.

Figure | Overview of Apple-π.

The specific design is as follows:

1. Video Dataset: Orchard contains 400 classical mechanics videos, where physical laws and initial conditions are determined first, then corresponding videos are generated or filtered. The tasks are divided into two categories: single law and multi-law combination, covering gravity, collision, inertia and multi-segment motion connection, to examine whether the model can continuously update the position and velocity of objects.

Figure | Overview of Orchard.

2. Benchmark Protocol: Apple-π splits Orchard cases into three types of tasks: perception, formulation and deduction, with five sub-tracks for evaluating the model's physical reasoning capabilities in separate dimensions. Perception-Text is used to reproduce the numbers and annotations in the frame, Perception-Graphics is used to locate and retain experimental objects; Formulation-Text selects the governing equation from candidate physical formulas and substitutes variables, Formulation-Graphics predicts the object position at the target moment and marks the velocity; Deduction generates a complete motion video based on the initial conditions of the first frame.

Figure | Apple-π Benchmark Protocol.

3. Evaluation Suite: Apple-π adopts a hybrid scoring mechanism. Multimodal Large Model scoring is used to evaluate the rationality of responses, format compliance and picture clarity; in tracks with object masks and trajectory ground truth, physical ground truth metrics are used to quantify object positioning, trajectory and velocity errors, to judge whether the motion process conforms to physical laws. The scores of all tracks are uniformly normalized to the range [0,1].

All Types of Models Show Limited Performance in Physical Deduction

The research team evaluated 11 representative models on the complete Apple-π benchmark. The overall results show that video pre-training, reasoning supervision and unified understanding can all improve the perception and formulation capabilities of models; however, all types of models show very limited performance in physical deduction.

Specifically, proprietary models generally outperform basic open-source video models, and training with large-scale, high-quality videos can distill useful physical priors; VBVR-Wan2.2, which is fine-tuned for video reasoning, also delivers more competitive performance among video models; the significantly leading results of GPT Image 2 and Nano Banana 2 indicate that explicit understanding and controllable visual generation contribute to annotation reading, object grounding and final frame generation.

Even the SOTA model only scores 0.473, and physical motion reasoning remains the main bottleneck at present.

Figure | Main Results.

In addition, to further locate the problems of models in different reasoning links, Apple-π splits the evaluation process into three stages: perception, law modeling and dynamic deduction, to examine the model's capabilities in perception, alignment, reasoning and generation.

The results show that model errors do not only occur in the final generation stage, but also in the intermediate physical reasoning links: even if some models can complete annotation reading and object positioning, they still have obvious deficiencies in physical rule selection, state update and subsequent motion prediction. They mostly rely on superficial visual pattern matching and have not yet formed a stable physical state modeling capability.

Limitations

Although Apple-π can evaluate the physical reasoning capabilities of video models in a more detailed manner, it still has some shortcomings.

For example, the coverage of Apple-π is still limited. It mainly focuses on classical rigid body mechanics, including gravity-driven motion, momentum-conserving collision, Newton's first law, and simple combinations of these laws, but it does not evaluate fluid mechanics, thermodynamics, electromagnetism, fracture, granular media, biological motion or quantum phenomena.

At the same time, the objects and scenarios covered by Apple-π are not comprehensive either. The dynamic objects in the dataset only include spheres, cubes, cylinders and cones. As a single fixed camera is used, it does not directly examine multi-view consistency or novel view generation capabilities.

The research team pointed out that the input setting of Apple-π still relies on the annotation of the first frame to fix the correspondence between the scene and physical quantities. Therefore, the benchmark mainly examines the model's physical reasoning capability under existing visual scenes, and does not test whether the model can construct a complete physical scene solely based on text.

In addition, the current evaluation method also has deficiencies. MLLM judges cannot fully verify whether the physical process is correct, so Apple-π needs to combine metrics such as mask, pixel and velocity, which are also affected by segmentation quality, frame differences and annotation noise.

Finally, Apple-π also has limitations in time normalization. The evaluation treats the fully decoded output as the physical duration required by the prompt by default, but video generation services may return a fixed-length video container that does not change with the requested duration, which will affect the evaluation results.

This article is from the WeChat official account "Academic Headlines" (ID: SciTouTiao), written by Xia Qiansi, and published with authorization from 36Kr.