HarnessEval: Ushering in a New Era of Evaluation with Harness | Geespace Portfolio Updates
Evaluation defines the taste of technology.
How far a technology can go often depends on the ruler we use to measure it. Evaluation is never just about verifying experimental results, but also about setting coordinates for cutting-edge exploration. However, in the RSI (Recursive Self-Improvement) process oriented to the physical world, there has always been a lack of a reliable ruler for world models. Especially when facing physical causality, geometric consistency and observation realism, existing evaluation methods still struggle to give stable and credible automatic judgments. In contrast, humans are often extremely sensitive to anomalies in generated content: did an object disappear out of thin air? Is the motion after collision reasonable? Did the scene change suddenly? These judgments are almost instinctive for humans, but have always been difficult to be automatically realized by AI. The existing benchmarks also make it hard to see clearly where the model went wrong and why it made the mistake.
At MirroS, we have introduced Harness, a key concept in the LLM ecosystem, into the evaluation field for the first time.
Harness is not a simple code encapsulation, but a reliable execution framework. It organizes complex human workflows, such as problem decomposition, tool invocation and reasoning verification, into clear and executable steps, thus supporting a complete agent workflow. Human evaluation is essentially a set of workflows. When we evaluate a generated world, we do not simply glance at the image, but unconsciously complete a series of complex processes: locate objects, track their existence and changes over time, judge whether actions are reasonable, and then verify the causal relationships, geometric constraints and physical laws in it. We automate this process into an agentic benchmark via Harness: an agent coordinates the overall situation, dynamically derives multiple professional subagents according to specific cases, provides them with corresponding context and diagnostic tools, and examines the model output from different perspectives. This evaluation harness is just like Sherlock Holmes: it does not rush to a conclusion based on a single phenomenon, but constantly looks for clues, cross-verifies evidence, and organizes scattered information into a complete reasoning chain to finally form a judgment.
Thus, evaluation has evolved from a set of static Q&A rules to an intelligent system that can actively find evidence and make judgments. Its core is not a fixed evaluation pipeline, but a set of evaluation capabilities that can be invoked, combined and recursively expanded on demand:
Path Planning: Determine the required evaluation methods according to the case;
Problem Decomposition: Schedule multiple subagents to decompose the problem layer by layer;
Intention Understanding: Identify the action to be executed and its expected changes;
Evidence Search: Locate clues related to the judgment;
Tool Invocation: Use measurement and reasoning tools as needed;
Conclusion Formation: Organize complete evidence and reasoning chains.
HarnessEval-W: Applying HarnessEval to World Models
Today, we take World Models as the experimental field of HarnessEval, and officially open-source HarnessEval-W. We also invite the community to jointly participate in the construction of this new agentic benchmark workflow. First, we decompose the core capabilities of world models into three evaluation dimensions: observation quality, state transition correctness and world persistence, and build our overall evaluation system around these three dimensions.
Perspectives of Evaluation
Referring to the common definitions in existing research, we describe the physical world model as:
Predict the future state of the world based on historical observations and user actions.
All models with this capability, no matter what generation architecture they adopt, can be included in the evaluation scope of HarnessEval-W, including bidirectional video diffusion models, autoregressive video models, etc. This formal description also presents the basic evaluation logic of HarnessEval-W: whether the world is correctly presented, whether actions lead to correct state transitions, and whether the world can remain continuous and consistent over time. Thus, the following three core evaluation dimensions are naturally derived.
1. Observation Quality
It measures whether the visual observations output by the model are credible. It focuses on the perceptual-level quality of the generated video itself, including visual continuity, structural rationality and realism.
2. State Transition Correctness
It focuses on whether the model can respond to given actions in the right way at the right time, and make the world state change as expected. We further consider three types of state transitions: Exploratory Transition mainly corresponds to the movement of the observer in the world, such as changing the camera position, perspective or observation area. Intentional Transition refers to the user's active request to change a certain entity, relationship or event, such as moving an object, opening a door, or requiring a certain character to perform a specific action. Physical Transition focuses on whether the world evolves in accordance with reasonable physical laws after physical control is applied, such as collision, pushing, falling, bouncing or other dynamic processes.
3. World Persistence
It focuses on whether the model maintains a continuously existing and internally consistent world as the world keeps evolving. We focus on three typical scenarios. Drift Resistance: Test whether the overall layout, style and object appearance of the world remain stable during long-time generation, instead of gradually drifting meaninglessly as the generation time increases. Revisit Consistency: When the observer temporarily leaves a certain location or object and then returns later, whether its state and attributes are still consistent with the previous ones. Offscreen Evolution: When a dynamic process temporarily leaves the field of view, whether it can continue to evolve in accordance with time and physical laws, instead of being frozen, reset or arbitrarily changed when it is not visible. World persistence does not mean that all content in the world must remain unchanged. It requires stable attributes to be consistent, while dynamic states must also evolve along a continuous and reasonable trajectory.
Harness-based Evaluation System
The evaluation of world models naturally relies heavily on specific situational contexts. Different cases may correspond to completely different environments, action types, time structures and observation states, so it is difficult to cover all situations with a set of fixed questions or static indicators.
The core of HarnessEval-W is an agent capable of dynamically formulating evaluation strategies. For each case, it first understands the current world and evaluation objectives, then plans the evaluation path, schedules subagents with different professional capabilities, and gradually forms a verifiable scoring judgment by combining the evidence collected by them. The entire evaluation process will be adjusted according to the current specific world. In other words, HarnessEval-W does not preset a fixed evaluation process, but allows the evaluation path to be dynamically generated along with the case itself. It not only needs to answer "whether this result is correct", but also further judges: is the evidence we collected really sufficient to answer the current evaluation question? Therefore, HarnessEval-W can provide an interpretable, reproducible and on-demand customized evaluation process for each case.
Skill Selection
The first step of evaluation is not to immediately start Q&A scoring, but to first determine: What exactly questions should be asked for this case? HarnessEval-W first understands the context and objectives of the current evaluation case, and then invokes the skills that are qualified and capable of evaluating this case. For example, in the robot operation scenario, whether the action is executed correctly and whether it follows reasonable physical laws is often more important than whether the picture has a "cinematic feel". HarnessEval-W will not mechanically apply exactly the same evaluation indicators to all cases, but select truly relevant evaluation capabilities according to the scenario.
Skill Decomposition
After selecting the evaluation skill, the system will further decompose it into smaller, clearer and more measurable sub-problems, and then assign them to appropriate subagents or tools respectively. For example, to judge whether a collision is reasonable, HarnessEval-W will further ask: Where is the target object? Did they make contact at the right time? How did the velocity change before and after the collision? Tools such as target tracking, time sequence verification and velocity calculation will find answers respectively. The evidence returned by each subagent will finally be uniformly summarized and verified by the main agent to form the final conclusion. If the problem is still too complex, the subagents can also further schedule new subagents for layer-by-layer decomposition, until each task is clear enough to be directly verified through tools or visual evidence.
From a Single Score to a Traceable Evidence Chain
The ultimate goal of HarnessEval-W is not just to output a numerical score. We hope that the benchmark will eventually form a hierarchical evidence tree, which completely records what has been tested, which specific tool provides the visual evidence, and the complete logical chain supporting the final score. With this executable, traceable and verifiable evidence chain, researchers can not only accurately locate the error source, but also improve the model design and training in the next stage accordingly.
Construction of Evaluation Dataset
A truly effective benchmark must cover sufficiently complex and diverse environments to continuously explore the capability boundaries of world models. To this end, we also adopt a set of agentic pipelines to construct evaluation cases, so that the benchmark can be scaled up.
The entire process starts from a predefined scene taxonomy. First, we sample the initial configuration of the world, determine attributes such as environment, objects, spatial structure, visual style and observation perspective; then we select evaluation dimensions for the case, especially the state transition and world persistence indicators defined above.
According to these conditions, an image generation agent generates the initial frame of the world; an action planning agent then understands the content of the picture and designs specific actions around the target evaluation dimensions. For example, if we need to test Revisit Consistency, the planning agent can design a camera trajectory of "first leave the current area, then detour back to the original position". Finally, we introduce an independent verification agent for quality control. It will check both the initial image and the action design to confirm that the environment, foreground, background and actions are coordinated with each other. Candidate cases that fail to pass the verification will be automatically returned and regenerated until the evaluation requirements are met.
Through this closed-loop process, the benchmark can be continuously expanded while maintaining the necessary complexity and verifiability.
What's Next: Let Agentic Benchmark Evolve Continuously
The world is evolving, and evaluation should evolve along with it.
Evaluation should never be a set of static evaluation standards. As world models continuously enter more complex scenarios and demonstrate stronger capabilities, the evaluation system itself must also expand its boundaries synchronously. HarnessEval-W is just our starting point for exploring the new paradigm of realizing agentic benchmark through Harness. As an open-source project, we also sincerely invite researchers and developers in the community to participate together, so that the agentic benchmark can continue to grow in real collaboration and iteration.
Next, we will focus on three directions:
Test-Time Scaling
LLMs generate complex reasoning trajectories during inference, and use external tools such as MCP to complete long-term tasks. The evaluation process itself must also obtain corresponding scaling capabilities. Traditional benchmarks often assume that the evaluation cost of each sample is fixed, while agentic benchmarks allow us to invest more computing resources at test time: deeper search, multi-step verification and tool invocation can be performed during evaluation. As the models under test become more and more powerful, the evaluation system also needs to synchronously improve its own capabilities through more reasoning and verification calculations, so that the capabilities of the evaluation model can always catch up with the cutting-edge generative models.
Scaling Skill Libraries
Future world models will cover a wider range of application scenarios and generate environments with more complex physical processes and finer-grained physical details. Therefore, the agentic benchmark model also needs to have a continuously expanding set of capabilities. We will continue to build a rich skill library: when the model simulates more refined physics and more diverse scenarios, the evaluation agent can dynamically retrieve and combine specialized skills from this growing library to accurately evaluate the physical understanding ability in the generated world. The capabilities of the benchmark are no longer hard-coded in a fixed set of codes, but can continue to expand as the skill library grows.
RSI (Recursive Self-Improvement)
Furthermore, we hope that the benchmark itself has the ability of self-improvement. When HarnessEval-W encounters an out-of-distribution (OOD) scenario while evaluating a cutting-edge world model, it should not simply output a low-confidence score. On the contrary, the system should first: Evaluate the Evaluator.
It needs to judge: does the current uncertainty come from the model under test, or because it lacks the corresponding skill itself? If an internal capability gap is found, the system can further expand its skill library by introducing new external skills, learning new tool usage methods, or even constructing new evaluation capabilities through autonomous exploration. The new capabilities will then re-enter the evaluation Harness to support more complex evaluation tasks in the future.
What we hope HarnessEval-W has opened up is not just a new world model benchmark, but a redefinition of the benchmark paradigm with Harness: capable of reasoning, capable of verification, and capable of evolving together with the objects being evaluated.