New work from Yan Shuicheng's team: Scaling Harness Intelligence

The capabilities of an Agent are not entirely determined by the underlying base model. The Agent Harness, which is responsible for managing memory, formulating plans, orchestrating tool calls and triggering recovery mechanisms upon failures, is equally critical.

A SOTA base model, if placed under incorrect memory strategies or action protocols, will deliver significantly degraded performance; conversely, a well-designed Agent Harness can "amplify" the actual performance of a medium-sized model.

However, Harness design still relies on manual work, is limited to specific tasks, and is fundamentally difficult to scale.

In this work, the team led by Professor YAN Shuicheng from the National University of Singapore and their collaborators propose JIT-Agent, which transforms Harness construction from an artificially engineered, pre-designed artifact into a learnable capability, so as to synthesize task-adaptive Agent Harness in real time for any off-the-shelf Agentic LLM.

Paper Link: https://arxiv.org/pdf/2608.25593

The results show that in deep research, daily work, planning and workspace tasks, the Harness generated by JIT-Agent can continuously enhance the capabilities of its underlying backbone model, staying in the same performance range as mature Agent runtimes such as OpenCode and Claude Code, and expanding the cost-performance boundary.

The research team stated that JIT-Agent is the first model built specifically for instant Harness generation, establishing Harness intelligence as a trainable, transferable, and composable Agent capability dimension independent of model scaling.

Research Methodology

To enable stable generation of Harness by the model, the research team did not let JIT-Agent directly output completely unconstrained Agent programs, but established a fixed four-module protocol as follows:

• Memory: Determines how historical interactions and intermediate states are stored, compressed and presented to the model;
• Planning: Forms the next-step instructions, splits and adjusts task objectives;
• Action: Advances the action execution loop, handles tool calls and execution results;
• Capability: Orchestrates available tools and skills, determines which external capabilities to expose at different stages.

These four modules collectively form a composable Harness. The fixed protocol specifies the module interfaces, lifecycle, validation rules and execution semantics, while retaining the internal design space for each module.

Based on this, they constructed HarnessFactory, which unifies representative Agent Scaffolds of different types under the same set of interfaces. JIT-Agent outputs structured, executable modules instead of a natural language description for reading only. This approach not only reduces irrelevant language discrepancies, but also allows the generated results to directly enter the execution workflow.

To address the adaptivity, reliability and evolvability challenges in JIT-Agent training, the research team adopted the following three-stage training strategy:

Stage 1: Customizing Harness for Tasks

In this stage, the model learns how to select different memory, planning, action and capability orchestration methods according to the task structure through protocol-compliant examples generated by the teacher model, so as to understand the main bottleneck of the current task and which modules require special design.

For example, if a task relies heavily on external evidence, the Harness may need to pay more attention to the organization of retrieved content and working memory; if a task involves multiple execution phases, the planning and state management should be allowed to switch as the phase changes; if a task requires file modification, the action module and verification mechanism need to be organized around the file status.

Stage 2: Learning to Repair from Failure Trajectories

Generating only a structurally sound Harness is not enough. Model outputs may have compilation errors, interface mismatches or runtime failures, so the model also needs to have recovery capabilities.

This stage converts failed Harness generation processes into bounded repair trajectories. The model learns to identify the cause of failures and correct the relevant modules without modifying the entire system.

The key of this step is that reliability no longer fully depends on manual debugging by external engineers. The Harness generator itself is also trained to handle problems in its own outputs.

Stage 3: Continuous Self-Evolution

In this stage, they propose Evolutionary Group-Decoupled Policy Optimization, Evo-GDPO, which aims to enable the model to continuously use execution feedback to autonomously generate optimal Harness.

Different from the goal of only pursuing a single task score, Evo-GDPO normalizes rewards, latency and cost respectively. In this way, the model does not only look for "better-performing" Harness, but also takes execution efficiency and resource overhead into consideration, pushing the Model-Harness combination to continuously expand the cost-performance boundary.

After the above training, they finally obtained a Harness Intelligence model that can instantly customize, repair and evolve task-specific Harness — JIT-Agent-27B.

Experimental Results

The research team evaluated JIT-Agent across multiple Agent benchmarks and multiple model backbones.

The results show that JIT-Agent can bring significant performance improvements to different models. For example, with JIT-Agent acting as the Harness assistant, the score of DeepSeek-V4-Flash on DeepPlanning-Shopping increases by 24.8%, surpassing GPT-5.6; GLM-5.2 achieves SOTA scores on all OfficeBench, AgentIF, and DeepSearchQA leaderboards.

Meanwhile, the performance of Harness generated by JIT-Agent falls in the same competitive range as mature Agent runtimes such as OpenCode and Claude Code, bringing continuous performance improvements to the three model families of DeepSeek V4, Mimo-V2.5 and Qwen3.6.

What's Next?

The research team regards Harness Intelligence as another fundamental scaling dimension on a par with model capacity and inference computing power, and JIT-Agent is an important step toward this goal.

The longer-term opportunity lies in a model–harness co-design paradigm, where the base model and the operational structures that shape its memory, planning, action and tool usage are trained jointly.

However, the research team noted that future systems may not need to adopt the "deliberately radical form" in this study — the entire scaffold can be completely redesigned on the fly. Retaining a stable core harness while allowing the model to build, revise or replace specific components of it when the task requires may be a better choice.

They believe that as this paradigm matures, the capabilities of Harness synthesis, repair and evolution will gradually be internalized into the base model itself. Models will not only learn to act under a given Harness, but also learn to improve the Harness they rely on, further deriving broader research topics on adaptive interfaces, verifiable runtime modifications, and the collaborative scaling of models and execution systems.

This article is from the WeChat official account "Academic Headlines" (ID: SciTouTiao), authored by Academic Headlines, and published by 36Kr with authorization.