Nvidia Stuns the Industry by Open-Sourcing Harness, Allowing AI to Radically Overhaul AI on Its Own

After OpenAI, Jensen Huang has made his move!

Finally, NVIDIA has officially open-sourced its self-developed Harness, SoL-Pi, to the public.

It takes Pi as the base, and builds a brand-new Harness efficiency enhancement layer on top of it, enabling AI to achieve self-iteration and optimization.

In this strict automated pipeline, AI itself has become a researcher.

They observe how Agents work, identify which steps waste Tokens unnecessarily, propose solutions, fix bugs, and then send the solutions to testing.

The entire AI search starts from 152 candidate directions, and finally leaves four killer architecture mechanisms.

The actual test results are staggering —

Token consumption is reduced by up to 64%, and API call costs plummet by 50% to 54%.

GitHub portal: https://github.com/NVlabs/SoL-Pi

In professional research scenarios, you can directly save $8.75 to $13.5 per hour. NVIDIA has open-sourced the switch that makes AI cost less.

The configuration is extremely simple: you only need one line of code:「pi install git:github.com/NVlabs/SoL-Pi」 to install it on the existing Pi.

Codex Speeds Up Code Generation

Half of the Tokens Are Spent on Idle Operations

Before letting AI recursively improve itself, can we first cut the cost of its self-improvement process?

Harness, the full runtime framework used by the model when executing tasks, is the best solution available at present.

The model is responsible for reasoning, and Harness is responsible for organizing tools, context, execution feedback and task processes, so that the model can read files, modify code, run tests, and continue to act based on the results.

With the same model, the work efficiency can vary greatly when placed in different Harnesses.

The current problem is that the tasks of programming agents are getting longer and longer.

From completing a few lines of code, to fixing problems across repositories, to working autonomously for a long time, a single task can last for several hours.

At this time, some trivial Token waste that is usually unnoticed will accumulate continuously.

After modifying a file, even though the next step is to run the test, the model will still perform another round of reasoning. A large file that has already been read will still be repeatedly carried in subsequent requests.

Among thousands of lines of logs, there may only be a few lines that actually affect decision-making, but the expensive model has to read from the very beginning.

Recursive Self-Improvement (RSI) cannot avoid this overhead either. Every time AI tries to improve the system, it consumes Tokens. Failed solutions also generate costs as usual.

The emergence of SoL-Pi is designed to solve exactly this problem.

The final results show that compared with the base Pi, SoL-Pi uses 45% to 49% less tokens, costs about one third less, and retains about 94% of the average score.

Compared with the original harness that comes with Codex and Claude Code, it uses 35% to 64% less tokens, and the listed cost is 50% to 54% lower.

How did SoL-Pi achieve such impressive performance? Now let's break down its four core mechanisms in detail.

AI Evolves by Itself, Four Core Features Unveiled

NVIDIA's pipeline finally retained four mechanisms, all of which avoid modifying the main program and are specially designed to eliminate repetitive work.

Layer 1: Action Fusion, Complete Editing and Verification in One Call

Action Fusion targets the redundant model decision-making link between two tool calls.

In the runtime trace of the basic Pi, the most common sequence is modifying a file, receiving the result, and then calling a command to test or build.

Since the subsequent commands are already very clear, there is huge room to compress this round of back-and-forth model decision-making in the middle.

Action Fusion keeps one edit and its subsequent commands in a local sequence, eliminating the intermediate model decision-making step.

Action Fusion directly encapsulates the editing and subsequent commands into the same local execution sequence.

Harness completes the modification and runs the commands at the underlying layer, then returns the merged results at one time. Tests are executed normally and results are obtained as expected, but the intermediate model request is successfully eliminated.

This is equivalent to combining two separate requests into one efficient closed-loop operation.

Layer 2: Online Context Compaction, Dynamically Calculate Costs by Subtasks

Online Context Compact solves the core pain point of "when to trigger compaction".

The longer the context is, the more the historical materials will become a burden on computing power and capital.

However, compaction also has a cost: rewriting the context may break the existing KV-cache reuse, and you need to pay the processing cost again.

Therefore, compaction that is performed too early may not save money, and past strategies usually try to delay the compaction action as much as possible.

Online Context Compaction regards completed subtasks as potential compaction points, and waits until subsequent requests can cover the rewriting cost before executing compaction.

SoL-Pi reconstructs the judgment timing: it splits large tasks into subtasks, and re-evaluates after each step is completed.

Its core logic is precise calculation: the system will only perform the compaction operation when the "expected future cost savings" can cover the "cost of this rewriting".

Layer 3: Output Archiving and Indexing, On-Demand Retrieval of Large Files

ObservationPack is specially designed to deal with the "repeated billing" problem caused by large segments of tool output.

If a large file or an extra-long result stays in the context after being read for the first time, it will be carried as-is in every subsequent round of requests, continuously consuming cache resources.

ObservationPack replaces the repeated full output with a stable handle, while retaining the original content for page-by-page retrieval.

The approach of SoL-Pi is to archive the full content directly to the local disk, and only keep a short stable handle and a small excerpt in the context.

This is just like archiving the original document directly after reading a long report, only keeping the index and key abstracts at hand. When the model needs to check the details, it can directly retrieve the content page by page according to the index.

Layer 4: Small Model Pre-Screens Logs, Introduce Strict Evidence Verification Mechanism

The core of Evidence-Preserving Reducer is to delegate the first pass of reading long logs to a lower-cost model.

Build and test logs often have tens of thousands of words, and usually only a few lines of error reports affect the next decision. It is extremely expensive for cutting-edge large models to read the full text every time; but if you directly hand it over to a small model for summarization, hallucinations are likely to be introduced.

The Evidence-Preserving Reducer only accepts the compact diagnostic receipt when it can verify its evidence against the archived logs.

SoL-Pi adds a strict line of defense in the middle — evidence verification. After the auxiliary model finishes reading the logs, it must output a compact diagnostic receipt.

The system will compare this receipt with the archived original logs item by item. Only the original text that matches perfectly will be forwarded to the cutting-edge large model.

This ensures that the main model only receives verified information, cutting off error propagation from the source.

152 Ideas Compete, Only 4 Are Left

Why these four mechanisms? This comes down to NVIDIA's real ambition: RSI (Recursive Self-Improvement).

Since AI can modify code, it should also be able to transform the "system that produces AI".

But RSI is extremely costly, and every trial and error incurs token fees. So NVIDIA changed its goal — don't rush to make AI smarter first, make it more cost-effective first.

Compared with directly improving task scores, token efficiency is harder to cheat. It is easy to overfit when brushing task scores by writing rules for specific problems; but optimizations such as deleting duplicate contexts, compressing tool outputs, and merging invalid decisions can be migrated to different tasks and models.

Thus, an "Agent researches Agent" pipeline was launched:

The team first built 535 verifiable environments, then let AI propose 152 optimization directions, followed by three rounds of screening —

First, use historical traces to estimate the benefits, and directly eliminate those with no potential; then let AI modify the code and run experiments by itself, with the Reviewer Agent pointing out flaws;

Finally, the solution is frozen and verified on a completely isolated held-out task: performance cannot drop, and efficiency must be improved.

Out of 152 ideas, only 4 survived in the end, with an average of about 40 ideas needed to produce one valid mechanism.

Thus, the core method of SoL-Pi gradually becomes clear: let AI generate a large number of hypotheses, then automatically conduct experiments, eliminate unqualified ones, and verify, so as to screen out the few truly effective mechanisms.

Let the RSI Flywheel Spin on Its Own

Behind SoL-Pi is the Efficient AI team led by Han Song, Associate Professor of MIT and Director of Research at NVIDIA.

For them, how much token overhead SoL-Pi saves at present is just a phased footnote.

What they really value is how much larger this "AI researches AI" flywheel can spin.

On the official project homepage, the team also laid out two extremely ambitious long-term foreshadowings:

One is "Pretraining the harness".

The current 535 environments are still manually built. In the future, Agents will collect tasks from the entire network, build environments, perform verification, and update their own Harness.

Once computing power and environment diversity are increased, Harness is very likely to develop its own Scaling Law.

The other is "Efficiency for efficiency".

Use the more cost-effective Harness to run larger-scale, lower-cost automatic research loops, so as to find more efficient mechanisms and form a compound interest effect of cost reduction.

In this flywheel, humans are only responsible for providing prior directions in the initial stage; once the loop is entered, the whole process from research to verification requires zero human intervention; after AI works out the effective mechanism, humans come back to figure out what exactly AI has discovered, and reconstruct the rough machine-generated code into a version that meets industrial standards.

Nowadays, the Scaling Law of models is still being debated by various parties; but the Scaling Law of Harness is already being explored by some pioneers.

And the one running fast on this track is AI itself.

References:

https://nvlabs.github.io/SoL-Pi/

https://github.com/NVlabs/SoL-Pi

https://x.com/MaxForAI/status/2098050525279478059

This article is from the WeChat official account "AI Era", Author: ASI Revelation, Editor: Taozi, Moses, Published with authorization from 36Kr.