Is the 76% performance improvement model-agnostic? Karpathy's 700-loop experiment reveals the biggest misconception in Agents

"The biggest misconception in today's AI industry is that everyone is pushing Agents to rush into tasks as quickly as possible, without first thoroughly understanding the underlying model and system mechanisms."

This is a viewpoint shared by Andrej Karpathy, former co-founder of OpenAI and current pre-training researcher at Anthropic, in last year's podcast "AGI is still a decade away".

He gets straight to the point, warning the industry to place greater emphasis on the model framework. "OpenAI fell into a similar trap in its early days. When fundamental capabilities were not yet mature, we rushed to make Agents tackle complex tasks, wasting five years of time in vain."

This statement still holds true today.

Over the past year, Agents have become the most crowded track in the AI circle. Everyone is connecting models, integrating tools, and stacking workflows, as if a real independently working Agent will naturally emerge once a few more layers of capabilities are added.

But the problem lies precisely here: many Agents do not fail because "the model is not smart enough", but because of that more fundamental, more tedious, yet more critical system engineering beyond the model.

Recently, an experiment titled "Don't Train the Model, Evolve the Harness" by Hugging Face machine learning engineer Joel Niklaus shows that:

Using the same DeepSeek-v4-pro, without modifying model weights and only optimizing the outer execution mechanism of the model, can significantly improve the Agent's performance on professional tasks (the pooled score jumps from 3.5% to 80.1%).

Meanwhile, Karpathy's 90k-star open-source project AutoResearch (Loop Cycle) further illustrates that the true value of AI is not generating answers in a single attempt, but lies in the cycle of "proposing revisions, running experiments, automatic evaluation, and retaining progress".

Getting the answer right once relies on the model itself, while continuous iteration depends on the combined capability of the model and the outer execution system.

Optimizing Only the "Outer Shell" Can Lead to Skyrocketing Performance?

If you are still obsessing over prompt engineering or complaining that open-source models are not smart enough, this recent experiment from Hugging Face may subvert your perception.

To explore where exactly the bottleneck of Agent performance lies, Niklaus used the open-source model DeepSeek-V4-Pro and conducted experiments on a specific legal Agent benchmark test.

Task performance of DeepSeek-V4-Pro under the original LAB harness. The significant differences across different legal practice domains and tasks indicate that model performance is first affected by the outer execution mechanism, not just the bare model capabilities.

Niklaus did an extremely restrained thing: completely freezing the model's weights, meaning no fine-tuning or continued training at all. The only variable is the code and execution logic wrapped around the outer layer of the model, the so-called Agent Harness (outer execution mechanism).

It is both amusing and thought-provoking that in the initial tests, the model scored 0% under some outer mechanisms. The research team uncovered the truth: the model's legal reasoning process was completely correct, but it always saved the results to the wrong file names, making the test program unable to read the results at all.

This means that the 0 score was never measuring the model's intelligence, but testing whether the Harness is effective.

Since the problem lies in the "outer shell", how much can performance improve by optimizing it?

The experimental data gives a striking answer. With the same DeepSeek-V4-Pro, the same set of tasks, and the same evaluator, simply switching between five different outer execution mechanisms caused the pooled score to fluctuate drastically between 3.5% and 80.1%:

Rising from 3.5% for mini-swe-agent, 23.2% for Goose, 45.4% for Pi, all the way to 63.4% for the original LAB harness.

Performance comparison of different outer execution mechanisms on held-out test tasks, with a fluctuation range as high as 76 points.

After about 22 rounds of automatic iterative code optimization, on 100 held-out test tasks, the pooled score finally increased from 63.4% under the original LAB harness to 80.1% (a 16.7 percentage point improvement), and the all-pass rate also rose from 0% to 5.0%.

This open-source model, whose weights were never modified at all, matched the performance of the industry-leading closed-source model Claude Sonnet 4.6 just by optimizing the Harness, while its operating cost was only 1/7 of the original.

More critically, when this optimized Harness was migrated to the smaller sibling model DeepSeek-V4-Flash, it still delivered a 14.4-point improvement.

This proves that the execution mechanism at the code level is far easier to solidify and migrate across models than prompt tuning.

So what exactly is Harness?

Akshay, former engineer at Lightning AI and co-founder of DailyDoseOfDS, once analyzed it this way:

A raw LLM is just a CPU with no memory or hard drive; if it is not the model itself, then it is the Harness — it is the operating system that manages memory, I/O, and drivers.

He believes that a production-grade Harness should contain 12 core components, covering every aspect from process orchestration, tool calling, hierarchical storage, to context management and error handling.

For example, the common "context rot" problem: when the model places key information in the middle of the context window, performance can drop by more than 30% directly.

A mature Harness already has a complete set of engineering methods to solve this problem — compressing history records, masking old outputs, dynamic retrieval, and proxy summarization. Its core is to obtain the best results with the minimum number of high-information-density Tokens.

Based on the above experiments, Niklaus drew a core conclusion:

What a Benchmark measures is never the bare model, but the combined capability of "model + Harness". The biggest performance improvements often come from simple automated steps like file handling, rather than consuming massive Tokens to tweak prompts.

So, does this mean that in the future we do not need to train models anymore, and can just stack Harnesses?

The answer is no. The improvements brought by Harness have their limits, and the remaining gaps still need to be filled by the underlying capabilities of the model. But the core problem right now is: most people have not even built a passing Harness, yet they are eager to stack larger models.

When you cannot even tell if there are vulnerabilities in your testing tools, you will never be able to determine whether the performance bottleneck lies in the model itself, or in your bug-ridden outer code.

700 Automatic Iterations, 5x Performance Surge

Since the execution mechanism beyond the model is so important, how exactly can we build a continuously evolving Agent?

AI researcher Codila further refined Karpathy's 90k-star AutoResearch project, condensing 630 lines of open-source code into a five-step "Loop Engineering" methodology.

This method has now gained over 2 million views on X, because it hits a core pain point: do not expect Agents to get it right on the first try, but approach the optimal solution through low-cost continuous trial and error.

The logic of this loop seems extremely simple:

1. Based on a fine-tuned model, write a document that tells the agent which directions to explore and which constraints to follow.
2. Only allow the agent to modify training scripts; the evaluation and scoring scripts are locked and cannot be modified. This prevents the agent from arbitrarily lowering standards to get higher scores.
3. Let the Agent run in the following loop: propose changes → train → evaluate → retain good improvements, refine and discard bad ones

Karpathy ran the model he manually tuned with 20 years of experience for two days, and the result was astonishing:

The Agent automatically ran 700 experiments, identifying 20 code improvements that he himself had overlooked. For example, a missing scalar multiplier in the attention mechanism caused attention to be excessively dispersed across multiple "heads". This kind of meticulous optimization that requires massive patience will exhaust humans after a dozen rounds, but Agents will not.

Tobi Lutke, CEO of Shopify, tested it overnight with an internal model, and woke up to find a 19% improvement in quality, while the optimized model size was also reduced by half.

For this loop to operate effectively, the core is not how smart the agent is, but whether three basic elements are properly implemented:

Verifier: A mechanism that automatically judges whether results are good or bad. Without it, the Agent is grading its own homework, and running it ten thousand times makes no sense.

State file: Records the results of each attempt, avoiding restarting from scratch after the page is closed or the process is rebooted.

Stopping condition: Must stop when the goal is reached or the maximum number of rounds is hit, otherwise it will keep running and burn through your Token budget.

Of course, not all tasks are worth the effort of building a full Loop cycle. Codila proposed a four-point eligibility criterion:

• First, high task frequency (at least repeating once a week, otherwise the construction cost cannot be recovered; one-off tasks only need a good prompt);
• Second, verification can be automated (no manual intervention required for reading and fixing);
• Third, the Token budget can absorb redundancy (loops inevitably involve a large number of retries, which a small budget cannot support);
• Fourth, the Agent can access the real operating environment (cannot iterate blindly with eyes closed).

Please note that none of these four conditions can be missing, otherwise the cost will far exceed the benefit.

Following this line of thought, researchers further proposed "Bilevel Autoresearch", as detailed in the paper: 《Bilevel Autoresearch: Meta-Autoresearching Itself 》.