DeepSeek + Pi, the unbeatable powerhouse combo outperforms Claude Code? Pi's founder: I bet on this exact combination long ago.

On August 11, Mario Zechner, founder of Pi Harness, reposted a set of data:

Developer 0xEvan used Pi to call DeepSeek V4 Flash, processed nearly 1 billion input Tokens, achieved a cache hit rate of 99.93%, and finally spent only 2.65 USD. Without caching, the same usage was estimated to cost 132 USD.

On the same day, another developer Shantanu Goel stated that the cache hit rate of DeepSeek V4 Flash in other Harnesses usually ranges from 94% to 97%, but it can consistently reach over 99% in Pi. Mario commented on this: "This works especially well when using local models."

There are many other similar cases in the community.

This also reminds people of Mario's evaluation of the combination of Pi and DeepSeek V4 in May this year: "pi + ds4 == sovereign AI enterprise ready clearly." Translated into more intuitive expression, it roughly means: "Pi + DeepSeek 4, enterprise-grade AI is ready to go."

At that time, it was just a casual comment after he saw a developer run a terminal Tetris game with Pi paired with DeepSeek V4. Three months later, a public horizontal test unexpectedly added supporting data to this statement.

Paired with DeepSeek, did Pi outperform Claude Code?

Composio is a company that develops tools for AI agents, and it recently conducted a public comparative test. They used the same model DeepSeek V4 Flash, ran it in 8 different agent Harnesses respectively to complete 30 highly difficult agent tasks. These tasks require agents to take actions, call tools, and independently complete the entire workflow.

Pi Agent took the first place: it passed 20 out of 30 tasks, with a success rate of 66.7%; Oh My Pi ranked second with 17 passed tasks; Claude Code, Codex and Deep Agents all passed 16 tasks; Prime Agent passed 15 tasks, and there were 6 additional ungraded runs, 2 of which could not be scored due to scorer processing timeout, and the other 4 left no records; Hermes Agent also passed 15 tasks; OpenCode ranked last with 14 passed tasks.

With the same model, only by changing the Harness, the success rate rose from 46.7% to 66.7%, a full 20 percentage point difference.

The cost gap is even more obvious. Pi spends only 0.028 USD on average to complete one successful task, while Claude Code needs 0.195 USD, which is nearly 7 times that of Pi.

The median time for Pi to complete tasks is 132.2 seconds, which is slightly slower than 122.7 seconds of Claude Code and 129.7 seconds of OpenCode, but considering the combination of success rate, speed and cost, it delivered the most outstanding performance in this round of testing.

This test demonstrates the so-called "Harness multiplier effect": tools built around AI will amplify or weaken the actual performance of the model. Choosing the right Harness can make the same model more reliable and efficient at the same time; choosing the wrong Harness will lead to a significant drop in task success rate and operation efficiency even if the intelligence level of the underlying model is exactly the same.

Therefore, Composio emphasizes that models should not be evaluated in isolation; if an Agent leaderboard only lists the model name without specifying which Harness is used, the score is incomplete.

Why did the minimalist Pi win instead?

There is another noteworthy detail in Composio's test: Pi hardly added extra configurations, it adopted a brand-new, unmodified default installation, and only accessed the MCP server plugins required for the test. Apart from that, Pi had no custom settings, tuning or special configurations. It is such an almost out-of-the-box solution that finally passed the most tasks.

Then look at Prime Agent. It generated the largest sessions among the 8 Harnesses, some of which consumed as many as 3.5 million Tokens and made 33 tool calls. You can imagine it as an agent that hasn't really started working yet, but first lists a task list as long as a phone book for itself.

These sessions are too large, and the scorer timed out just to process them. Two runs could not be scored, and the other four left no records, so a total of six runs were not counted in the results. Even looking only at the valid runs, the number of tasks passed by Prime is only comparable to that of Hermes, but the time consumed is nearly twice that of Pi.

This set of data shows a clear contrast: Prime, which has the most complex functions and sessions, was eventually slowed down by its own operating burden; the lighter Pi passed the most tasks with lower overhead. At least in this test, adding more layers did not bring better results.

This also challenges the past idea that "the more configurations, the better the effect". People usually choose the largest model, stack every plugin, every extension and various complex function layers, and default that the more functions, the stronger the agent will be. This test provides another way of thinking: choose a fast, low-cost model, put it in a clean, lightweight Harness, and test the combination of the two with real tasks.

DeepSeek V4 Flash, as the name suggests, is a Flash model, which focuses more on speed and operation efficiency, and does not aim to win model intelligence competitions. The lightweightly configured Pi took the lead this time for a simple reason. Every additional layer gives the agent one more place to get lost; every additional tool gives it one more choice to make; every additional huge instruction file makes it read more noise before taking action.

Therefore, a clean Harness provides the model with a short path from receiving the task to completing it, while a bloated Harness makes it detour around. This is why the default installation can beat heavyweight configurations: the path is shorter, and there are fewer chances of going in the wrong direction.

How is the 99.9% cache hit rate achieved?

Pi itself is not a Harness specifically designed for DeepSeek. It is more like an open Agent base for developers: Pi allows developers to modify system prompts through extensions, filter dialogue history, customize context compression, and dynamically add, delete, start or stop tools; before the request is sent to the model, developers can even directly check and rewrite the final payload. This programmability leaves a lot of space for DeepSeek's cache optimization.

The DeepSeek API caches the prefix of the prompt in the request. If the Token sequence at the beginning of the next request is exactly the same as the previous one, the server will directly read these Tokens from the cache and charge at a price much lower than that of ordinary input Tokens. The price for a cache hit is much lower than that for a cache miss.

The key point is that this is a kind of prefix caching, and matching needs to start from the first Token. If the front part of the context changes, a large number of subsequent Tokens may no longer hit the original cache. The earlier the prefix changes, the more Tokens that will be "implicated" afterwards.

A typical Agent request usually contains system prompts, tool definitions, dialogue history and new content added in this round. Each time the Agent takes a step, it has to carry a large amount of context that has already appeared before. The longer the session, the more repeated content there is, which theoretically is more suitable for caching. But if the Harness reorganizes this content every round, adds new timestamps, changes the order of tools or rewrites the historical summary, no matter how long the context is, it will be difficult to be reused stably.

This has also spawned a number of Harness projects specifically optimized for DeepSeek caching. Open-source Reasonix is a terminal programming Agent designed around DeepSeek prefix caching, and its caching performance has attracted the attention of many developers. A developer who prefers Pi even specially made DeepPi, trying to port the caching optimization method of Reasonix to Pi. He claims that when DeepPi calls the DeepSeek API, the cache hit rate can stably reach 99.7% to 99.9%.

I tried to port some of Reasonix's performance advantages into the Deepseek package optimized for Pi. It only activates when using the Deepseek API, but after activation, my cache hit rate is stable between 99.7% and 99.9%.

The core design principle of Reasonix is: keep the front end of the context stable, use appending instead of modification, and minimize the change cost.

In terms of specific implementation, Reasonix injects a streamlined and stable environment summary at startup, instead of regenerating it in each round of dialogue. Outdated tool outputs are truncated and cleaned before triggering summary compression, so a large number of results generated by a cat command 20 rounds ago will not remain in the prompt prefix all the time. Reasonix also documents the Schema contracts of built-in tools and conducts regression reviews when changes are made, because once the tool definition is readjusted without notice, it will cause cache invalidation without any obvious abnormality on the surface.

In the dual-model mode, the execution model and the planning model run in their own independent and cache-stable sessions respectively, instead of being interleaved into the same context. This is the most ingenious design in this project. The simplest way to introduce the planning model is to insert the planning rounds directly into the same dialogue, but this will destroy the cache stability of both roles. Putting them in independent sessions can keep their respective prompt prefixes unchanged.

Many similar third-party extensions have also emerged in the Pi ecosystem. Taking pi-deepseek-cache as an example, it also takes DeepSeek cache optimization as its core goal, and its design ideas have many common points with Reasonix.

Reasonix emphasizes "injecting a stable environment summary at startup", and the P0 layer of pi-deepseek-cache does exactly the same thing — freeze the date and current working directory when the Agent starts, and fundamentally eliminate the cache invalidation caused by dynamic content such as "Current date: YYYY-MM-DD" and "Current working directory:" in Pi's default system prompt.

Reasonix emphasizes that "outdated tool outputs should be pruned and cleaned before compression", and the P3 layer of pi-deepseek-cache achieves this through cache-friendly compression — when the dialogue history is too long to be summarized, use deepseek-v4-flash to generate a deterministic summary under the condition of temperature 0, and perform hash caching on the summary results to ensure that the same historical input always reuses the byte-consistent summary result, avoiding breaking the prefix due to fluctuations in the summary text.

Reasonix also mentions "documenting the tool Schema contract and conducting regression reviews when changes are made", and the P2 layer of pi-deepseek-cache provides a similar protection mechanism — diagnose the prefix through SHA-256 hash, track when the prefix changes, so that developers can find the root cause of cache invalidation in time.

The cost reduction effect of this extension is very intuitive: taking deepseek-v4-flash as an example, the input Token cost drops from 0.14 USD per million Tokens to 0.003 USD, a reduction of 98%; deepseek-v4-pro drops from 3.00 USD to 0.025 USD, a reduction of 99%.

Final Notes

Interestingly, DeepSeek has not yet launched its own Harness officially. Just yesterday (August 11), the official WeChat public account "DeepSeek Harness Team" has been registered, which is interpreted by the outside world as an important signal that the Harness product is about to be officially released. In addition, the internal beta of the product has already started, and the official release should be coming soon.

What special features can the official Harness bring? The most core point may still be "native adaptation". Third-party Harness can only do reverse optimization through public APIs, while the official team can work back-to-back with the model training team, allowing the model to make targeted optimizations for the Harness's calling mode, and the Harness can also use non-public information inside the model. This level of deep integration is something no third party can achieve.

This article is from the WeChat public account "InfoQ" (ID: infoqchina), author: Tina, published with authorization from 36Kr.