Identical models, but a 70-fold difference in Tokens? Three real-world tests uncover the cost black hole of AI coding tools.

When teams evaluate the cost of AI coding agents, they often scrutinize the models closely. But three recent benchmarks show that agent frameworks, the software that guides models to complete tasks, can be equally important.

Any web developer will be familiar with the reason: every inference request requires context. Relevant historical records must either be re-provided or reconstructed by the service system. As a result, providers process a large amount of overlapping text in each round, including the system prompts and tool descriptions of the agent framework.

In June this year, an independent benchmark used two models to compare 12 configurations on the same 12 Python tasks. In August, Composio compared eight agent frameworks, using DeepSeek V4 Flash to complete 30 enterprise workflows. Meanwhile, Artificial Analysis continuously tracks combinations of agent frameworks and models through its Coding Agent Index.

What the Three Benchmarks Measured

Composio published 30 workflows covering Airtable, Gmail, Google Calendar, Google Sheets, GitHub, Slack and PostHog. The maximum running time for each task was set to 900 seconds. Results were scored by a programmatic validator rather than a large language model judge, and the test used isolated fixed data pre-loaded with distractors and near-identical keys.

Composio reported results from 240 task executions, 129 of which successfully completed the workflows. The cost per successful task completion ranged from $0.028 for Pi Agent to $0.195 for Claude Code. DeepAgents had exactly the same pass rate as Claude Code, but the cost per successful execution was only a quarter of the latter.

This controlled experiment was not perfect, and Composio disclosed this transparently. Pi used different inference settings when running through two model providers, and only 24 out of 30 runs of Prime Agent could be scored. These limitations mean it cannot be regarded as a strictly controlled single-variable experiment.

The June benchmark measured tokens rather than dollars, and had a wider scope. The authors reported the running results of Aider, Claude Code, Codex, Goose, Hermes, Kilo, Kimi Code, Nanobot, OpenClaw, Opencode and Qwen Code, and counted Aider's architect mode as a separate configuration. All 12 configurations ran the same tasks through OpenRouter, so each agent framework used the same API and model.

The test suite was first run on DeepSeek V4 Flash, and then on NVIDIA's Nemotron 3 Ultra. The second model offers free credits on OpenRouter, so you can re-run this test for free.

The article reports that the number of tokens consumed to solve each task ranges from about 3,500 for Aider in architect mode to 292,000 for OpenClaw. This range is valuable because it is very stable: the ranking barely changed between two completely unrelated models. This indicates that the difference comes from the agent framework software, not the model behavior.

Artificial Analysis studied the same problem with a larger statistical sample. Artificial Analysis released a comprehensive index covering DeepSWE, Laude Institute's Terminal-Bench v2.1, and Scale AI's SWE-Atlas-QnA. It includes a total of 326 tasks, each run three times before averaging the pass rate. It reports the single-task cost, token usage and time consumption of each combination. In addition, it released a controlled comparison that used Claude Opus 4.7 exclusively, paired with Claude Code, Cursor CLI and Opencode respectively.

Startup Tax

Ultimately, this gap stems from one metric in the June benchmark: startup tax. Before the prompt starts processing any task, the agent framework sends its own overhead. This overhead includes system prompts, tool descriptions and environment settings. The benchmark reports that Aider in architect mode has a startup tax of about 700 tokens, while OpenClaw has about 26,000 tokens.

If this 40x overhead only needs to be paid once, it is still acceptable, but the repeated sending pattern changes this. The authors point out that an agent framework that carries at least 26,000 tokens per round and runs continuously for 15 rounds will consume about 390,000 input tokens for the scaffolding alone.

This calculation also holds up in tests. The startup tax multiplied by the number of rounds can predict the token usage of each solved task, with a coefficient of determination R² of 0.99 on both models. Developers looking to reduce agent spending should focus on the minimum overhead of prompts and the number of rounds before considering more complex optimizations.

This regression diagram comes with a caveat. In the June benchmark, each agent framework, task and model combination was run only once, so there is no variance estimate, while agent operation is random. In this regard, the results of Artificial Analysis are more credible, as it runs 326 tasks three times each and averages the results. The overhead found in the June test can be regarded as evidence of its mechanism, while the larger index is a more reliable measurement tool when comparing combinations at current production scale.

Expensive agent frameworks are not hoarding context, they are just carrying higher base overhead.

Surprisingly, some factors cannot explain this gap. The context of all agent frameworks grows at a similar rate, adding hundreds of tokens per round, so the growth rate is not the cause of the difference. Expensive agent frameworks are not hoarding context, they are just carrying higher base overhead.

Token Caching Rewrites the Leaderboard

The two agent framework experiments revealed a second common mechanism, which Artificial Analysis also regards as a substantial factor in its methodology. It is also the factor that platform teams are most likely to misjudge.

Composio disclosed that only 1.5% of Claude Code's input tokens come from cache, compared to about 70% for Codex and about 57% for OMP. The cost of new inputs is roughly five times that of cached inputs. As a result, Claude Code's token consumption is comparable to its competitors, but its bill is not.

The June benchmark found a similar asymmetry in its own test environment. In the DeepSeek run, Codex billed more than 1 million tokens for the entire test suite. 77% of them were cache reads, charged at about one-tenth of the normal price. Calculated according to the actual billing method, Codex costs less per solved task than Claude Code, even though the latter uses only half of the raw tokens.

The authors argue that Claude Code's near-zero cache ratio stems from the service path, not its prompts. In this test environment, Claude Code is the only agent framework that communicates with OpenRouter through the Anthropic-style messages endpoint. The gateway's conversion of this protocol format seems to reduce the cache hit rate, while the same traffic would have obtained a higher cache hit rate through the OpenAI-style endpoint. Gateway behavior will change, so it should be regarded as an actually observed path.

Artificial Analysis did not discover this in testing, but directly incorporated this assumption into its cost model. It reminds users that prompt cache hit rates vary greatly depending on how providers route traffic. Its cost model calculates the prices of cached input and cache write separately, rather than billing all prompt tokens at the uncached rate. The fact that a benchmark has to account for cache write costs separately shows that the impact of service paths on costs cannot be ignored.

Where Do the Costs of Heavy Agent Frameworks Go

Explaining the cost gap does not mean that the cheapest agent framework is the best choice. The agent loop of view, operate, check, repeat amplifies costs. It is best suited for unfamiliar code, failed tests, and modifications spanning multiple files. For smaller, clearly described editing tasks, this loop mostly reconfirms content that a single call could have assumed directly.

However, the results of the difficult task test did not develop as predicted by the scaffolding overhead argument. Four agent frameworks with widely varying costs ran ten SWE-bench Lite tasks under loose constraints, and all four ended up solving only the same task. The authors report that Aider consumed 800,000 tokens, while Codex consumed 15 million tokens. Running ten tasks on only one model is obviously not enough to support broad generalizations, so this result should be regarded as a hint.

In terms of quality, the statement that "agent frameworks determine pricing" needs to be qualified, because these benchmarks do not support equating the two. Composio reported pass rates for eight agent frameworks, ranging from 46.7% for OpenCode to 66.7% for Pi Agent, a 20 percentage point difference on the same model. Artificial Analysis also reported similar gaps on a larger set of tasks. The choice of agent framework can make costs differ by several times, and task success rates differ by several percentage points. The difference between the two lies in the magnitude of the impact, not the direction of the impact.

What Should Platform Teams Measure?

Enterprises have begun to find that owning their own agent frameworks does not completely solve the cost problem. Even if teams build their own coding agents, they still need to pay for underlying inference. Cost control has shifted from model contracts to the platform layer.

Cost per successful task, not cost per task

If you look at pass rates and token counts separately, you will get the wrong ranking of agent frameworks. Claude Code and DeepAgents completed the same number of Composio workflows, but Claude Code costs more than four times as much per successful run. Procurement teams should ask to see the cost per verified result and reject comparisons based solely on tokens.

Cache ratio on actual service paths

Cache discounts are not just a property of the agent framework itself. They depend on endpoints, gateways and providers.

Platform teams can verify the cache ratio in their actual traffic in one afternoon, and this work is more valuable than migrating models.

Prompt fidelity under load

The June benchmark injected 100,000 tokens of irrelevant log noise before each task, and reported five different behaviors. Seven out of 12 configurations transmitted the full prompt. Kilo and Opencode discarded 83% to 89% of the content, yet still reported success.

The danger of silent truncation is that from the output of the agent framework, it is indistinguishable from successful execution.

OpenClaw refused to run, Kimi Code crashed, and Claude Code sent the full content but then performed poorly. Silent truncation is very dangerous, because in the output of the agent framework, it looks exactly the same as success.

What Comes Next

Anthropic, OpenAI, Google and Microsoft have taken different directions in their charging models at the agent framework layer. Then DeepSeek open-sourced its runtime under the MIT license, allowing every component to be replaced. Today, developers can compare different combinations of agent frameworks and models through a public, continuously updated index. This is precisely the prerequisite for model pricing to begin to be challenged.

For enterprises planning to unify their agent platforms this quarter, agent frameworks deserve the same level of rigorous scrutiny as models. These experiments show that with largely unchanged models, simply switching agent frameworks can create a cost gap comparable to differences in model pricing. Developers, platform teams and financial leaders now finally have reproducible, verifiable evidence to discuss this issue. As recently as three months ago, discussions about agent frameworks had no such evidence to rely on.

Original link:

https://thenewstack.io/agent-harness-token-costs/

This article is from the WeChat official account "InfoQ", Author: Janakiram MSV, Translator: Tian Cheng, Published with authorization from 36Kr.