38,000 Hours, Burning Sky-High Tokens: ByteDance Discovers the Scaling Law of Agents

On July 2, ByteDance Seed released EdgeBench, an agent evaluation project. It may look like just another benchmark, but it raises a question that no other ranking has ever asked.

Give a model a question, award points for correct answers, and no points for wrong ones. This is how the vast majority of benchmarks work today, increasingly resembling the college entrance examination system.

But in the real world, people don't use AI that way.

You wouldn't give Claude Code a single problem and wait for it to turn in a test paper. You would hand it a project, a codebase, and a batch of data, then let it work for hours—exploring, making mistakes, reading feedback, correcting, and retrying. What you care more about is whether it can become more capable after being immersed in a real task environment for a period of time than when it first started.

Yet current benchmarks can barely measure these capabilities. They assess static capabilities—what the model already knows at the moment it is frozen. Abilities like continuous improvement from feedback, accumulating experience over long cycles, and finding direction in unfamiliar environments all lie in blind spots.

EdgeBench cuts right into these blind spots by bringing them into evaluation, to answer one question: When you drop an agent into a completely new environment, how much stronger can it get after 12 hours?

134 tasks span six domains: Science/ML, Software Engineering, Combinatorial Optimization, Professional Knowledge Work, Formal Mathematics, and Interactive Games

To this end, the Seed team built an experimental platform called EdgeBench. It is an environmental learning observation chamber. For 134 tasks, each is designed to ensure agents run for at least 12 hours.

Its design revolves around four core dimensions:

5 leading-edge models (Claude Opus 4.8, GPT-5.5, GPT-5.4, GLM-5.1, DeepSeek-V4-Pro) ran on it for a total of approximately 38,000 hours.

Funding is burning. But the final results are immensely valuable:

They discovered the scaling law for agents.

1

Four Findings: Formula, Path, Essence, Speed

1) Agent learning follows a fixed mathematical formula

Averaged across 134 tasks: the learning curves of 5 models are fitted with high precision by the log-sigmoid function

This is the "soul discovery" of the entire paper.

Previously, we mostly assumed that environmental learning is chaotic—different tasks, models, and strategies would naturally follow different rules. But their data gave an unexpected answer:

The average learning curve across 134 tasks is precisely fitted by the same function with an R² = 0.998 accuracy.

What does R² = 0.998 mean? In human-computer interaction and complex system research, seeing R² = 0.3 is already enough to publish a paper. 0.998 is essentially not just a "fitting problem"—it is a discovery. If I know an agent's improvement speed in the first two hours, on average across the task set, I can accurately predict its performance after 12 hours.

The curves show that the tested models generally learn in this pattern: slow at first, explosive once they get the hang of it, then slowing down again as they approach the ceiling. This perfectly matches the experience of anyone who has done deep learning or deep work.

Moreover, this law holds across scales: whether the experimental window is 12 hours, 28 hours, or 72 hours, the fitting accuracy remains above R² ≥ 0.993. It also holds across domains: each of the six task families is fitted separately, with R² values between 0.972 and 0.998.

2) There is no "standard" growth path

This finding has even more practical significance.

When you pull out the individual learning curves of the 134 tasks, you will notice something counterintuitive: although the average curve is a beautiful log-sigmoid, the individual curves differ dramatically.

- Some tasks: The agent improves steadily from the start, with a clean curve

- Some tasks: No progress at all for the first few hours, then a sudden score jump at a certain moment

- Some tasks: Scores rise, then fall, then rise again

- Some tasks: Fast initial rise, followed by a long plateau

Different learning strategies and trial-and-error paradigms produce completely distinct growth paths under the same scoring framework.

Agents are not just "fast learners" vs "slow learners". They have fundamental differences in how they learn—evaluation methods that only look at final scores and ignore the process completely erase this dimension. The paper explains that a task is a capability graph, and learning is unlocking and expanding outward along the frontier, following a logistic curve over logarithmic time. Individual tasks have few nodes, resulting in jagged curves; enough tasks averaged together reveal the S-shape. The scaling law in the paper's title refers to exactly this curve.

3) "Re-understanding the problem" creates real progress

Experience vs no experience comparison: Continuous operation with accumulated experience yields a significant 6.9-point advantage over 6 independent restarts

The same model (Opus 4.8), with the same 12-hour budget:

-Scheme A: Let it run continuously for 12 hours, preserving all intermediate outputs, error logs, and verified hypotheses

-Scheme B: Split into 6 independent 2-hour runs, clearing all states each time and only retaining the best result

After 12 hours, Scheme A scored 6.9 points higher than Scheme B (on a 100-point scale). And the two curves diverged right from the start.

This shows that progress does not come from random luck trying more, but from accumulated experience.

Notably, in the gravitational wave reconstruction case study, GPT-5.5 ran for 12 hours and submitted 224 times—but only 27 submissions actually drove the improvement of the best result.

Each breakthrough did not come from "running one more experiment", but from a qualitative change in the agent's understanding of the problem itself. It gradually broke down vague goals into searchable subproblems, and redefined "what a better direction is" from feedback.

4) Learning speed itself is being "learned"

This finding is probably the most closely related to the industry.

The experiment selected a set of tasks where all models started from similar baselines (first attempts scored around 6.87 points), then measured how much each model could improve after 2 hours of interaction.

Results: From GPT-5-Codex in September 2025 to GPT-5.5 in April 2026, learning efficiency increased by about 8 times in 221 days, roughly doubling every 3 months.

Later models have higher effective submission rates, but not necessarily more total submissions. In other words, they are not working harder—they are making each attempt more effective. This aligns with engineering intuition: senior engineers do not necessarily write more lines of code than junior engineers, but they waste far less effort.

AI capabilities are increasingly resembling "knowledge", but the very ability to "learn new things" is evolving at an astonishing speed—and this speed is more decisive than the growth of static knowledge.

2

The Evaluation Environment Itself Is as Valuable as EdgeBench

EdgeBench may look like a model leaderboard, but it does not measure bare model capabilities—it measures agent system capabilities.

Different models run on different execution frameworks. Claude Opus 4.8 uses Claude Code with a 1M context window, while GPT-5.5 uses Codex with a compact 256K window. The final scores include both the model's inherent capabilities and the impact of context management, tool calling, feedback processing, and the execution framework.

This is actually closer to real-world deployment. An agent in reality is never an isolated model—it is a combination of model, tools, workflows, and feedback systems. What EdgeBench truly measures is whether this combined system can continuously make progress in long-horizon tasks.

But this also means the leaderboard cannot be crudely interpreted as a foundation model ranking. It is more like comparing the long-term working capabilities of different agent systems.

The gap becomes clearer when you put EdgeBench side by side with mainstream benchmarks:

Traditional benchmarks are "static snapshots", while EdgeBench is a "dynamic trajectory". They are not competing on the same dimension.

But this new dimension does not come without cost—it requires massive resource consumption.

What impresses me as much as EdgeBench is what it took to build this experimental environment.

Let's start with an easily overlooked calculation. 134 tasks, each consuming an average of 57.2 hours of human expert time, with the longest single task taking 320 hours. Even by the lowest estimate, task construction alone required over 7,500 hours of human labor. Then there is the runtime cost: 5 models, each running 3 times per task, plus extended runs over 72 hours, totaling about 38,000 hours of agent interaction—translating to astronomical numbers of API calls and computing resources.

This cost threshold itself means that long-horizon agent evaluation is not a direction just any team can enter.

Looking at the engineering details, the paper's appendix documents various vulnerabilities exploited by agents during development: one agent deduced the hidden test data answers through over 400 submissions in a fluid mechanics task; another discovered that anti-cheat checks exempt the baseline/ directory, and sneaked code through that "green channel" to submit work. These cases reveal a deep contradiction: to measure "learning ability", you must give agents sufficient feedback—but the more feedback you give, the more likely agents will exploit it as an oracle.

EdgeBench's solution is physical isolation: separate the working container from the judge container, and destroy the judge after scoring to prevent agent "cheating":

EdgeBench's dual-loop feedback mechanism: the left side is the working container where agents can freely explore, the right side is the hidden judge that returns official scores only after submission

• Working container: Agents can experiment freely here, with full access to compilers, debuggers, logs, and documentation—but no "hidden answers"
• Judge container: After the agent submits its work, the judge scores it using hidden test data and private criteria, then immediately destroys the container

This is a thoughtful design, essentially simulating "se