Our practical test covering 28 different Agent combinations shows that Claude can only deliver 14% of the required output for a 5000-word requirement document.
Nowadays, Agents are already proficient in invoking tools, modifying files, and running tests, but when faced with a large software system that evolves over a long period, can they deliver complete functionality starting from a new requirement?
When product managers put forward new requirements, what developers receive is often not a list of files or function signatures. They need to first understand constraints, track module dependencies, and then determine where to modify, how to modify, and how to prove the modifications are correct. These analyses affect subsequent delivery.
Institutions including Fudan University, City University of Hong Kong, and The Chinese University of Hong Kong proposed LoLBench, which places Agents in software systems with an average of about 2.4 million lines of source code, evaluates the complete process from functional proposal to module delivery, and tests the perceptual ability to translate requirements into the system and the implementation ability to complete cross-module modifications.
Paper link: https://arxiv.org/abs/2609.37143
Dataset: https://huggingface.co/datasets/lolbench26/LoLBench
The experiment exposed key breaking points: Agents may modify the local part correctly, but miss the modifications that must be synchronized in other modules. Even with supplementary implementation guidelines, most tasks remain unsolved, and there is still a long way to go from understanding requirements to completing delivery.
Redefining Long-Range Code Tasks
What kind of long-range code tasks have practical engineering value?
"Long-range" cannot only be measured by time consumption, number of rounds, or Token consumption. For example, letting an Agent output a number every minute and count from 1 to 1000 takes a long time but only requires a simple loop. The local Bug fix tasks of the SWE-bench category also cannot cover the challenges from requirement analysis to module delivery.
The long-range code tasks that LoLBench focuses on are module development for complex systems. After receiving new requirements, developers will analyze repeatedly: which behaviors must be implemented? Can the existing architecture support them? Which dependencies will be affected? How to verify? LoLBench also incorporates this part of the analysis hidden in developers' minds into the evaluation, allowing Agents to independently complete the derivation from user intent to implementation solutions.
Such tasks have both perceptual complexity and implementation complexity: the former refers to the difficulty of reading proposals, identifying implicit constraints and cross-module dependencies, and deriving modification scopes and implementation specifications; the latter refers to the difficulty of converting specifications into correct cross-file modifications, and avoiding damage to existing functions through testing and fixes.
Figure 1 Comparison of perceptual complexity and implementation complexity.
How LoLBench Evaluates Both Types of Complexity Simultaneously
Deriving Implementation Specifications from Functional Proposals
LoLBench selects 100 tasks from community Enhancement Proposals and their corresponding merged PRs, covering 29 systems, 8 languages and 5 fields.
Agents only receive the proposal and the code repository before implementation, and the proposal has an average of about 5000 English words. The evaluation is carried out in a disconnected Docker environment to prevent Agents from retrieving upstream implementations.
For example, PEP 617 [1] proposes to replace CPython's LL(1) parser with PEG. The implementation needs to connect grammar definition, parser generation, abstract syntax tree and error reporting. Agents must identify the entire implementation link.
In terms of perceptual complexity, the average mention rate of the target class name of LoLBench reference implementation in the proposal is only 7.3%, less than half of SWE-bench Verified (15.9%), and less than one fifth of FeatureBench (38.7%). Agents must infer the connection between requirements and code on their own.
Translating Specifications into Engineering Delivery
In terms of implementation complexity, the LoLBench repository has an average of about 2.4 million lines of source code, about 9.4 times that of SWE-bench Verified and 4.9 times that of FeatureBench; the average implementation modification involves 5499 lines, while the latter two are 38 lines and 2050 lines respectively. Each task modifies an average of 64.9 files, 40.1 classes and 195.9 functions; the number of files is 25 times and 4.2 times that of the two respectively. Cross-module collaboration and verification are more difficult.
Table 1 Comparison between LoLBench and existing code task evaluation datasets.
LoLBench verifies delivery correctness through two types of tests: the new functional test (F2P) requires failure before implementation and pass after implementation, F2P tests behaviors through public interfaces and allows different internal implementations. The regression test (P2P) requires passing both before and after implementation.
Based on the original tests of the project, the research team constructed incorrect implementations such as missing functions and tampered requirements as variants, supplemented test cases to identify missed errors, and improved the test coverage of reference patches.
After enhancement, the detection rate of 1688 variants increased from about 61% to 96%, and the average line coverage of F2P per task increased from about 60% to 79%.
Performance and Bottlenecks of 28 Agent Combinations
Even with Different Models or Scaffolds, the Maximum Solution Rate of Agents is Only 14%
The research team evaluated 28 available combinations of 6 cutting-edge models and 5 scaffolds, with a 6-hour time limit for a single task. Opus 5 paired with Claude Code or mini-SWE-agent both ranked highest with a 14% solution rate, and the median of all combinations was only 2%.
Table 2 Main experimental results in the paper
Using Opus 5, mini-SWE-agent and Claude Code have the same solution rate, the former takes 3.2 times as much time as the latter and generates 5 times as many Tokens, but more consumption does not bring a higher solution rate.
A Large Number of Interactions and Tokens Are Spent on Code Location
In terms of Token consumption, on average per Agent, the location phase accounts for 52.4% of non-cached input Tokens and 45.4% of generated Tokens. In terms of the number of Agent interaction rounds, the number of interaction rounds in the location phase is also significantly higher than that in other phases.
Figure 2 Average number of interaction rounds in each phase.
Compared with the existing files and functions modified by the reference implementation, the file modification precision of all 28 groups of Agents ranges from 38.0% to 80.4%, and the recall rate ranges from 30.3% to 65.0%; the function modification precision ranges from 54.2% to 65.6%, and the recall rate ranges from 20.8% to 53.4%. The code location accuracy is low and the coverage is still incomplete.
Even with Supplementary Modification Guidelines, Agents Still Struggle to Deliver Completely
The research team retained the original functional proposal, and additionally provided the modified file tree and API implementation specifications derived from the reference implementation to help Agents determine the modification scope. The solution rate of Opus 5+Claude Code increased from 14% to 34%, GPT-5.6 Sol+Codex increased from 3% to 25%, and DeepSeek V4 Flash+OpenCode increased from 1% to 17%.
Table 3 Experimental results after supplementing the modified file tree and API implementation specifications
Supplementary modification guidelines can alleviate the perceptual burden, but the best combination still has 66% of tasks unsolved. Agents still need to coordinate modules, implement interfaces and pass verification, which indicates that the implementation complexity of large systems is also very high.
From Code Generation to Large-Scale Software System Evolution
These results provide verifiable directions for Agent R&D: whether repository retrieval can cover key dependencies, whether the implementation specifications before coding are complete, and whether verification can discover cross-module omissions in time.
The value of software engineering lies in whether user requirements can be reliably fulfilled. Agents need to understand why changes occur, how they affect the system, and implement the changes completely. LoLBench hopes to promote the evaluation of Agent capabilities from generating code to supporting the continuous evolution of software, making delivery results the measure of Agent capabilities.
References: PEP 617: https://peps.python.org/pep-0617/
This article is from the WeChat official account "AICTimes", Author: AICTimes; Editor: LRST, 36Kr is published with authorization.