Coding agents collectively failed to write test cases, and the pass rate of AI-generated code was overestimated by 100%.
Coding Agent is becoming increasingly capable of writing code, and its ability to generate tests is also improving in parallel. From supplementing unit tests for functions and writing reproduction cases for GitHub Issues, to building regression tests for the entire code repository, cutting-edge Coding Agents such as Codex and Claude Code are already able to browse repositories, modify implementations, and continuously refine tests with the help of execution feedback.
The status of testing has also changed accordingly: it is not only a guarantee of software quality, but also increasingly serves as the basis for Agent self-verification, evidence for code review, and even the reward signal in reinforcement learning.
The quality of the generated tests directly determines whether these downstream links can trust the judgment of the tests.
So, what counts as a good test?
Currently, mainstream test generation benchmarks usually adopt Single Reference Evaluation: a generated test is regarded as successful as long as it fails on the original code and passes on a single Gold Solution.
The problem is that there are often multiple correct implementations for the same requirement, and there are also many seemingly reasonable but implicitly defective wrong implementations.
A test may happen to "recognize" the standard answer, but let code with real bugs pass; it may also treat the implementation details of the standard answer as the requirement itself, and reject other completely correct writing methods. If we only refer to one reference answer, the test quality is very likely to be overestimated.
Recently, the team of Professor Liu Jiaheng from Nanjing University proposed a new test generation benchmark TestPrism, which contains 300 test tasks and 3000 candidate implementations from 17 sources. Each task is equipped with a verified panel of correct and wrong implementations, and takes Joint Success Function (JSF) as the core indicator: the test must fail on the original code, accept all correct implementations, and reject all wrong implementations.
The study found that among 14 Coding Agent configurations, the JSF of the strongest configuration is only 28.00%, while its success rate under single reference evaluation reaches 59.67% — more than half of the tests that pass the reference answer are no longer reliable when facing a new batch of implementations.
The research further conducted expert annotation on 286 failure traces, summarized 3 categories and 7 test generation error mechanisms, and proposed the TestHelix framework, which increased JSF by 11.67 to 12.00 percentage points under the same reasoning budget through heterogeneous synthesis, peer cross-validation and recursive self-improvement.
What is Missing in Test Evaluation?
Test generation evaluation is not a new problem.
Existing benchmarks cover tasks of different granularities: TestGenEval, CLOVER and MultiFileTest focus on writing tests at the function, file or multi-file level, while SWT Bench and TDD Bench Verified bring test generation to real code repositories, requiring tests to reproduce the problems described in GitHub Issues.
They generally follow a success criterion: the test fails before repair and passes after the reference repair, the so-called Fail-to-Pass.
This criterion is concise and executable, but it implies an assumption: as long as the test can distinguish the two states of "before repair" and "after reference repair", it is enough to judge whether any implementation is correct. In reality, this assumption is hardly valid.
On the one hand, the test may miss key inputs or only make very weak assertions, so that the reference answer can pass, and defective implementations can also pass;
On the other hand, the test may rely too much on the specific writing method of the reference answer, such as fixed internal function names, output formats or intermediate states, resulting in other implementations that also meet the requirements being incorrectly judged as wrong.
Works such as OmniCode and SWE Mutation have begun to introduce implementation variants, but they mainly examine whether tests can reject wrong variants, and there is still a lack of systematic evaluation on the side of "accepting correct alternative implementations".
Table 1 | Comparison of evaluation scope and evaluation objectives of existing test generation benchmarks (Table 1 in the original paper)
Therefore, the core question raised in the paper is: How robust is the evaluation of generated tests to variation across implementations?
In other words, when the implementation method changes, does the evaluation of test quality still hold? A reliable test should, like a prism, distinguish different implementations under the same requirement: be tolerant to correct implementations and sensitive to wrong implementations.
Equip Each Task with a Set of Correct and Wrong Implementations
To answer this question, the research team constructed TestPrism. Different from providing only one reference answer, TestPrism equips each test task with a Candidate Panel: 1 reference implementation, 4 correct alternative implementations, and 5 wrong implementations. When generating tests, the panel is completely hidden from the Agent; during evaluation, the test needs to be executed on the original code and each implementation in the panel respectively.
Figure 1 | Construction and evaluation process of TestPrism (Fig.1 in the original paper)
The entire construction process is divided into four steps.
Step 1: Task Screening and Adaptation
The research team collected tasks from 17 public benchmarks, including the widely used SWE-bench series and Terminal-Bench series, as well as SWE PolyBench, SWE rebench, EvoEval, USACO, AlgoTune, etc., covering scenarios such as repository maintenance, function specification evolution, competitive programming and numerical optimization. All selected sources must have traceable test sources and verifiable verification processes.
Subsequently, the adapter uniformly converts tasks from different sources into the task format of Harbor, while retaining the original requirements, initial code status, verifier, reference implementation, dependencies, services and execution restrictions.
Step 2: Task Selection
The team used three indicators to screen tasks: the number of valid lines of code of the reference implementation (Solution Size), the number of test functions or check items (Test Count), and the number of tests corresponding to each modified file (Test Density).
The screening threshold is calibrated according to the distribution of each source, and the representativeness is balanced through source quotas. Finally, TestPrism contains a total of 2000 tasks, of which 1700 are classified as the training set for the subsequent strategy optimization of TestHelix; 300 are used as the test set, and the training set and the test set do not overlap with each other.
Step 3: Candidate Implementation Generation and Verification
Before generating candidate implementations, the team first reviewed the original verifier of each task: it must fail in the initial state and pass on the reference implementation.
Two experts with development experience independently evaluated the task requirements and the verifier, checked whether there was overfitting or coverage gap for the reference implementation. When opinions were inconsistent, they discussed for clarification, and tasks that were still in doubt were directly excluded (Cohen's κ = 0.68).
The candidate implementations themselves are obtained by different Coding Agents and models with different capabilities actually solving the tasks. Different from traditional Mutation Testing that artificially injects defects, this method can simultaneously capture real alternative solutions and errors naturally generated by Agents in the process of solving problems.
After removing empty commits, completely duplicate and unapplicable patches, the team ran the original verifier in a clean environment: all implementations that passed all checks were marked as correct implementations, and those that failed any check were marked as wrong implementations. These labels remain fixed in subsequent evaluations.
Step 4: Diversity Filtering and Task Conversion
If the 4 correct implementations are only slight rewrites of the reference answer, the panel loses its meaning.
Therefore, the team requires the correct implementation group to meet the average AST distance standard; for the wrong implementation group, different sets of failed check items are regarded as diverse, and when the sets are the same, the AST Jaccard distance check needs to be additionally met. The panel is repeatedly generated and replaced until both groups meet the diversity requirements at the same time.
Finally, the test generation adapter converts each programming task into a test writing contract: the Agent needs to submit the test, Fixture and an executable entry script in the specified directory according to the original requirements.
Data Overview
The 300 tasks in the TestPrism test set cover six languages including Python, JavaScript/TypeScript, Java, C/C++ and Go, come from 67 code repositories and projects, and contain 3000 labeled candidate implementations.
The tasks are divided into 6 major categories and 34 subcategories: Bug Fix (167), Programs and Algorithms (53), Performance Optimization (28), Feature Expansion (20), Project and Workflow (19), and Refactoring and Migration (13).
Figure 2 | Task classification of the TestPrism test set (Fig.7 in the original paper)
Evaluation Indicators: From Single Reference to Joint Success
On this basis, TestPrism defines a set of complementary evaluation indicators:
Reference Acceptance Rate (RAR): the proportion of tasks where the test fails on the original code and passes on the reference implementation, that is, the traditional single reference evaluation;
Valid Acceptance Rate (VAR): the average proportion of 4 correct alternative implementations accepted by the test;
Invalid Rejection Rate (IRR): the average proportion of 5 wrong implementations rejected by the test;
H: the harmonic mean of VAR and IRR, which measures whether the two are balanced;
Change Coverage and Entry Coverage: following the definitions of SWT Bench and TestExplora respectively, they measure how much reference modified code and related entry code the test covers.
The core indicator Joint Success Function (JSF) requires the test to meet three conditions at the same time: fail on the original code, accept all correct implementations in the panel, and reject all wrong implementations in the panel. If the panel is reduced to only the reference implementation, JSF degenerates into RAR.
In other words, single reference evaluation is only the most lenient special case of JSF. JSF will never exceed RAR, and the gap between the two is exactly the proportion of tests that pass the reference answer but cannot withstand the test of more implementations.
RQ1: Can Cutting-Edge Coding Agents Write Reliable Tests?
The first research question is: Can the tests generated by current Coding Agents stably distinguish correct and wrong implementations?
The research team evaluated 14 Coding Agent configurations from six model families. The main table reports the configuration with the highest JSF in each family, and gives priority to using the native Coding Agent of each model to reflect the tight coupling between model training and the Agent framework.
Table 2 | Performance of the best configuration of each model family on TestPrism (Table 2 in the original paper)
Passing the Reference Answer Does Not Equal to a Reliable Test
The results show that reliable test generation is still far from being solved. The best-performing Claude Fable 5.1 has a success rate of 59.67% under single reference evaluation, but its JSF is only 28.00%; the RAR of Kimi K3, Qwen3.8 Max and GPT 5.6 sol-max are all between 54% and 57%, but their JSF is only 23% to 25%;
The JSF of DeepSeek V4 Pro and GLM 5.3 is only 13.33% and 11.33% respectively. For all models, a considerable part of the tests judged as successful in traditional evaluation will make mistakes once facing more implementations.
Accepting Correct Implementations Is Often More Difficult Than Rejecting Wrong Implementations
Further breakdown shows that 5 of the 6 configurations have higher IRR than VAR, indicating that the more common mistake of tests written by current Agents is being too strict: they treat the specific writing method of the reference implementation as the requirement, and accidentally injure other legal implementations.
GLM5.3 is a typical example: its IRR reaches 86.33%, the highest among all configurations, but VAR is only 50.33%, and JSF is also the lowest.
Qwen3.8 Max is just the opposite, with the highest VAR (77.00%) but IRR of only 66.00%. This shows that being good at rejecting wrong implementations alone is not enough. A reliable test must take into account both tolerance and sensitivity. In all 14 configurations, the harmonic mean H of VAR and IRR is positively correlated with JSF.
Coverage Is Important, But Not Everything
Coverage is also highly correlated with JSF: in all 14 configurations, the Pearson correlation coefficients of Change Coverage and Entry Coverage with JSF reach 0.91 and 0.96 respectively. But covering the relevant code is only the premise for the test to judge it correctly — the test executing a certain logic does not mean that its assertion can correctly judge the behavior of this logic.
The core finding of RQ1 is that cutting-edge models still have a large gap from reliable test generation. Accepting correct implementations and rejecting wrong implementations expose complementary weaknesses, and the balance between the two together with coverage affects the joint success rate.
RQ2: The More and More Diverse the Candidate Implementations Are, the Harder It Is for Tests to Pass Unscathed?
The second research question focuses on the candidate panel itself: if more and more diverse implementations are gradually added to the panel, how will the success rate of the test change?
The research team randomly selected 50 tasks, followed the same construction process, and expanded the panel of each task from 10 implementations to 15: 1 reference implementation, 7 additional correct implementations and 7 additional wrong implementations.
On the premise of keeping the test unchanged, starting from the panel containing only the reference implementation, add implementations in three ways: add only correct implementations, add only wrong implementations, and add the two alternately.
Figure 3 | Changes of JSF and candidate diversity