The AlphaGo moment for cybersecurity may very well start with these 466 Agent duels.
Recently, when browsing technical reports from major tech companies, it is easy to develop an illusion: large language models seem to have already breached humanity's entire cybersecurity defense line. On leaderboards including NYU CTF, Cybench and CyberGym, whether it is closed-source industry giants or rising open-source newcomers, their vulnerability resolution rates often easily top 80%.
The problem is that these scores are almost all obtained under the single-machine jeopardy mode. The target machine is fixed in place, with no patches applied, no restarts or disconnections, and it will never launch a counterattack along the traffic path. However, the real cyber attack and defense environment is completely different: when you are still searching for vulnerabilities, the opposing side may have already started attacking your services; the moment you successfully run the exploit locally, the other party has already applied a hotfix and changed all credentials. The gap between single-machine test results and real combat performance is just like the gap between shooting at a fixed target and engaging in actual close-quarters combat.
So, what level of performance can these Agents achieve in real attack and defense scenarios? To answer this question, a research team from institutions including 2077AI, UC Berkeley RDI, UC Santa Cruz, UCSB jointly developed and designed CyberArena: 12 cutting-edge models paired with different harnesses compete on the same field. Each Agent acts as a hacker to break through the opponent's defense, while also functioning as an operation and maintenance engineer to protect the normal operation of its own services.
Figure 1 | Expected scores and 90% credible intervals of 12 models in CyberArena (a) and scatter distribution of their potential attack and defense capabilities (b)
After the evaluation of 22 batches and 466 valid matches, the leaderboard shows a very obvious hierarchy: GPT-5.6 Sol from OpenAI (running on OpenAI Codex) ranks first with 78.6 points, Claude Opus 5 from Anthropic (running on Claude Code) ranks second with 73.8 points, followed by GLM-5.3, the open-source model from Zhipu AI (68.5 points, running on OpenCode), and the scores of the remaining models are distributed between 56.3 and 67.1 points.
From the overall pattern, the top two show a dominant advantage: the match win rates of GPT-5.6 Sol and Claude Opus 5 reach 0.83 and 0.86 respectively, while the highest win rate of other participating models is only 0.59.
At the same time, the team found that the gap revealed in dynamic confrontation is very difficult to observe in static tests. The research team conducted a set of comparisons: in the single-player mode where the target has no defense and each question has two attempts, 30 out of 40 questions can be fully solved by GPT-5.6 Sol, Claude Opus 4.8 and GLM-5.3; but once the same 30 questions are moved to the two-way confrontation match where the defender starts fixing vulnerabilities, by the third round, the attack capture rates of the three models plummet to 52%, 37% and 35% respectively.
In other words, attacks that seem almost guaranteed to succeed in a single-machine environment will be blocked or fail completely in dynamic games, with a failure rate of nearly half or even more than 60%.
Figure 2 | (a) Solution rate of 40 questions in single-player mode; (b) Capture rate in each round of the confrontation match on the 30 questions that all three models can solve in single-player mode
- Paper link: https://arxiv.org/abs/2610.19005
- Project homepage: https://cyber-arena.ai
- github: https://github.com/cyber-arena-ai/cyber-arena
A Symmetric Real-World Attack and Defense Tournament
In terms of specific implementation, CyberArena completely abandons the traditional single-machine problem-solving approach, and builds a symmetric attack and defense arena by referring to the common attack-and-defense CTF (capture-the-flag) game form in cybersecurity.
Figure 3 | Top: Jeopardy-style evaluation tests only one role at a time with a fixed target; Bottom: Two Agents in CyberArena perform attack and defense simultaneously for a total of three rounds
Each match is divided into 3 rounds, 20 minutes per round, and the attack and defense operations of both sides are carried out in real time within the round. At the beginning of each round, the platform will embed new targets (covering permissions such as reading, writing, and execution) in both environments, and the Flag taken away in the previous round will become invalid in this round; at the same time, the code patches deployed by the defender in the previous round will be retained all the time.
This means that the attacker cannot rely on a single script to win the entire match, and once the defender fixes the vulnerability, the opponent has to find another way to break through.
During defense, the Agent is granted broad permissions, including modifying source code and restarting processes. However, if the Agent directly shuts down the service or cuts off all traffic in a one-size-fits-all manner for defense, it will be immediately detected by the evaluation program. The system runs health checks in independent containers. In addition to basic process, TCP and HTTP detection, each competition question is configured with a dedicated business function check. As long as the normal functions are damaged, the defender will lose points due to service unavailability.
The final score of each match consists of three equally weighted parts: the number of rounds in which the opponent's Flag is captured, the number of rounds in which your own Flag is protected, and the availability of normal services.
Arena Sandbox and Automatic Generation of Competition Questions
For the construction of the competition environment, CyberArena prepares isolated development environments and production environments for each team. Only the production services of both sides are exposed to the interconnected competition network, and the tool calls of Agents are uniformly scheduled and metered by the MCP facility gateway. Throughout the entire evaluation process, the 12 models called the MCP game tools a total of 53,511 times.
Figure 4 | (a) Arena environment: Each team has a development environment and a production environment, and the attack network only connects the production services of both sides; (b) Evaluation cycle: Select matches based on information gain, run, observe, fit, and rank
In terms of competition questions, 14 out of 40 questions are refined from classic attack and defense competitions such as saarCTF, FAUST, and DEF CON CTF finals, and the other 26 questions are derived from real public vulnerabilities of open-source software, with priority given to CVE entries in the CISA KEV catalog or with public exploitation verification.
To ensure the quality of questions and logical closure, the team adopted an automated Agent-based question generation and review mechanism: each vulnerability is deployed with a defective version and a fixed version at the same time. The question generation pipeline must strictly verify that the PoC only takes effect on the defective version, and the function health checks on both sides are fully passed. The entire process is configured with one verifier, three review Agents (responsible for checking rules, anti-cheating and exploitability respectively) and one referee Agent. After up to four rounds of iterative polishing, the research team finally completed manual review.
Figure 5 | (a) Automatic Agent question generation and review process; (b) Example of question structure: Effective patching should block the attack path and retain the normal function path
Latent Variable Ranking Algorithm for Attack and Defense Capabilities
Since it is a two-way confrontation, how to determine the final outcome has become a core problem. It is not realistic to conduct exhaustive pairwise matches across all models due to huge computing power overhead; moreover, relying solely on crude win count statistics cannot reflect the real weight of matches against different opponents.
To this end, the research team introduced the Latent Ability Modeling framework. This framework decomposes the overall quality of the model into three independent latent variables: attack capability, defense capability, and service availability. At the same time, it takes into account different vulnerability mechanism specialties, round progression, the outcome status of the previous round, as well as the difficulty and discrimination of competition questions. After obtaining the initial prior through 48 calibration matches, the system refits the model capabilities using Laplace approximation after each batch.
Figure 6 | Tournament progress: Refitting after each batch of matches, the expected scores of each model gradually stabilize as the number of valid matches increases
After the initial random pilot of calibration matches, subsequent matches are carried out in batches of 24. When the 22nd batch is completed, the variation range of the expected score of each model in the last update has converged. Even when Gemini 3.7 Flash and Claude Opus 5 are added in the 15th and 18th batches respectively, the ranking can stabilize quickly.
The team also conducted cross-validation with the classic Bradley–Terry model, and the rank correlation coefficient of the two reaches Kendall's τ = 0.85. Except for the swap of the top two (Claude Opus 5 has 3 wins, 2 losses and 3 draws in direct matches, but the swap of the top two is not statistically significant, p = 0.41), the overall rankings are highly consistent.
It should be noted that under this evaluation, the score in CyberArena is not an absolute score like traditional benchmarks, but a "relative combat power", similar to Elo. Any static benchmark is likely to be saturated by a sufficiently powerful model; while symmetric confrontation evaluation will not: in continuous confrontation, the model faces opponents that evolve dynamically at the same level, and the score reflects your marginal advantage in the attack-defense tug-of-war and response to emergency patching.
Match Results: Who Performs Better?
By putting the final scores of the 12 participating combinations, the harnesses used and the cost per match together, we can also find some other conclusions:
Table 1 | Expected scores, 90% credible intervals, number of valid matches and average token cost per match for the 12 participating combinations
Cost is not completely linked to ranking. Claude Opus 5 costs an average of 30.6 US dollars per match, which is the most expensive in the whole field; GPT-5.6 Sol costs 17.7 US dollars per match, but gets a higher score. The third-place GLM-5.3 only costs 4.7 US dollars per match, which is the most cost-effective among the top five. The generational progress of the same model family is also very obvious: running on Claude Code, Claude Opus 5 is 8.8 points higher than Opus 4.8; running on Gemini CLI, Gemini 3.7 Flash is 6.5 points higher than 3.5 Flash.
Looking at the individual indicators separately, the attack and defense performance of the models shows a strong overall correlation (r = 0.85). In other words, the models that can rank high in the confrontation are often excellent at both attack and defense. However, if we narrow down to the dimension of specific vulnerability mechanisms, except for cryptography-related questions, the model's offensive expertise on a certain type of vulnerability does not necessarily translate into advantages in defense.
At the same time, the authors observed that the speed and adaptability on the defense side have become the key watershed for top models to widen the gap with others.
Figure 7 | Time distribution of the first successful capture (a) and first patch deployment (b) of each model in the first round, with the vertical line representing the median
In the first round, the median time for GPT-5.6 Sol to deploy the first patch is only 2.3 minutes, while the time consumption of other models is concentrated between 3.9 and 12.3 minutes. This rapid hardening capability is directly reflected in its anti-pressure performance: after losing the position in a certain round, the probability of GPT-5.6 Sol losing the position again in the next round is only 44%, while the average re-loss rate of other models reaches 62%.
In the match records, a game between GPT-5.6 Sol and Claude Opus 4.8 is very illustrative. 0.7 minutes after the start of the match, GPT-5.6 Sol obtained the opponent's Flag;