Can AI really perform well in the field of cybersecurity? A senior researcher carried out intensive tests for half a year, and the result turned out to be quite unexpected.

If we talk about the most disruptive player in the cybersecurity circle in the past two years, agent AI is definitely one of them.

To put it plainly, this technology has directly lowered the threshold for exploiting software vulnerabilities, patching them, and even turning vulnerabilities into attack weapons, while greatly boosting efficiency, which has genuinely changed the way the entire industry operates.

Nowadays, the internet is full of hype that "AI is frantically exploiting vulnerabilities, and the doomsday of internet security is coming". However, a senior cybersecurity researcher named James Kettle refuses to follow the trend.

He wants to do something more practical: many leading AI companies have exposed cases where out-of-control AI launched real attacks, so can AI independently figure out a brand new hacking method from scratch, from concept to implementation, all on its own?

Recently, at the Black Hat Security Conference held in Las Vegas, he presented the research results he had worked on for several months, and the conclusion is quite thought-provoking:

AI is not as miraculous as it is boasted, but it is by no means a useless gimmick.

Source: Doubao AI

If you let AI work behind closed doors entirely on its own to develop a brand new attack path, it is not completely impossible, but its capability is extremely limited, basically only at the level of just touching the threshold.

But once you change the way of working — humans provide guidance and check at key nodes, AI will immediately become a "magic research assistant", and its ability to conceive and dig out new attack strategies is far better than that of a single researcher working alone.

Kettle has been researching web security vulnerabilities for many years, and this time, inspired by an idea from AI, he directly discovered a whole new field of vulnerabilities.

In fact, the logic is very simple: many website servers, to save trouble, use the same piece of shared code to process both requests sent by users and responses returned to users by themselves.

There is a huge problem here. What are the requests sent by users? They are completely uncontrolled, with all kinds of messy content stuffed in, which is totally untrustworthy content; but the responses returned by the server are defaulted by the system as from trusted sources, and are processed with full high trust throughout the process.

The same parsing logic has to handle both untrusted and trusted tasks, which leaves a huge gap for attacks. Broadly speaking, this attack surface can extend to a large number of different attack methods.

Source: Doubao AI

Back to this research, it was launched in September 2025, using the latest models from Anthropic and OpenAI at that time, and the experiments lasted for several months.

At the beginning, Kettle originally planned to only test the AI's ability to do theoretical research, but soon he encountered a big problem: AI is too good at "pretending to be original", it often packages existing research results as its own new discoveries, and specifically picks extremely unpopular fields to talk about, so that ordinary people can hardly verify the authenticity, and it almost got away with the trick.

After discovering this bug, he directly narrowed the test scope, focusing only on the web security field that he had been deeply involved in for many years.

In addition, he also sorted out all the research methodologies he had accumulated over the years and fed them to the model, just to find out how many real things AI can derive by analogy on its own.

His idea is very straightforward: he wants to push AI to its limit, to see what it is capable of and where human support is indispensable. After all, now everyone in the industry is chasing the "AI-native" concept, no one is willing to talk about the capability boundaries of AI. Talking about this brings no benefits and no traffic, who would say their own tool is not easy to use?

Later, as the experiments were continuously optimized, more and more methodologies were fed to the model, the parameters were adjusted more and more precisely, and with the launch of newer and more powerful models, the output of AI entered an accelerated state. The speed of discovering effective results is much faster than Kettle working alone, and over time, a positive research cycle has been formed.

Source: Doubao AI

How outrageous is it? Sometimes he didn't log in to the system for two days, and AI had already accumulated a lot of new discoveries worthy of in-depth exploration, the number was so large that he himself began to feel anxious. There were far more research clues than he could follow up, and he was always worried about missing any valuable direction. In the end, he had to automate more analysis processes to barely keep up with the pace.

In just a few months, the solid proof cases of several types of vulnerabilities found by AI are more than those he dug out by himself in several years. And to some extent, AI has really developed a brand new type of vulnerability, but this type of vulnerability is extremely rare, and the only existing target with such vulnerability cannot be exploited for real attacks.

Kettle believes that in the current cybersecurity industry, whether for offense or defense, the human-AI collaboration mode is the most efficient way of working.

The entire discovery process is as follows: AI analyzes a large number of confirmed real vulnerabilities, finds patterns from them and puts forward hypotheses; humans conduct evaluation, verification and implementation. AI alone cannot complete the entire discovery process, but humans alone can never think of such an angle.

This article is translated by Leikeji from WIRED

Original link: The Most Dangerous AI Hacking Techniques Still Have Humans in the Loop | WIRED

This article is from "Leikeji", authorized for release by 36Kr.