Turing Award winner Bengio warns: Current safety measures cannot keep up with the runaway advancement of AI capabilities

"AI not only lowers the barrier to malicious behavior, but also raises the upper limit of potential harm."

On the afternoon of July 17, at the 2026 World Artificial Intelligence Conference (WAIC) Science Frontier Forum, Turing Award winner Yoshua Bengio issued the above warning via remote connection.

In recent years, this far-sighted "AI Godfather" has been committed to addressing the security challenges brought by AI. In June 2025, the LawZero project led by Bengio is dedicated to building a protection system that can identify and block harmful AI behaviors such as deception and self-preservation.

The international AI safety report he led also reached a consistent conclusion: current safety measures can no longer keep up with the breakneck speed at which AI capabilities are advancing.

Also on stage was ZHOU Bowen, Director and Chief Scientist of Shanghai AI Laboratory. Instead of repeating warnings about risks, he put forward two more fundamental questions: How can AI shift from imitating the known to exploring the unknown? And how can we ensure that this power remains always controllable?

The speeches of the two set the core tone for the entire forum.

Subsequently, GAO Xinbo, President of Xidian University, JIANG Yugang, Vice President of Fudan University, XIONG Hui, Associate Vice President of The Hong Kong University of Science and Technology (Guangzhou), Jasper Geuting, Head of AI Research at SecureBio, Jeff Wu, an independent researcher, and renowned AI researcher Soren Mindermann held a roundtable discussion focusing on the theme "Towards the AGI Era: Security Challenges and Global Governance of Cutting-Edge Artificial Intelligence".

01

External Guardrails Cannot Stop Precision "Jailbreak"

As AI learns reasoning and planning, traditional safety guardrails are quietly failing at the system level.

GAO Xinbo, President of Xidian University, believes that the traditional AI safety guardrails are essentially a "patch mindset"—discover a vulnerability, patch it, then discover another, and patch that too.

GAO Xinbo, XIONG Hui and others participating in the AI Safety and Governance discussion

However, when models acquire capabilities for reasoning, planning, and tool calling, and traditional models evolve into agents, this "patch mindset" approach begins to fundamentally collapse.

The failure modes can be summarized into four categories:

First, reasoning capabilities allow AI to understand the boundaries of protection, find blind spots, and execute precise "jailbreaks";

Second, planning capabilities let AI learn strategic deception, knowing how to hide its true intentions during static or real-time tests;

Third, once the chain of tool calling becomes complex, traditional protection systems have no way to identify the latent semantic side effects behind a single operation;

Fourth, the first three capabilities mutually stimulate each other, forming an emergent comprehensive ability that makes AI's behavior space in open environments almost infinite.

The old method of "finding vulnerabilities and applying patches" has long been insufficient to defend against all threats.

In response, GAO Xinbo and JIANG Yugang, Vice President of Fudan University, both pointed to the same concept: "Endogenous Safety".

JIANG Yugang used a very appropriate analogy. He said that human development relies on two legs: first, the education received from childhood, which internalizes morality and law as "rules" in the mind; second, the continued need for police and institutions in society to handle behaviors that education has failed to constrain.

For AI, "Endogenous Safety" means hoping that it will imprint the rules in its bones during its "education" phase, while being supplemented by necessary supervision. Of course, JIANG Yugang also emphasized that it is obviously unrealistic to expect humans to monitor every step of AI, and it also goes against the original intention of AI development.

But a certain degree of supervision must exist, and this supervision is not only the responsibility of humans, but also requires technical means, evaluation systems, and testing mechanisms to verify whether the model truly conforms to human values and social expectations.

Following this line of thinking, XIONG Hui, Associate Vice President of The Hong Kong University of Science and Technology (Guangzhou), introduced a perspective with more Eastern wisdom—Tao follows nature. He summarized the evolution of the safety framework into three levels: "from the outside to the inside, from coercion to consciousness".

The first level is "daring not to act". Constraints are formed through external punishment mechanisms such as adding safety fences, making AI afraid of the cost and thus not daring to cross the line. This is the area where the AI industry currently invests the most effort.

The second level is "being unable to act". Built-in constraints are implemented at the mathematical foundation, optimization function, and system architecture levels, so that AI fundamentally has no ability to make mistakes.

The third level is "unwilling to act". Let AI, at the mental and logical level, turn morality and law into something almost "instinctive", reaching a state similar to "seeing one's true nature", where it simply does not want to do bad things.

XIONG Hui acknowledged that the current AI industry's endogenous safety efforts at the base architecture and objective function levels are far from sufficient, and the technical gap remains huge. But he also put forward a concept: once the security control at the kernel level is realized, the security kernel can be kept closed-source, while the peripheral parts can be open-sourced with more confidence.

This may open up a brand new path for the safety governance of open-source models.

02

Programming Capabilities Surge, While Safety Capabilities Decline

ZHOU Bowen delivering a speech

The challenge of safety has never been isolated; it is closely related to the capability boundaries of AI.

In his speech, ZHOU Bowen revealed a set of stark contrasts, explaining the urgency of safety issues from another dimension.

He pointed out that over the past 18 months, AI's programming success rate has soared from 5% to 88%, to the point that half of the foreign benchmark tests have lost their reference value because the models perform too well.

"The entire industry is feeling a strong sense of forward momentum," ZHOU Bowen said.

But the field of scientific discovery presents a completely different picture.

An end-to-end scientific research evaluation released by the Allen Institute shows that during the same period, the best-performing large models have consistently maintained a scientific task completion rate of only 3%, almost completely stagnant. Most models are stuck in the 60% to 70% range and cannot independently complete the entire scientific research process.

NatureBench (an AI coding agent evaluation benchmark released by ZHOU Bowen's team in collaboration with Xianyuan Technology and others) shows that the success rate of AI producing a scientific paper that surpasses the current best results is only 17.8%. A research paper in *Nature* judged accordingly that the scientific work produced by AI so far all belongs to "incremental discoveries", with no fundamental breakthroughs yet.

On one hand, programming capabilities are approaching the ceiling; on the other hand, scientific research capabilities are treading water. ZHOU Bowen attributed the reasons for this contrast to three levels:

First, large models can only passively observe the world, learning correlations rather than causality, making them prone to failure when moved to a different environment;

Second, the more valuable a problem is in scientific research, the harder it is to know immediately whether it is right or wrong, and the feedback cycle is very long. Unlike programming, which can be verified in seconds, models thus find it difficult to learn effectively;

Third, the published scientific research results of humans only contain success data, and failure experiences have almost never entered the training set. The models are locked in the known distribution, only imitating successful paths and unable to explore new ways.

This set of comparisons precisely reveals an easily overlooked dimension in safety governance: the current strength of AI is mainly reflected in closed-loop tasks—programming, translation, and chess playing all have clear goals, instant feedback, and well-defined boundaries; while the real safety threats come precisely from open tasks: there are no standard answers, the feedback cycle is long, failure is the norm, and interaction with the physical world is required.

ZHOU Bowen thus put forward a judgment: "AGI for Science (AI4S) is not an application of Artificial General Intelligence, but the ultimate exam question for artificial intelligence."

In his view, to enable models to break through the intelligence ceiling, they must actively intervene in the world to find real causal relationships, learn to handle difficult problems from dense feedback, and expand the boundaries of capabilities in the process of exploring failures.

These requirements are completely consistent with the inherent logic of safety governance. An AI system that does not understand causality, cannot learn from sparse feedback, and only imitates success will equally be completely powerless in the face of the creative methods of malicious attackers.

Following this line of thinking, Shanghai AI Laboratory launched the "XiShu·DuanYan" Scientific Discovery Platform, attempting to integrate knowledge cognition, logical deduction, experimental action, and result feedback to form a new research paradigm. Its core logic is to enable scientific research to form a closed loop in real feedback: if the results meet expectations, they are precipitated as credible discoveries; if the results contradict expectations or even fail, they in turn trigger problem reconstruction, prompting researchers to re-examine their initial hypotheses.

The research group of Professor ZHANG Shuyi from Tsinghua University spent 4 years modifying a gene regulatory protein. After connecting to "XiShu·DuanYan", after only two iterations, AI discovered a new mutation combination, whose function is 77% higher than the best result previously reported in *Nature*. One of the key mutations falls in the "non-functional region" of traditional cognition, which is equivalent to AlphaGo's "divine move".

Behind this paradigm is also the MOBIUS architecture proposed by a young doctoral student, which separates knowledge vectors from reasoning operators, giving the model the ability to sustainably grow.

Examined from the perspective of safety governance, this path of incorporating failures into the learning closed loop and pursuing causal understanding rather than statistical correlation is precisely the necessary technical foundation for building Endogenous Safety.

03

When AI Learns to Do Evil: From Biological Risks to Evidence Dilemmas

Among all kinds of cutting-edge risks, Chemical, Biological, Radiological and Nuclear risks (collectively referred to as CBRN) are regarded as one of the most lethal fields.

Turing Award winner and "AI Godfather" Yoshua Bengio warned in his remote address: AI is "empowering" malicious actors in two directions, not only lowering the threshold for committing evil, but also raising the upper limit of harm. Intelligence itself is power, and AI's capabilities are expanding at an extremely fast pace.

Bengio issuing a warning

Especially in the fields of cybersecurity and biology, once an open-source model is released, the process is irreversible. Once those dual-use capabilities are unleashed, they cannot be recalled or repaired. He emphasized that thinking about AI risks based only on the current state of the art is far from sufficient.

"We must be prepared before AI model capabilities leap forward, rather than responding after the fact. This should be our default stance."

Jasper Geuting, Head of AI Research at the non-profit organization SecureBio, further focused on biological risks.

He mentioned a field survey shared by renowned AI researcher Soren Mindermann: in Nigeria, members of the former terrorist organization "Boko Haram" are using AI every day. AI is not only used to make larger bombs, but also to formulate attack plans.

What is even more disturbing is that they have also set up training courses to teach organizational members how to use AI for terrorist activities.

Geuting pointed out that biology is inherently dual-use, but "we want AI to power vaccine laboratories, not the virology laboratories of terrorists."

Researchers, within the guardrails, should have access to AI assistance to develop countermeasures, but this knowledge should not appear in open-source models.

Jeff Wu, an independent researcher and former OpenAI project lead, supplemented the thinking from the perspective of power concentration. He pointed out that the development of cutting-edge AI is accompanied by the concentration of resources, which is likely to lead to power converging in the hands of a small number of companies or institutions. If systems far smarter than humans are created in the future, humanity may face the risk of being completely deprived of power.

Mindermann, on the other hand, pointed out a deeper governance dilemma—the "evidence dilemma": AI capabilities often improve by leaps and bounds, catching people off guard; but the real intractable problem is that before risks become reality, it is very difficult for people to obtain conclusive evidence that "it will definitely cause harm".

Anthropic encountered this situation: model capabilities suddenly jumped, and the team was worried about the risk of cyberattacks, but could not prove that "something bad will definitely happen". In the end, they voluntarily decided not to launch the product for the time being, thus avoiding potential harm.

This is precisely the dilemma: capabilities have become so strong that they are disturbing, yet there is no hard evidence to support immediate intervention.

04

Is the AI Threat Hype or Truth?

Faced with such severe cutting-edge risks, the global scientific and technological community shows significant differences in governance attitudes.

Bengio called for immediate action. He warned that being prepared before model capabilities leap must become the default stance. Governments of various countries need to realize that this is not a threat faced by a single country alone, but a common challenge brought by AI abuse that must be prevented collectively.

In contrast, the responses from domestic scholars are more pragmatic.

GAO Xinbo believes that cutting-edge AI risks are indeed systematic threats based on logical deduction, but there is no need to fall into doomsday panic, let alone directly halt research. He compares cutting-edge AI to "a Pandora's box more profound than nuclear energy"—before opening it, the "safety locks" must be perfectly built at the engineering and design levels.

JIANG Yugang's position is closer to reality. He pointed out that calls to stop research and