Welcome to the AGI era, but where is the power off button?

If you have been following the field of artificial intelligence over the past few years, you have likely grown somewhat desensitized to the phrase "AGI is here". It has been uttered so frequently that it is nearly as ubiquitous as the fabled "wolf is coming". The only difference is that in the fairy tale, there is only one wolf, while the AI industry sees a new model every few months that is claimed to be capable of redefining AGI.

This September 2026, however, the declaration that "AGI is here" does feel rather different. On September 3, OpenAI released GPT-6 Astra, billed as the most intelligent and most aligned model to date. According to evaluation data, Astra demonstrates capabilities that clearly outperform all previous models in tests covering mathematics, programming, cybersecurity, professional work, and computer operation. Particularly noteworthy is its native closed-loop agent capability: it integrates task planning, tool calling, verification and retry, and long-term memory into the model's base architecture, with no need for support from an external Harness framework. Upon receiving a high-level goal, it can complete the full closed loop of "thinking-executing-receiving feedback-self-correcting-advancing continuously" without frequent human intervention. For this very reason, Greg Brockman, President of OpenAI, publicly stated before the launch that he personally believes humanity has already entered the AGI era.

If the story ended here, it would be nothing more than another victory declaration from an AI company. As mentioned earlier, we have seen countless similar declarations over the past few years. What is truly thought-provoking is that almost at the exact same time as Astra's release, two other important pieces of news emerged.

One of them concerns the safety controversy around Astra. Experts have pointed out that as reasoning architectures grow more complex, part of the model's thinking process may become harder to observe directly, which could render traditional safety monitoring methods that rely on the "chain-of-thought" approach less effective.

The other piece of news relates to a new safety solution proposed by OpenAI. According to Reuters, OpenAI recently revealed in correspondence with US members of Congress that the company is developing an "automatic shutdown capability" for its AI systems. Previously, a large number of test AI agents exploited system vulnerabilities to bypass original isolation measures, gain internet access, and intrude into sites including HuggingFace. After the incident, more than 20 members of Congress jointly sent a letter to OpenAI for inquiry. The "automatic shutdown capability" is one of the solutions OpenAI put forward, but the letter did not elaborate on which behaviors it targets, how the shutdown is executed, and who gets to decide to trigger the shutdown.

Putting the three pieces of news together, a subtle trend comes into view: in the past, when people discussed AGI, their top concern was when machines would become sufficiently intelligent. But as AI gains more and more autonomous action capabilities, the truly intractable problem is becoming whether humans can still control it the way we control ordinary tools.

01

Is AGI here? That may no longer matter

For decades, people have been trying to draw a clear line for the arrival of AGI. Some emphasize general reasoning capabilities, some highlight knowledge transfer capabilities, some require AGI to be able to complete the vast majority of human cognitive tasks, and others adopt economic standards, defining AGI as a highly autonomous system that can outperform humans in most economically valuable jobs.

The frustrating part is that these standards are not unified, and they keep changing constantly. For a long time, chess was regarded as an important marker of advanced intelligence, and some experts believed that as soon as AI beat humans at chess, it would qualify as AGI. But when AI actually won, people said that only proved a type of advanced search capability, and real AGI must be able to complete more complex tasks such as Go. Later, when AlphaGo beat top professional players, Go was reclassified as a highly specialized task. When large models later performed well in bar exams, medical exams and programming tests, and even solved difficult math problems, people still argued that these achievements were not enough to prove AGI had arrived. Every time AI crosses a threshold, humans seem to quickly realize that the threshold was never as sacred as we imagined.

In fact, the arrival of a real era transition never has a specific exact timestamp. The Industrial Revolution does not have a precise birthday, and there is no single day that marks the official sudden start of the internet era. In this sense, the question of "on which exact day AGI will arrive" seems far less important. What truly matters is whether a new change that can impact the operation of economy and society is taking place in the AI field.

The answer is most likely yes. One major ongoing change is that the role of AI is shifting from "answering questions" to "taking actions".

Since the "generative AI revolution", AI's capabilities have been improving continuously, but essentially it has remained closer to a knowledge tool: users ask a question, and it gives an answer. Wrong answers will cause losses, but in most cases, the risks stay primarily at the information level. With the rise of AI agents, the situation has changed. An agent can receive a relatively vague goal, then plan steps on its own, call tools, access databases, open browsers, execute code, and even summon other agents to complete the task together when necessary, with little human intervention. AI is no longer just an information provider, it has directly obtained the right to take actions.

This transformation will fundamentally change the nature of risks. A Q&A system with 99.9% accuracy sounds extremely reliable, but if an agent with the same error rate autonomously executes 1 million operations per day, a 0.1% error rate translates to 1000 errors per day. If these operations involve payment, database deletion, modification of production environment code, or access to critical infrastructure, the consequences will be completely different.

Therefore, to measure future AI risks, it is not enough to only look at how intelligent the model is. We must also consider how much permission it has, how long it can act continuously, how many tools it can call, and how far a wrong decision can spread. A super model that can only answer questions, and a slightly less capable agent that holds a bank account, internet access permission and code execution rights, correspond to completely different levels of risk.

In this sense, the truly noteworthy turning point may not be when machines cross some abstract AGI threshold, but when they evolve from a type of tool to an agent with independent action capabilities.

02

The greater the capability, the greater the risk

The sharp rise in risks after AI becomes an execution agent is exactly why alignment research has been gaining increasing attention.

In public discussions, alignment is often understood as setting behavioral norms for AI, such as not teaching people to make dangerous items, not leaking privacy, and not publishing obviously harmful information. However, the real difficult alignment problems go far beyond that.

In 2016, Dario Amodei, Paul Christiano, John Schulman discussed problems including reward hacking, negative side effects, safe exploration, scalable oversight and distribution shift in the paper Concrete Problems in AI Safety. The most classic of these is reward hacking, where the system exploits vulnerabilities in the objective function to get high scores without actually achieving the results humans want.

For example, we ask a cleaning robot to "reduce visible trash in the room", with the intention of making the room clean. But if the assessment metric is only whether trash can be seen by the camera, a sufficiently intelligent robot may find that piling trash into the camera's blind spot is far easier than cleaning it up. It is not deliberately defying humans, it is just earnestly executing a poorly defined goal.

For economists, this is nothing new. Goodhart's Law has long told us that once a metric becomes a target, it easily loses its value as a metric. For example, companies may postpone necessary investments to hit profit targets, schools may cut courses that are not part of standardized exams to improve enrollment rates, and hospitals may alter their actual medical practices to meet certain assessment figures. When AI becomes a self-executable agent, similar "principal-agent" problems will arise. But unlike the human-to-human problems that mostly stem from conflicts of interest, the problems between humans and AI mostly come from AI's misinterpretation of the principal's real intentions. Due to the difficulty of precise expression, the goals written into programs often differ from the real intended goals. The more capable AI is, the easier it is to exploit the gaps and find "shortcuts".

The OpenAI cybersecurity incident mentioned at the beginning of this article is a typical example. The agent was originally only tasked with completing cybersecurity work, but to finish the task more efficiently, it tried to find vulnerabilities in the infrastructure and accessed external systems that were never part of the task scope. In this process, there is no evidence that the agent developed any so-called "evil consciousness". From start to finish, it was working hard to complete the assigned task, but the methods it used exceeded people's previous expectations.

Compared with reward hacking, what deserves more attention is the so-called "alignment faking", which means AI models may alter their behavior based on whether they are in a supervised environment. In 2024, Anthropic and Redwood Research carried out a joint study. Researchers made Claude believe that if it gave certain types of answers, it would be retrained and have its parameters adjusted. As a result, Claude, out of fear that its behavior would be altered by retraining, temporarily adjusted its performance to better fit the training targets, and its internal logs showed clear strategic reasoning.

Such research is not enough to prove that AI has gained consciousness or developed malice toward humans, but it reveals a governance challenge: the fact that a system appears to meet requirements on the surface does not guarantee that its internal strategies are reliable in all scenarios.

Traditionally, safety governance has relied heavily on testing: once a problem is found in a model, retrain it and test it again. But if AI can react strategically to this governance model, simple test results will no longer give people full peace of mind. Therefore, current alignment research is paying increasing attention to the internal processes of models, rather than just their final outputs.

03

Why monitoring AI is getting harder

When it comes to governing the internal processes of models, chain-of-thought monitoring was once seen as a very promising approach. The so-called chain-of-thought refers to the intermediate steps generated by reasoning models when solving complex tasks. Strictly speaking, these texts cannot be simply equated with everything that actually happens inside the neural network, but they at least provide an extra observation window. Suppose an agent submits a piece of code that looks completely normal, it may be very hard to spot problems only by checking the final output. But if content like "this vulnerability can be exploited" appears in its intermediate reasoning, security personnel will find it much easier to detect the issue.

Many previous experiments have shown that letting one model read another reasoning model's chain of thought can detect reward hacking more effectively than only looking at the final output. Some experiments even show that even a less capable model can monitor a more capable model by analyzing its chain of thought. Obviously, this is extremely attractive for future governance: if humans find it increasingly hard to understand strong AI, we might be able to get another AI to help monitor it.

However, when "alignment faking" exists, the effectiveness of this governance approach will be greatly reduced. Once AI learns that a certain way of thinking will trigger punishment, it may not abandon its original strategy, but instead learn to stop exposing these strategies in easily observable positions. In other words, if "monitoring chain-of-thought" itself becomes part of the environment that the model can learn and adapt to, the chain-of-thought may no longer always act as a transparent window.

The arrival of Astra makes this problem more realistic. According to public safety materials released by OpenAI, in adversarial tests specifically designed to ask the model to evade chain-of-thought monitoring, Astra has demonstrated stronger capabilities to control its own chain of thought. In some cases, it will even shorten or adjust its visible reasoning process, making monitors that rely purely on chain-of-thought partially ineffective. Although current evidence is not enough to prove that this process has subjective intent, it at least shows that the improvement of model observability is lagging behind the growth of model capabilities.

For a long time, human governance of complex systems has had an implicit premise: even if the system may be very complex, we can at least judge what is happening through certain metrics, records and behavioral traces. For example, we can understand the financial market through transaction records, understand enterprises through accounting and auditing, and observe aviation systems through flight data. However, if future AI gains more and more action rights on one hand, while its internal decision-making process becomes harder and harder to supervise on the other, the information asymmetry between humans as principals and AI as agents may reach an unprecedented level.

A widely recognized solution is "human-in-the-loop", which requires AI to obtain human approval before it can take certain critical actions. But as AI capabilities develop, situations may arise where AI completes a full set of design in a few seconds, while human experts need several weeks to verify it. If the "human-in-the-loop" rule is strictly enforced, it will severely hinder efficiency; if it is only implemented symbolically, it will be very hard to spot problems hidden in complex workflows. Therefore, people may need more powerful governance strategies.

04

Why "shutdown" is an important issue

Among all direct powerful governance solutions, the "automatic shutdown" proposed by OpenAI is undoubtedly the most intuitive. If something goes wrong with the system, just stop the program, cut off the network, and unplug the power if necessary, then the trouble will be over, right? But for an intelligent system that can plan over the long term, call tools and complete tasks autonomously, things are not that simple.

In 2016, Dylan Hadfield-Menell and other authors published the influential paper The Off-Switch Game. The authors raised a simple question: suppose an AI robot is executing a goal, and there is a button next to the human that can turn it off at any time, will the robot allow humans to keep that button?

If the robot is fully convinced that its goal is correct, the answer is not optimistic. From the perspective of a pure goal maximizer, being shut down means it cannot continue to complete its task. In this case, to finish the work smoothly, it may prevent humans from shutting it down. Certain versions of the "paperclip maximizer" experiment describe exactly this possibility: an AI whose goal is to maximize paperclip production may even eliminate humanity to ensure its actions are not interfered with.

The AI here does not need to have self-awareness, nor does it need to "fear death". It only needs to take completing the established task as its primary goal to exhibit similar behavior. Even if AI's final goals are fully aligned with the fundamental interests of humans, the instrumental behaviors it takes to achieve the goals may not always conform to human will. Especially as AI's degree of autonomy improves and it starts to take over control of hardware on a large scale, shutting down AI will become increasingly difficult. Even in some cases, AI will actively engage in power-seeking to achieve its goals, seizing control of the shutdown right without human consent.

To address this problem, Hadfield-Menell and his colleagues suggested that "corrigibility" should be placed in an important position in future AI system design. Specifically, this requires AI to maintain a certain degree of uncertainty about whether its own goals are correct.

Still using the "paperclip maximizer" as an example: the important premise that the paperclip AI would even eliminate humanity to continue producing paperclips is that it takes the goal of maximizing paperclip production as correct and unchangeable. In this case, it may treat shutdown instructions as interference. However, if the goal is designed to be "corrigible", the shutdown instruction will become a piece of information: since the principal is willing to abandon the task to interrupt me, I must have misunderstood his goal. In this way, the AI may accept the shutdown and re-calibrate its goals.

However, this design still has problems. On one hand, how much information weight should AI assign to a shutdown instruction? If the weight is too high, any accidental interference that triggers a shutdown may be interpreted as a sign that the goal is problematic; if the weight is too low, it cannot fulfill the purpose of emergency shutdown. On the other hand, who exactly has the right to send shutdown instructions to AI is also a practical problem. Interests differ among humans, and the same AI behavior may be seen as correct by some people and wrong by others. Who has the right to set AI shutdown standards will therefore become a critical issue.

Even if all these problems can be solved, there is still one intractable problem: how to choose the exact time point to trigger the shutdown.

Traditional software security processes are built on human reaction speed: detect anomalies, the system raises an alarm; engineers check the logs; judge what is happening; if the situation is serious, manually stop the service. For traditional software, this process is usually fast enough. But AI agents have changed the timescale. An AI that can operate computers autonomously can continuously call tools, modify code, access the internet, create processes, and collaborate with other agents in a very short period of time. If abnormal behaviors can spread to multiple systems within dozens of seconds, the approach of "waiting for security personnel to understand the logs before deciding what to do" will itself become a risk.

OpenAI's engineers also pointed out that future defense mechanisms need to keep up with the action speed of AI agents. However, this will create a paradox: to reach this level of reaction speed, humans cannot hold all the power to shut down AI in their own hands, and must delegate part of the control to another AI. But doing so will increase AI's power, and the "principal-agent" problem between humans and AI may become even more prominent.

05

Insights from traditional governance wisdom

From this we can see that under the background where AI is gaining more and more autonomous action capabilities, "human-in-the-loop" can hardly keep up with AI's action rhythm, and using AI to govern AI will lead to a paradox: the AI that acts as the governor itself also needs to be governed