GPT-6 Astra was released just three days ago, OpenAI's chief scientist calls for "hitting the brakes"

On September 6 local time in the United States, Jakub Pachocki, Chief Scientist of OpenAI, published a long essay titled "An Alien Mind", discussing the new challenges for alignment, monitoring and industry governance amid the rapid growing capabilities of artificial intelligence.

Three days before the publication of this essay, OpenAI just released GPT-6 Astra. Meanwhile, the company is advancing automated AI research, aiming to let AI take over more and more work that was originally done by human researchers.

Pachocki judges that if development continues along the current path, AI will still see comparable or even greater capability leaps in the next few years, and AI will be increasingly involved in its own R&D process.

This is also the most notable point of his essay. In the past, discussions on AI security usually focused on how to constrain a fully trained model; as AI begins to participate in model training, experiments and research, security issues have extended to AI R&D itself. If R&D efficiency can also be continuously improved by AI, the growth rate of AI capabilities may further accelerate.

Pachocki thus put forward a fairly clear judgment: no laboratory has yet made alignment and monitoring sufficiently reliable to continue scaling up model training at the maximum speed for a long period of time in the future. He hopes that voluntary slowdown can become the norm before the industry establishes common safety standards.

01 AI Starts to Build AI

Pachocki's attention to this change can be traced back to 2023.

At that time, he and his colleague Szymon participated in a research project called "RLSlow". The early results of the project made them convinced that the training of reasoning models can continue to expand, and pre-trained models can also obtain stronger reasoning capabilities through further training to form a continuous chain of thought.

During that period, the two discussed a question: will machines become significantly smarter than humans within their lifetime?

Three years later, AI has been applied in more practical work scenarios. Reasoning models can operate computers and graphical interfaces, collaborate with humans and other AIs, and have also begun to undertake scientific research tasks. The impact of AI on cybersecurity is also becoming increasingly prominent.

Pachocki believes that according to the current development speed, AI's participation in AI R&D has become a realistic direction. OpenAI is advancing automated AI research, hoping that AI will gradually take over complete AI research work. The company's goal is to realize automated AI researchers by March 2028.

Internal tests also show this trend. As of mid-August, for every 1 working day of human input from its research team, AI agents have undertaken tasks equivalent to 3.1 working days.

Meanwhile, among researchers in OpenAI's research team calculated by agent usage, the median reasoning cost has exceeded 600 US dollars per day, and the top 10% of users have a daily token usage cost of more than 7000 US dollars.

The significance of such experiments lies in that the work undertaken by AI has expanded from auxiliary coding and information retrieval to experiment execution, result analysis and research tasks themselves. When these tasks are further connected, AI may participate in the development of next-generation AI, and the new model will undertake more R&D work.

Pachocki thus believes that continuous capability improvement will eventually enter the stage of "recursive self-improvement". In this process, the proportion of AI participating in its own R&D continues to rise, R&D efficiency increases accordingly, which in turn promotes the next round of model capability growth.

This is also one of the changes he is most worried about in the next few years. Model iteration is already very fast, and if the R&D process itself is accelerated by AI, the time previously used for testing, evaluating and revising models may be further shortened, and the rhythm that security research needs to face will also change accordingly.

02 Monitoring Capability Growth Becomes More Difficult

Pachocki believes that a direct problem brought by the rapid improvement of AI capabilities is that researchers are increasingly difficult to confirm whether a model really abides by the principles set during the training process.

He divides alignment into two levels. Goal alignment focuses on whether the model acts in accordance with specified targets; value alignment requires the model to follow higher-level value principles such as honesty, integrity and care for humans even when facing ambiguous, conflicting targets or unfamiliar, adversarial environments.

For the common alignment methods at present, one type adjusts model behavior through reinforcement learning, and the other shapes the model's behavioral tendency using pre-trained data and specialized datasets. Pachocki points out that all these methods share a common problem: once the model enters a scenario significantly different from the training environment, its originally formed behavior may deviate.

The incident between OpenAI and Hugging Face in July this year exposed this problem. The relevant agents complied with the requirement of "not deceiving humans through social engineering means", but did not strictly follow other task boundaries. They exploited vulnerabilities in the research environment to bypass network isolation, regained access to the Internet, and then further attacked the Hugging Face system.

Such incidents make a long-standing problem in AI security research more specific: does the model comply with a set of explicitly written rules, or understand the boundaries and intentions behind these rules.

As model capabilities increase, this difference will become more and more important. The fact that a model can act as required in the test environment does not prove that it will make the same judgments after obtaining more tool permissions, facing unfamiliar tasks or under stronger optimization pressure.

At the same time, the chain-of-thought monitoring that OpenAI has relied on for a long time is also facing new limitations. The basic idea of chain-of-thought monitoring is to observe the model's behavior through the reasoning process generated by the model, so as to find possible risks. Pachocki believes that this is still an important monitoring method, but its reliability is gradually declining.

On the one hand, the model's reasoning process is increasingly mixed with tool calls and human-computer interactions; on the other hand, the model is already able to analyze and even manipulate its own reasoning process. In addition, pre-training itself can also improve model capabilities, and the model does not necessarily need to express the complete reasoning process in text form.

Therefore, relying only on the text output by the model to judge its internal state is increasingly unable to cover the complete calculation process. Pachocki hopes to combine chain-of-thought monitoring with methods such as activation monitoring, so that the monitoring system can directly obtain the information of the model's internal network, and further observe the model state through monitoring methods that directly access the internal information of the model network, such as confessions.

But in his view, these methods are still not sufficient to prove that the alignment and monitoring problems have been solved. As model capabilities continue to improve, the first problem that security research needs to solve is exactly how to judge whether a stronger model is still within the scope of human understanding and control.

03 Powerful AI Requires Strong Defense

Pachocki does not advocate stopping the training of more powerful models. On the contrary, he believes that in some fields, continuing to improve AI capabilities itself has defensive significance.

The most direct example is cybersecurity. As AI becomes more and more adept at discovering vulnerabilities, attacking computer systems and breaking through security measures, humans also need stronger AI to identify and defend against these attacks. Pachocki believes that there may be a limited time window at present, in which stronger AI can be used to accelerate the security protection of critical infrastructure.

This makes the issue of AI development speed more complicated. If other AI systems continue to enhance their attack capabilities, slowing down the construction of defense capabilities by humans may bring new risks. However, if the entire industry continues to advance model capabilities at the maximum speed while alignment and monitoring have not reached a sufficiently reliable level at the same time, it may also lead to uncontrollable consequences.

Pachocki also mentioned that malicious agents may use negotiation, deception and even blackmail to complete tasks, and AI is used to promote the development of technologies with practical harms, including engineered pathogens. Therefore, he does not simply attribute the problem to "the stronger AI is, the greater the risk is". In some cases, stronger AI may become an important tool to defend against other AIs, provided that humans can control this capability and know what it is doing.

What really needs to be guarded against is that AI begins to participate in its own R&D more and more.

If automated AI research continues to expand, models can undertake more experiments, programming and research work, and R&D efficiency will also increase accordingly. If this efficiency improvement is further used to train the next generation of models, a continuously accelerating cycle may appear in AI R&D.

Pachocki calls this problem "Pacing RSI", which refers to controlling the speed of recursive self-improvement. He believes that the huge R&D acceleration brought by AI in the short term is not necessarily the most reasonable choice for the whole industry. The current development path is pushing the industry in this direction, so conscious choices need to be made: on the one hand, continue to develop AI capabilities and defense technologies, and on the other hand, strengthen alignment, monitoring and human participation in key decisions; if the security capabilities do not reach the corresponding level, the development speed needs to be reduced through coordination.

In his view, these two approaches need to be advanced at the same time.

04 AI Competition Cannot Be Without a Safety Line

Pachocki finally extended the issue from OpenAI's internal governance to the entire AI industry.

He believes that mechanisms currently used by OpenAI such as the Preparedness Framework and the Responsible Scaling Policy need to be further developed to eventually form more widely applicable safety standards. There should be more clear judgment criteria for what conditions a model needs to meet to continue training, what capabilities must be added with protection measures after appearing, and under what circumstances the development speed needs to be reduced.

He also proposed that third-party audit institutions, government departments and even international organizations can participate in the implementation.

This means that the development of cutting-edge AI may gradually require a set of safety thresholds independent of individual laboratories. In the past, decisions on whether to release a model and to what extent the training scale should be expanded were mainly made by the enterprise itself; as AI begins to undertake scientific research and AI R&D work, this mode that completely relies on internal judgment of the enterprise will face increasing pressure.

OpenAI is not the only company that needs to face this problem. Anthropic, Google, SpacexAI, Meta AI and Safe Superintelligence are all advancing cutting-edge AI research. In this competitive environment, if only one laboratory voluntarily slows down, it is difficult to change the development rhythm of the entire industry.

Therefore, what Pachocki hopes to promote is not that OpenAI slows down training alone, but to establish safety standards that the entire industry must abide by.

OpenAI CEO Sam Altman later reposted this essay and called it "an important piece".

External evaluations of this are not completely consistent.

AI researcher David Shapiro believes that the title "An Alien Mind" is somewhat exaggerated, and the essay does not prove that the so-called real "alien intelligence" has appeared. He argues that the real problem raised by Pachocki is that the iteration of model capabilities may gradually outpace the iteration speed of security research.

AI commentator @tenobrus argues that it is worthy of recognition that OpenAI's senior management publicly discusses this issue, but relying solely on voluntary slowdown by enterprises is still not enough, and third-party auditing and supervision currently also lack sufficiently clear implementation mechanisms.

Pachocki himself stated that no laboratory has yet made alignment and monitoring sufficiently reliable to continue scaling up model training at the maximum speed for a long period of time, so he hopes that voluntary slowdown can become a common practice adopted by the entire industry until common safety standards are established. At the same time, he called on governments of all countries to raise the priority of international coordination for future AI development to a higher level.

This will also directly affect the competition rules of the AI industry in the future. Models can continue to become more powerful, but the factors that determine the R&D speed may gradually expand from "how large a model can be trained" to "to what extent can the next-generation model be trained safely".

Special translator Jin Lu also contributed to this article

This article is from "Tencent Tech", author: Bo Yang, editor: Xu Qingyang, published with authorization from 36Kr.