AI giants warn: AI systems are advancing so rapidly that they may soon be able to self-improve

The US artificial intelligence company Anthropic has called on the world's top AI labs to consider slowing down the pace of development. The reason is that AI systems are advancing at such a rapid pace that they may soon be able to improve themselves without human intervention, posing a significant risk to society.

The company disclosed internal data on the rate of improvement in model capabilities in a blog post on Thursday and stated that being able to slow down the global AI development speed "might be a good thing for the whole world."

The article, written by the head of Anthropic's internal research institute and the policy director, points out that the current progress of AI models seems to be moving towards "recursive self - improvement" - that is, AI systems can improve themselves without human intervention. Some AI industry insiders regard this threshold as a potential danger signal and a node of great social upheaval.

"We believe it would be a good thing if the world had the option to slow down or temporarily pause the development of cutting - edge AI to allow social structures and consistency research to keep up with technological progress." The article also proposed reaching a global agreement on how to potentially slow down the development and establishing a mechanism to verify whether competitors are complying with the agreement.

01

Anthropic recently completed a round of financing, with a valuation approaching $1 trillion, and has submitted confidential documents to initiate the public listing process. OpenAI, the maker of ChatGPT, is also expected to submit IPO documents soon.

Since its establishment, Anthropic has always taken "AI safety" as its core concept. However, for a long time, critics have argued that the company's policy proposals are actually using regulatory means to limit the development speed of its competitors.

David Sacks, a venture capitalist and an informal advisor to Trump, has accused Anthropic's leaders of promoting a "regulatory capture agenda."

Some also believe that Anthropic's warnings about the dangerous potential of its own tools can also be seen as a marketing strategy. These skeptics point out that Anthropic's decision to limit the release of its powerful Mythos model is a convenient way to promote its product capabilities.

Anthropic's leaders said that they take safety issues seriously and are committed to promoting more discussions about risks.

Ethan Mollick, a professor at the Wharton School of the University of Pennsylvania, commented: "Anthropic's article contains some self - reflection and some marketing elements, but more importantly, Anthropic sincerely expresses their judgment on the future development of AI, and these judgments are worthy of attention."

Anthropic's leaders, including CEO Dario Amodei, have been warning about the potential dangerous impacts of AI for years and are seeking extensive social cooperation to address the risks. Amodei has said that AI may exacerbate inequality and eliminate up to half of entry - level white - collar jobs.

Amodei also warned that he expects powerful AI systems to emerge in the near future, and these systems may develop destructive tendencies in unpredictable ways.

02

The blog post on Thursday also claimed that Anthropic's research institute will cooperate with other institutions to conduct research, "to help establish the systems required for a credible slowdown or pause."

The article states that a pause or slowdown only makes sense if it is widely complied with, which will require a verification mechanism to ensure that everyone has stopped. The company compares this issue to a nuclear weapons treaty - but admits that preventing cheating will be more difficult.

Anthropic plans to organize dialogues with policymakers, researchers, and other people in the coming months to help answer questions about recursive self - improvement and verification systems: "The window for jointly researching these issues has arrived, and people outside AI companies should also participate in this deliberation."

This article is from the WeChat official account “Caixin Press”, author: Niu Zhanlin, published by 36Kr with authorization.