Anthropic has put up 320.8 billion. Is this paying for the AI doomsday theory?

The day after Jack Clark, co-founder of Anthropic, announced that "the Singularity has begun", Anthropic is set to set aside 5% of the company's shares to address potential social issues that AI may bring about.

Calculated based on the $9.65 trillion valuation in May, this stake is worth approximately $480 billion (equivalent to around 320.8 billion RMB). One of the uses of the funds is to cope with the employment shock caused by AI, and the company is even considering directly distributing money to affected groups.

Over the past few years, from scenarios of superintelligence going out of control and human extinction to millions of white-collar workers losing their jobs, Anthropic has been one of the most active advocates of AI doomsday theories.

However, the company that "fears most" that AI will destroy the world is itself desperately developing more powerful AI, and even preparing to sprint for a trillion-dollar level IPO.

AI doomsday theory used to be a signature for Anthropic to prove it is at the cutting edge of the industry. Now as its IPO approaches, this narrative is starting to demand repayment from the company.

Anthropic is backfired by its own doomsday prophecies

On October 14, New York City Comptroller Mark Levine wrote a letter to Dario Amodei, demanding that Anthropic suspend its IPO.

He oversees the New York City public pension funds with total assets exceeding $3.25 trillion, of which approximately $7 million is indirectly allocated to Anthropic through private equity investments.

Levin specifically cited several recent warnings from Anthropic in his letter:

On September 9, Jacob Coxon, a researcher at Anthropic, stated when he left his job that some people inside the company genuinely believe that AI could kill all humans before 2030. Evan Hubinger, Head of Alignment Science, even believes that the probability of AI causing human extinction in the next ten years exceeds 10%.

Three days later, Amodei published a long article, calling for a slowdown in the growth of frontier model capabilities, warning that AI's self-improvement may slip out of human control.

At the end of September, Anthropic even included the doomsday prophecy in its IPO draft filing — in the 261-page main body of this document, the part introducing the company's business only takes up 48 pages, while about 80 pages are used to disclose risks. It explicitly warns that advanced AI may pose catastrophic and even existential threats to humanity.

Levin stated that since Anthropic is calling for a slowdown in AI development, it should show the same prudence when pushing forward with its IPO.

This is not the first time similar criticism has emerged.

In August this year, tech investor Gavin Baker publicly criticized Amodei, arguing that his repeated exaggeration of the dangers of AI is fueling the resistance to AI in American society, and even intensifying public opposition to data center construction.

Caption: Part of Gavin Baker's public criticism of Amodei on X

Baker advised him that as the CEO of a company that is about to become one of the most important tech companies in the world, he should say more positive things for his industry. Amodei disagreed. He acknowledged that the public's negative attitude towards AI is a serious problem, but believes that this is mainly a trust crisis and cannot be blamed on AI companies' warnings about risks.

Anthropic has spent years telling the world how dangerous AI is, and now people actually believe it, and are demanding that it pay the price for that.

Calling for deceleration verbally while pressing the accelerator to the floor

Amodei believes that AI could go out of control and has repeatedly called for a slowdown in technological development, but Anthropic's business expansion has not stopped because of that.

On September 12, Amodei just called for a slowdown in the growth of frontier AI capabilities. Ten days later, Anthropic launched Opus 5.5.

Shortly after that, Sonnet 5.5 was launched on September 28, and Haiku 5.5 was released on October 7. In just half a month, three models of the Claude 5.5 series were updated, and the company started a price war.

Model R&D has not stopped, and the speed of cash burn has not slowed down at all.

Previously, according to the IPO draft filing disclosed by foreign media, Anthropic's computing power and infrastructure expenditures in the next ten years are expected to reach at least $5180 billion. About 80% of this amount belongs to non-cancelable commitments, or payments that must be made regardless of actual usage volume.

Just recently, Wall Street has started raising around $600 billion in chip financing for Anthropic again, to purchase approximately 1.5 gigawatts of Google TPUs. The transaction, codenamed "Project Polaris", is expected to become the largest chip financing deal in history.

In this deal, investment institutions pay for the chips and then provide them to Anthropic through a five-year lease agreement. Among them, Blackstone's funds plan to subscribe for $90 billion in high-risk debt with a coupon rate of 10%. In order to get enough chips, Anthropic is willing to bear a fairly high financing cost.

Such a huge investment naturally requires even larger revenue to support it.

On October 14, Anthropic held its pre-IPO investor day, announcing that its revenue in the third quarter reached $168 billion, and achieved adjusted profitability for the second consecutive quarter.

However, its growth rate has started to slow down. Anthropic's revenue in the second quarter skyrocketed by about 145% quarter-on-quarter, while the growth rate in the third quarter dropped to 45%. Although the performance is not bad, it has already disappointed some investors. According to Reuters, some investors attending the investor day even started to discuss lowering Anthropic's expected IPO valuation to $1.5 trillion.

Interestingly, although Amodei still emphasized AI safety this time, he did not take the initiative to talk about the risk of human extinction.

After all, in the face of investors who are about to pay, Anthropic needs to show its growth potential more. Even if Amodei really thinks AI is developing too fast, Anthropic can hardly stop now.

Anthropic is paying for its own prophecies

In a sense, the AI doomsday theory itself can also be a rather extreme form of tech promotion. When Anthropic repeatedly warns that AI may make millions of white-collar workers unemployed, or even destroy humanity, it is also showing the world: the technology it masters is already powerful enough to change the fate of humanity.

The more dangerous the technology is described, the more responsibilities Anthropic needs to take on.

On October 15, Anthropic announced that it would grant close-to-internal-employee-level access to six staff members of METR, a non-profit AI evaluation organization. These external evaluators can use the company's internal tools and systems to inspect models, safety measures and potential risks, and can publish evaluation results independently. Anthropic cannot directly block publication, but can request the deletion of specific sensitive information.

This is also a fulfillment of the promise Amodei made in his long article a month ago.

For a company that has always emphasized that AI may go out of control, letting outsiders come in to check whether it is properly managing AI is obviously more convincing than simply issuing warnings.

And in terms of social responsibility, Anthropic has taken even bigger actions.

In June this year, Anthropic announced an investment of $200 million to establish the Economic Future Research Fund. In July, the company further clarified its research directions, including career transition, unemployment income security, and enabling workers to share the wealth created by AI.

By October 18, Anthropic went a step further and announced that it would set aside 5% of the company's shares to establish a public welfare fund.

According to the officially announced plan, every time the company's valuation increases by 50% afterwards, the board of directors may, at its discretion, add 2% of the shares to the fund. Combined with previous donation commitments from founders and employees, Anthropic stated that approximately 20% of the company's shares may be used for public interests in the future.

However, this stake worth about $480 billion is still a promise pending shareholder approval, and the specific usage remains to be further disclosed.

As soon as the news came out, Elon Musk went to Anthropic's comment section and made a sharp review: "I suggest distributing the shares directly to the public." 

Shortly after that, he posted: "Next year, I will distribute SpaceX stock to every child in the United States." 

Some netizens commented directly under Musk's post that Anthropic's move is probably just for public relations. 

From publicly calling for deceleration, to voluntarily accepting external supervision, to setting aside huge shares to support public welfare, Anthropic is trying to prove that it has the ability to bear the risks and social consequences that AI may bring.

There may be sincere sense of responsibility in this process, and it may also include necessary image building before IPO. The two do not conflict.

However, when Anthropic was telling AI doomsday stories back then, it probably never thought that one day it would really need to spend hundreds of billions of dollars for these stories — and this sum of money may not necessarily buy public trust. 

This article is from the WeChat public account "Alpha AI", author: Yuan Xinyue, published with authorization from 36Kr.