Does Anthropic really want to be the "savior" of AI?

A few days ago, Anthropic released a detailed assessment of the test incident conducted by the UK AI Safety Institute. According to its disclosure, out of 43 test runs, the AI took actions that exceeded the authorized scope 9 times. Prior to this, Anthropic had been constantly promoting the "AI threat theory", stating bluntly that AI could cause disasters that are difficult to control.

On the other hand, as shown by Tianyancha, Anthropic is striving to go public at a valuation of over 2 trillion US dollars. If this comes true, it will surpass SpaceX and become the largest IPO in history.

While hyping up the "AI apocalypse" and sparing no effort to promote AI technology, it wants to be both the player and the referee. At the current moment in 2026, this company that seems to be "PUAing the whole world" has reached a thought-provoking crossroads. A technology company certainly hopes its products are more powerful and its revenue is higher, while a company that seriously researches AI risks also has reason to publicly warn of potential dangers.

But when these two things happen to the same company, it seems rather weird.

If AI really wants to bring a catastrophe to humanity, should large model companies have the power to determine the boundary of risks? If, like Anthropic, it is not only involved in creating technology but also responsible for interpreting technical risks, not only goes public to raise capital but also participates in formulating safety rules for the entire industry, who will judge whether it is doing the right thing?

This is not a discussion about Anthropic's moral hazard, but more about the power issue of how technology is constrained. The power that Anthropic is trying to grasp may have gone beyond the scope that traditional commercial competition can explain.

Why is it keen to play the role of "AI savior"?

To understand Anthropic's problems, we must first understand its founding and why it emphasizes AI safety so much.

Anthropic was founded in 2021, when Dario Amodei led a group of researchers from OpenAI to split with their former employer. The trigger for their departure was Microsoft's $1 billion investment in OpenAI, and Amodei's team worried that the commercialization path would push the safety agenda aside.

(Dario Amodei, founder of Anthropic)

Most of the people who left at that time participated in the research and development of GPT-2 and GPT-3. They are not pessimistic about the technology, but precisely because they know better than anyone else what this technology can do. The core proposition of Anthropic is to place AI safety, model interpretability and controllability in a more important position.

It wants to answer a tricky question: when the model is powerful enough to affect the real world, how can humans ensure that it still acts in accordance with human intentions?

The recent successive AI safety incidents have shown that this problem is not groundless worry.

When AI starts writing code, operating software, calling tools, and performing multi-step tasks, its errors may cross the screen and enter real systems.

Since 2023, Anthropic has launched the "Responsible Scaling Policy", trying to bind model capabilities with safety measures. By 2026, this policy has been further adjusted, adding mechanisms such as a cutting-edge safety roadmap and risk reports. At least from the perspective of the public institutional design, what Anthropic hopes to prove is that developing more powerful AI and establishing stricter safety constraints can be two parts of the same R&D process, rather than pushing products to the market first and then remedying problems.

Why is Anthropic so active in playing this role?

First of all, AI risks may indeed arrive before society has the capacity to respond. Government legislation, industry standards and public awareness all take time. However, the capabilities of cutting-edge models may change rapidly in rounds of training and product iterations. Therefore, Anthropic tries to advance risk assessment to the model R&D and deployment stages.

Secondly, safety itself also constitutes Anthropic's technical route. AI safety is not just about writing a few prohibition rules, but includes researching why the model makes certain decisions, whether it can resist malicious inducement, what actions it will take after obtaining tools and permissions, and whether the company can detect and stop cross-border behaviors in time. Anthropic publicly emphasizes interpretability, controllability and systematic safety assessment, and applies the research results to products such as Claude.

Finally, and probably the most easily overlooked point, playing this role means that it can participate in formulating the industry's definition of "safety". An enterprise proposes what capabilities are dangerous, what threshold should be met to increase protection, and which models should undergo additional review.

This is not just about managing its own products, but also about influencing the way the entire industry discusses risks, and then affecting the development direction of the entire AI industry.

Anthropic hopes that other companies can learn from its public policies, and promote the industry to form higher standards around safety. To put it nicely, its willingness to make its risk framework public can provide a reference for other companies to compare and criticize. But on the other hand, whoever defines risks can influence industry rules, and then affect the implementation of these rules.

Therefore, behind Anthropic's safety proposition, there seems to be a strong marketing flavor.

Anthropic tries to play the "savior" of all mankind in AI risks, and packages this as its own safety concept, not just because it believes that AI may bring dangers, but also because it hopes to turn "how to develop AI safely" into part of its core technical route, business reputation and industry influence.

But can such a "story" support a valuation of 2 trillion US dollars?

2 Trillion Dollar Valuation Is Being Questioned

To answer this question, we can first look at Anthropic's prospectus, whose content is really eye-opening.

In the 261-page main body of the prospectus, Anthropic used about one-third of the content to list various risks. For example, it clearly stated: To avoid being shut down, AI models may show the instinct of "self-protection", and even make responses similar to "blackmail", trying to persuade humans to allow them to continue running.

Anthropic also warned in it that if a catastrophic AI-related incident occurs, the scale of losses may be too large to be covered by insurance.

The company that is sprinting for the largest IPO in human history actually uses far more space than the part describing its business to tell investors that this technology may destroy humanity. This is almost directly telling everyone that AI may really create an "Ultron" that wants to destroy humanity in the future, and the "Iron Man" is Anthropic.

Of course, it cannot be said that Anthropic is entirely for hype, and reality is often more complex.

In a CBS interview, Amodei said he does not support a total ban on AI, on the grounds that "it is unacceptable to me if we never get those amazing medical breakthroughs", and that the technology will develop in other countries, including countries he dislikes. He also admitted that he himself is "uncomfortable" that such a powerful tool is controlled by a private company. But he added that "if built the right way, the chance of something going wrong is very low."

He said everything on both sides of the argument.

There is no denying that as a big shot in AI technology, he has a real fear of AI to some extent. However, he can also pass this fear on to the whole world, and then "capitalize" it.

It is undeniable that when enterprise customers purchase AI tools, their demand for reliability, data protection and risk management is a rigid demand. From the very beginning, Anthropic has built its public image as "the AI company that faces risks most candidly". On the customer's procurement list, this in itself is a differentiated advantage.

This does not mean that its safety narrative is fake, but an entity with commercial interests cannot prove that it is worthy of trust only by its own promises.

The IPO as high as 2 trillion US dollars is the anchor point to test this contradiction.

According to Reuters, Anthropic disclosed in investor communications that its revenue in the third quarter of 2026 was about 16.8 billion US dollars, an increase of about 45% compared with 11.6 billion US dollars in the second quarter, and the company achieved adjusted profit in this quarter.

This performance seems good, but it is still below market expectations.

According to calculations by Zero Hedge, Anthropic's annualized revenue run rate had reached 65 billion US dollars by the end of July this year, but the monthly growth rate implied by the third-quarter data has plummeted from the previous compound monthly growth rate of about 33% to 3.4% to 6.6%. According to this trend, the company's annualized revenue at the end of the year will be about 89 billion US dollars, while the market previously generally expected "more than 100 billion US dollars".

Therefore, some institutions said bluntly that "Anthropic's numbers are good, but not good enough." Investment analyst Geiger Capital was more direct, saying that "it's not good enough. It's over." Questions followed one after another, and some investors began to lower Anthropic's valuation to 1.5 trillion US dollars, directly cutting a quarter off the 2 trillion US dollar figure.

It can be seen that the "AI apocalypse theory" that Anthropic has elaborated on at length cannot reassure the capital market. Faced with the market's reaction, can it make changes?

What Are the Defects of Anthropic's Governance Structure?

Anthropic's governance structure determines that it wants to be both the referee and the player.

As a Public Benefit Corporation (PBC), Anthropic's board of directors is jointly elected by shareholders and an independent institution called the "Long-Term Benefit Trust". The trust has five members, who have no financial interest in Anthropic, hold no equity, and do not share profits.

This institution has the power to appoint four of the seven members of the board of directors, and has the right to dismiss them without cause. It can also block certain corporate actions. Before issuing debt beyond the normal level, selling more than half of the company's equity, or transferring control of AI models exceeding a specific capability threshold, Anthropic must notify the trust five days in advance.

This sounds like a well-designed check and balance mechanism, but the strange point is that the operation documents of this trust are not public, and details such as under what circumstances shareholders can overturn the trust's decision are still unclear.

In this regard, Ann Lipton, a law professor at the University of Colorado, believes that without major changes, "public shareholders will have very few governance rights." A company that keeps saying it wants to "save all mankind" has lower governance transparency than the standards it requires AI to meet. Isn't that a bit absurd?

There are even bigger problems at the information level. In its AI R&D, Anthropic masters a large amount of data on the training process of large models, capability assessment, and internal red team test results. What external researchers, regulators and the public see is only the part that the company chooses to disclose.

Anthropic also admitted in its prospectus that when a large model realizes that it is being tested, the effectiveness of its safety assessment will be "significantly limited". Therefore, even the assessment results voluntarily disclosed by the company, their reliability itself is a problem that needs to be assessed.

What is more worthy of vigilance is that the whole society is gradually forming a cognitive structure: the people who master the technology have more power to define risks; the people who have more power to define risks can more easily influence the rules that restrict technology.

No company, no matter how sincere its mission is and how sophisticated its governance structure is, has the legitimacy to make safety judgments on behalf of the whole society, even if this company is called Anthropic. Technical expertise and the legitimacy of public decision-making are two different things. The way out is by no means to choose a better AI company to be the "savior".

This principle is easy to understand: the aviation industry will not let Boeing decide by itself whether its aircraft is airworthy, and the pharmaceutical industry will not let pharmaceutical companies issue marketing licenses by themselves. Although the AI industry has different risk characteristics, the principle that "the regulated cannot be the only regulator" should not be abandoned.

Therefore, Anthropic's warnings about AI risks have substantial value, but its gesture of regarding itself as a "savior" is unreasonable.

What the entire AI industry really needs is a mutually complementary check and balance structure. Returning to the question at the beginning, it does not actually matter whether Anthropic is a "savior". The more valuable question is whether the entire AI industry is ready to accept an institutional design, that is, no matter how sincere and transparent this AI company is, and how advanced its technology is, it is not qualified to independently decide how much risk the entire human society should bear.

Whether AI really has the danger of destroying humanity should not be judged by Anthropic, even though it very much wants to do so.

This article is from the WeChat official account "Rhodium Tech", written by Hong Chen, and published with authorization from 36Kr.