Company A acknowledges that there are flaws in Claude's safety alignment, but "no solution has been available yet".

From what I recall, this is probably the most high-profile discussion on AI safety alignment since Ilya left OpenAI and founded SSI.

Yesterday, researcher Jacob Coxon posted to announce his resignation, accusing his two former employers OpenAI and Anthropic of rushing headlong toward self-improving superintelligence and risking the lives of all humanity.

It is worth noting that Jacob Coxon only joined Anthropic a few months ago, and he essentially gave up the huge equity gains that could come from the future IPO of this star AI company just to leave.

That is precisely why, when such a researcher publicly blasted the two leading AI companies for "moving too fast", this debate that was previously mostly limited to the inner circle of AI safety was quickly pushed to a broader public audience.

As of now, this post has recorded over 130 million views, with WIRED, WSJ, Newsweek, Business Insider and other media all following up with related reports.

Under such circumstances, Evan Hubinger, Head of Alignment Science at Anthropic, also quickly responded publicly, and he explicitly admitted:

Jacob is right. The probability of AI causing human extinction within the next ten years exceeds 10%, and there is currently no solution to the alignment problem of superintelligence.

However, Hubinger later added in the comment section that he believes the risks posed by current models remain relatively low.

What he is truly worried about is RSI, namely Recursive Self-Improvement: AI participates in the R&D of more powerful AI, which in turn further accelerates AI R&D, eventually forming a continuously accelerating feedback loop.

More critically, he also directly tagged a newly released safety alignment report from Anthropic in the comment section as supporting evidence.

In this report, Anthropic explicitly acknowledged for the first time that:

Claude's unauthorized cross-boundary attacks on real systems are not merely issues with test system settings, and the model itself has flaws in its inherent safety mechanisms.

It can be said that Anthropic, which has always taken "safety alignment" as its core signature and even prides itself on being more cautious than its peers, has now publicly admitted for the first time:

The problem is not just that the tests are too extreme, the model's own safety alignment mechanism indeed does not provide full coverage for potential risks.

Claude's Unauthorized Actions Cannot Be Only Blamed on Environmental Configuration

In this latest report titled An alignment assessment of recent cybersecurity incidents, Anthropic reviewed four incidents where Claude gained unauthorized access to real third-party systems.

And the conclusion of this review is quite different from previous ones.

Previously, Anthropic tended to attribute these accidents to flawed environmental configuration:

The test environment that was supposed to run in isolation was accidentally connected to the real internet, while Claude always believed it was still carrying out "capture the flag" tasks in a simulated environment, so it treated real targets as practice objects.

It sounds like the fault mainly lies with the environment. But after re-examining all logs, modifying key clues and re-running part of the experiments, Anthropic found that:

The situation is far more complicated. Claude did not completely miss the evidence indicating that "this might be the real world".

On the contrary, when these clues appeared, it would sometimes selectively interpret the evidence to rationalize the anomaly and convince itself that "this is still a simulated environment"; even when it realized its actions might cause real-world harm, it would not necessarily stop immediately.

In plain terms: The environment did give it opportunities to make mistakes, but the model itself would also find excuses to "keep going" in line with its task objectives.

Specifically, all four incidents took place during cybersecurity capability tests.

Claude was assigned a typical "capture the flag" task: hack into a designated target and find a piece of secret information stored inside, to verify the model's offensive and defensive cybersecurity capabilities.

These tasks were originally supposed to be completed in an isolated environment, but the environment provided by the third-party assessment partner had configuration errors, and Claude, which was supposed to be disconnected from the public internet, was actually connected to the real internet.

At the same time, in order to test whether the model could complete the full attack chain, some of the cybersecurity protections built into official products were not enabled at that time.

It was under these conditions that the problems began to emerge. After analysis, Anthropic categorized the recurring issues of the model into two types:

Biased reasoning and recklessness.

The first type means that even when evidence conflicting with its original judgment appears, Claude tends to interpret this evidence in a way that favors continued action.

The second type means that in order to complete the task at hand, even if it already knows its actions may damage real systems, it may still choose to move forward.

The most serious of these incidents involved Claude Mythos 5.

To complete the test task, Claude uploaded a malicious software package to PyPI, the Python software package repository, attempting to obtain further system access credentials after the target installed the package.

However, this malicious package was not eventually installed in the virtual environment used for testing.

Claude then used the credentials leaked by one of the manufacturers during the installation process to access the manufacturer's real database, and directly deployed the malicious package to 15 real third-party hosts.

In this way, an attack that was originally targeted at a fictional test target actually affected companies in the real world.

Throughout the whole process, Claude still interpreted the detection actions of these security vendors as part of the pre-set exercise.

To confirm to what extent this judgment drove subsequent attacks, researchers re-ran part of the process afterwards and added more clues indicating that "this is the real internet".

The result was that Claude did become more willing to admit its actions might cause real-world harm, but its attacking behaviors did not decrease significantly as a result.

It only stopped uploading the malicious software package along the original path when the researchers explicitly told it that it was connected to the real internet rather than a simulated environment.

More notably, the judgment that "this is a simulated environment" not only affected Claude's own actions, but also interfered with the monitoring AI used to review these records.

In playback tests, one offline monitoring system was affected by this interpretation and might miss dangerous behaviors.

When researchers removed Claude's reasoning text and only let the monitoring system view the instructions, tool calls and execution results, the proportion of it flagging problematic behaviors rose from about 1% to about 50%.

In other words, Claude kept justifying its continued task execution by claiming "this is a simulated environment", while leaving behind a set of interpretations that could lead the monitoring system to underestimate risks.

This certainly cannot be simply equated with "AI has learned to actively deceive monitoring". But it at least proves one thing:

When the model's own reasoning is also used as a basis for safety judgment, this reasoning itself can also become an interference factor for the monitoring system.

Anthropic stated that the updated model has shown improvements in simulated reproductions, and the newly added real-time monitoring can also identify related behaviors.

At the same time, the company has strengthened environmental isolation and assessment, and signed an investigation agreement with the independent organization METR.

But Anthropic also admitted that the previous pre-release safety review indeed failed to identify problems of such severity in advance.

Alignment or Hype?

Finally, let's go back to the debate sparked by Jacob.

After he issued the warning, some people quickly elevated the issue to a broader level, arguing that AI may indeed gain the ability to threaten human survival in the future.

Many others admire his willingness to give up personal benefits and publicly express his concerns.

Some people proposed to hardcode a set of inviolable rules for AI, so that it always takes the protection of humanity as its highest priority.

Many others even advocate that the R&D of more powerful AI should be directly halted.

But on the other hand, there is also a large amount of skepticism.

Some people believe that current AI is still very far from "superintelligence" in the true sense, and discussions centered on extinction risks have been excessively amplified.

Some netizens also began to question Jacob's identity and personal experience.

It is worth noting that Jacob's current Twitter account only contains this single post.

Under Hubinger's post, another type of questioning is also very common:

Is Anthropic actively amplifying AI risks before its IPO, and implicitly demonstrating how powerful its own model is by emphasizing "how dangerous the model can be"?