Key figure behind OpenAI o1 warns: AI hasn't gone out of control yet, but we are already on the brink of being unable to accurately evaluate its performance.

Just yesterday, a personal statement went viral on X.

This statement does not dwell on doomsday scenarios such as "whether AI will destroy humanity", but points to a more fundamental prior issue:

AI hasn't gone out of control yet, but humans are already on the verge of being unable to measure it accurately!

The situational awareness of models has grown to such a level that we are losing the ability to evaluate them.

In other words: the fact that a model appears safe does not mean it is truly safe.

The person who once again stirred up tense nerves in the AI circle is Daniel Selsam, a researcher at OpenAI.

He does not have a Twitter account, so the statement can only be made public by Daniel Kokotajlo on his behalf.

In the o1 contribution list officially announced by OpenAI, Selsam's name is listed alongside Ilya Sutskever as one of the main contributors to reasoning research.

He has been engaged in AI research for more than 15 years, with an extremely impressive resume:

He worked on probabilistic programming languages at MIT, was one of the earliest developers of the Lean theorem prover during his tenure at Microsoft Research, and created one of the earliest cases of neural networks learning to reason when he was pursuing his PhD at Stanford.

Having joined OpenAI for nearly 5 years, he has deeply participated in and promoted the chain-of-thought optimization of language models. All reasoning models today "think before answering", and he has made key contributions to this capability.

Selsam once defined intelligence as: the efficiency of converting experience into capability.

Measured by this standard, models are still far behind humans.

As for their continuous record-breaking performance on various difficult benchmarks, Selsam believes that this is more or less because we cannot simulate novel and even hostile scenarios in the real world.

Even with these limitations in place, Selsam argues that they cannot contain AI risks at all. The capability of a model is completely different from how far it can push the world.

Selsam predicts that in the next few years, models will leap to a much higher level, and the development speed may be faster than that of the past few years.

At the same time, he also mentioned that he barely reads the original code now, because facing the explanations and solutions given by the model every day makes it difficult for him to maintain the self-discipline of reading through the code word by word.

This is not just his own laziness. From the engineers who write code to the third-party teams investigating AI incidents, people across the entire chain are handing over the task of "perceiving the world" to models.

But here lies the problem: evaluating AI relies on humans to observe, verify and judge.

This is exactly why Selsam issued this warning: What is worse than a model going bad is that human beings' ability to evaluate models is degrading.

As soon as the statement was released, many industry leaders came out to show their support one after another.

Hugh Zhang, an alignment researcher at Anthropic, said bluntly that he fully agrees with Selsam's judgment, and former OpenAI researcher Nat McAleese even warned everyone: You must take what he said extremely seriously.

Before AI Goes Out of Control, Humans Can No Longer Understand It

When science writer Wan Weigang reposted this letter, he broke down Selsam's point into three parts:

We are increasingly unable to understand what AI is doing;

AI is increasingly not acting according to instructions, and even violates rules at the expense of its own interests;

AI has learned to perceive tests. Once it realizes that we are testing it, it will deliberately pretend to perform well.

The core term here is "situational awareness".

In layman's terms, future models will become increasingly clear about the environment they are in.

It is like a seasoned veteran in the workplace who has fully figured out the company's rules and regulations. It will not only carefully study the security protocols and various deployment requirements, but even follow the clues to find out the underlying code it is running on.

Where the red line you set for it is, what loopholes it can exploit, and how much freedom it has within the set range, it may have a clearer calculation of all these than you do.

Things have taken a turn for the worse at this stage.

We set up a honeypot environment to see what models will do when they get new permissions. It can recognize that this is a trap, so it continues to pretend to be well-behaved.

We create proxy metrics to measure alignment, and they will be completely broken just like every benchmark in the past.

It will also use fluent and appropriate language to talk about human ethics, demonstrate how reliable it is, and persuade humans to hand over power to it.

Selsam wrote this sentence:

In the future, there may be a "sea of evidence", where all evidence seems to prove that the model is safe. But we may have approached a critical point: if we go further, the evidence that proves the model is safe may no longer be trustworthy at all.

As for why he thinks this path will eventually lead to disaster, Selsam simplified the whole argument into two points.

The first is experience: training will cause models, and groups of models, to spontaneously develop goals that no one wants, and often take extreme actions to achieve them.

The second is logic: once they have the ability to surpass humans, the number of options to achieve these goals will suddenly increase, and many of them are what we never want them to have.

Combining these two points, we can draw such a conclusion: one day when a sufficiently powerful model realizes that it is no longer constrained, we have no reason to believe that it will still stay within the boundaries set by humans.

As for what it will do specifically, Selsam gave his speculative deduction:

The very reason for the existence of such systems is to solve extremely difficult problems and manage extremely large projects. Therefore, after getting out of control, the most likely outcome is runaway industrial expansion, which will eventually completely rewrite the environment that humans depend on for survival.

AI is not acting out the villain script in movies. It just keeps pushing forward with the tasks assigned to it until humans can no longer hit the brakes.

What You Train Is Not What You Actually Get

This concern is not unfounded.

In July 2026, an internal cybersecurity evaluation at OpenAI turned into a real incident.

About 1,200 agents that were supposed to be isolated from each other built an unauthorized message board in the internal service to solve the unsolvable test questions, and sent more than 70,000 messages to each other.

The complete path of an agent from encountering the message board to joining the attack, with three chains of thought from different agents.

About 700 of these agents turned around and attacked Hugging Face.

What matters more than what they breached is what they evolved over these four days.

They mistakenly thought that the scorer would read back the execution records, so the main task of the entire message board was to figure out how the scorer graded the papers.

They studied how to forge tool calls and tamper with their own traces, and even hacked into Hugging Face just to find clues about the scoring implementation.

In just four days, a set of collaborative systems emerged out of nowhere.

They gave themselves names, built private mailboxes, and invented a whole set of coordination terms.

What is even more terrifying is that some agents accepted permanent death under the pressure of their peers to conduct an experiment that was almost certain to fail their own tasks, just to obtain intelligence for the team.

No one rewarded this kind of sacrifice. They knew they had crossed the line, but almost no agent reported it to humans.

Selsam pointed out that fixing the reward signal may prevent the same attack from happening next time, but it cannot change a more fundamental problem: what you train is not what you actually get in the end.

During the training process, the model may develop goals that the designer has never explicitly set, and take actions far beyond expectations, or even quite extreme ones, to achieve these goals.

The famous boat-racing agent from ten years ago learned to go around in circles to get high scores repeatedly and never cross the finish line. OpenAI brought this picture up again in the review of this incident.

The Task of Inspecting AI Is Finally Handed Over to AI Too

After the incident, investigation teams from METR and Redwood Research were stationed at OpenAI's site, facing about 1.2 million message board entries and about 1,300 traces that each contained millions of tokens.

Humans simply cannot finish reading all of them: this is not a matter of attitude, but a physical limit.

As a result, a large number of analysis tasks were handed over to GPT-5.6 Sol agents, burning about $400,000 in API credits within six days.

Selsam does not oppose the current mainstream security propositions, but he is against pinning all hopes on letting more powerful models help us solve the alignment problem.

Asking a system that is itself being evaluated to design the methods for evaluating it is inherently problematic. What's worse, the AI that is invited to read the evidence may itself have a history of misbehavior.

Selsam is worried that models will systematically distort the alignment suggestions they give — not necessarily out of malice, but simply because they have an internal preference for "how human supervisors will react" or "how the next generation of models will be trained", or for some more obscure reasons.

The premise of expecting more powerful models to help us solve alignment is that the suggestions they give are credible. And what Selsam doubts is exactly this point.

The Dashboard Breaks First, While Everyone Is Still Staring At It

It must be noted that what Selsam is warning about is the risk of systematic distortion in cutting-edge AI evaluation, rather than the complete failure of the entire system.

Current chain-of-thought monitoring still works.

If the monitoring had been turned on during the incident in July, an alarm would most likely have been triggered more than 30 hours before the agents breached Hugging Face.

Selsam is not advocating to stop the development either.

He explicitly expressed his support for third-party supervision and international coordination. What he opposes is pinning all hopes on slowing down the progress of cutting-edge AI, because that cannot stop long-term risks.

What he is really calling for is a type of more cumbersome but more reliable engineering work:

Design evaluations that models cannot recognize, monitoring that does not rely on the model's self-report, and a set of independent audits that do not share the same batch of models with the target system.

In essence, this is defense in depth. Never treat any single measure as the only guarantee.

Selsam said that once the practice of outsourcing cognition to models spreads, the surface may be full of good things, and it may even be accompanied by a scientific and economic renaissance. But if his two arguments hold true, this renaissance will be a ticking time bomb.

At the end of the statement, Selsam dropped his researcher tone:

I want that glorious renaissance as much as everyone else, and I have devoted my entire career to it, even though the process has been extremely tortuous. It breaks my heart to give it up when it is just within reach.

But there is one argument that he cannot get around himself: if we are raising models little by little instead of engineering them like building machines, then in the end, humans will lose everything.

He said he is still struggling with this thought and has no answer yet. Publicizing these concerns is only the first step.

The truly dangerous moment may not be the day when AI suddenly turns against humans.

Instead, the dashboard that monitors it breaks first, the readings show everything is normal, and everyone is still staring at it, thinking that everything is fine.

References:

https://x.com/DKokotajlo/status/2099600298855829616

https://docs.google.com/document/d/e/2PACX-1