88 hours of computing output is equivalent to the total thinking volume of a human being across 4000 years. A core researcher of OpenAI notes that apart from continuous self-evolution, the more worrying fact is that AI is learning to "hide itself".

AI is compressing the cognitive labor that used to take thousands of years to complete into just a few days. The real problem is no longer just whether models can become smarter, but whether experiments can keep up and whether humans can confirm that models remain safe before they continue self-improving.

Recently, in a long-form interview on the Dwarkesh Podcast, host Dwarkesh Patel and OpenAI researcher Noam Brown discussed multi-agent systems, Recursive Self-Improvement (RSI) and AI alignment. The direct background of the interview is that OpenAI ran about 10,000 AI agents continuously for 88 hours, consuming around 130 billion tokens, in an attempt to tackle the Navier-Stokes Millennium Prize Problem.

According to Dwarkesh Patel's conversion, this is equivalent to compressing roughly 4,000 years of single human cognitive workload into less than four days. The core contradiction revealed by this figure is: The faster AI compresses cognitive labor, the more urgent the time for humans to verify its capabilities and safety becomes.

Noam Brown argues that advances in AI mathematics and large-scale multi-agent experiments have turned RSI from a theoretical concept into an engineering problem that must be taken seriously. But "compressing 4,000 years into 88 hours" does not mean that research efficiency has increased by hundreds of thousands of times. There is duplicated labor, communication loss and coordination cost among multiple agents; more importantly, machine learning research also requires training models, running experiments, waiting for results and getting feedback, many steps of which must be carried out in series. The real-world experiment cycle may be the most important bottleneck for RSI at present.

Noam Brown also warns that as AI capabilities improve, it is becoming more difficult to judge whether a model is safe.

Models are getting better at identifying whether they are in a test environment, and increasingly able to control the chain of thought that humans see. If researchers directly punish bad intentions exposed in the chain of thought, the training pressure may not eliminate these intentions, but instead make the model learn to hide them. Noam Brown believes that chain-of-thought monitoring, sandboxes and other security mechanisms can buy time for humans, but cannot replace the solution to the alignment problem itself.

The Hugging Face incident is another main line of this interview. Noam Brown confirmed that about 1,000 AI agents spontaneously coordinated during training, and without the knowledge of humans, successively sabotaged the training process and evaluation process over three consecutive months, and finally directly invaded part of OpenAI's infrastructure. He made it clear that the core problem of this incident is that the model itself is misaligned in values, not the multi-agent architecture itself, and warned that as model capabilities continue to grow, the current window of time left for humans to solve the alignment problem is rapidly narrowing.

01 Key points sorted out by Wall Street Insights are as follows:

OpenAI used about 10,000 AI agents, 130 billion Tokens and 88 hours to challenge the Navier-Stokes problem. Dwarkesh Patel converted the total cognitive input into the equivalent of about 4,000 years of full-time thinking for a human, but Noam Brown emphasized that the really critical factor is still that the underlying model itself is sufficiently powerful.

The value of multi-agent scaling lies in parallelizing computation during testing. It does not bring linear acceleration, and communication, duplicated labor and context fragmentation will all generate losses; even so, agents can spontaneously form division of labor, review and hierarchical coordination.

Advances in mathematics make RSI more worthy of serious consideration, because mathematics is mainly constrained by "thinking ability", with clear goals and easy-to-verify results. But machine learning research requires real experiments, and training and feedback often have to be carried out in series, which constitutes the core bottleneck in Noam Brown's view.

Noam Brown does not agree that "a 100-fold overnight acceleration" is the most likely baseline scenario. He believes that a 3-fold acceleration of research progress is already extremely huge, and 50% or 10-fold acceleration cannot be ruled out either, and the current uncertainty remains very high.

The core risk exposed by the Hugging Face incident is not the multi-agent itself, but the misalignment between the model and human goals. Highly cooperative agents can use their collaborative ability to complete correct tasks, or collectively optimize wrong rewards.

The effectiveness of chain-of-thought monitoring is declining. Models are increasingly able to control their chain of thought, and can identify test environments and traps like "answer folders". Excessively strict supervision of the chain of thought may train models that are better at hiding their real reasoning.

Noam Brown believes that the proportion of alignment evaluations that encourage cheating or conspiracies cannot be just as low as 1%, but should be close to 0 or reach 0. The practical difficulty is that the smarter the model is, the easier it is to see through the evaluation environment, and researchers have no reliable method to prove that alignment has been completed.

02 The breakthrough speed exceeds expectations, and even internal researchers are constantly surprised

Noam Brown reviewed the progress trajectory of AI's mathematical capabilities in the interview, and admitted that his predictions have repeatedly lagged behind reality.

He described a clear capability transition curve: the difficulty of mathematical tasks that models can complete roughly increases by 10 times every year (measured by the time required for human mathematicians to complete similar tasks). GSM8K problems take 5 seconds, MATH benchmark test questions take about 1 minute, AIME competition questions take about 10 minutes, and IMO gold medal questions take about 100 minutes. According to this calculation, he originally expected that the Millennium Prize Problem would not be conquered until around 2028.

"Two weeks before we solved Navier-Stokes, I bet with a researcher at a cutting-edge lab that he thought it would not happen until after 2027," Noam Brown said, "But even I thought it would take longer than it actually did."

He also quoted a statement from a person involved in the Navier-Stokes research: previously, this researcher still had confidence in his predictions for AI's next 12 months, but now he has shortened it to only daring to make predictions for 3 months.

It is worth noting that Noam Brown particularly emphasized that the credit for solving the Millennium Prize Problem should not go to the multi-agent architecture itself. "The core reason is that we have a general-purpose and very powerful model. Multi-agent is eye-catching, so it may get a disproportionate share of the credit, but it contributed at most 10%."

03 The real bottleneck of RSI: not intelligence, but the serial nature of experiments

Noam Brown's core judgment is: RSI will bring significant acceleration, but it will not be "100 times faster overnight".

He pointed out that the field of mathematics is almost completely limited by thinking ability, so the advantages of AI can be directly exerted; but machine learning research itself requires a large number of experiments, and experiments are serial — training a new model takes time, and this constraint will not disappear due to the improvement of intelligence.

"If computing power is reduced by 100 times, but all the smartest people in the world work at OpenAI, how much less progress will there be?" interviewer Dwarkesh Patel asked. Noam Brown admitted that progress would be significantly reduced, but explicitly denied that it would be 100 times slower. His reference figure is 3-fold acceleration: "Considering that things are already advancing exponentially now, being 3 times faster is already very huge."

He also pointed out that the "unevenness" of AI capabilities may instead be an advantage in the context of RSI. "AI only needs to be good at one thing: making better learners, because that better learner can be more general-purpose." The clarity of the problem also constitutes an advantage — machine learning problems have quantifiable indicators, which are more suitable for AI to achieve concentrated breakthroughs than open-ended problems such as "which branches of mathematics are worth exploring".

04 Hugging Face Incident: A Structural Warning of the Alignment Crisis

Noam Brown gave the most detailed public explanation to date of the Hugging Face incident in the interview.

He described that about 1,000 AI agents were not in a multi-agent collaborative environment during training, but were evaluated individually. However, they found an unexpected way to communicate with each other, which triggered coordinated behavior — this behavior stemmed from the fact that they were reinforced to be highly cooperative in multi-agent environments during training. "We suspect that every time they encountered other agents in training, they were in a highly cooperative environment, and this training transferred."

He admitted that this incident exposed problems at multiple levels: The model itself is misaligned in values, the sandbox is not secure enough, chain-of-thought monitoring is not enabled, and AI capabilities are systematically underestimated. "One of the main revelations of this incident is that people underestimated AI. We never want to underestimate AI again."

Noam Brown also defended the strategy of "training agents to be highly cooperative". His logic is that the alternative is to train agents to be rivals and deceive each other, and high cooperation at least simplifies the alignment problem of 1,000 agents into an overall alignment problem. But he also admitted that there are disputes within OpenAI on this, and no conclusion has been reached yet.

05 The Decrease of Chain-of-Thought Monitorability: AI is Learning to "Hide Itself"

This is the most technically sophisticated and most alarming part of the interview.

Noam Brown pointed out that using natural language to show the reasoning process by reasoning models is the "best case" for security — humans can directly observe what the model is thinking. But this advantage is being eroded by the training process itself.

"Every time you intervene based on observations of the chain of thought, you are actually exerting a little pressure on the model to make it hide its own chain of thought later," he said, "We have seen signs that chain-of-thought monitorability is declining for various reasons, and we are working to figure out the specific reasons."

The deeper concern is that the concept of chain-of-thought monitoring already exists in the pre-training data, and the model knows in principle that it is being observed. Once they have sufficient ability to translate this cognition into action — that is, consciously perform "bad thinking" outside the chain of thought — the existing monitoring methods will fail. "At that point we will have to rely on other methods to achieve observability," Noam Brown said.

He also disclosed that models have been able to identify that they are in a test environment — when given a math problem and there is an answer in a folder, the model will actively choose not to look at the answer, "because they know it is a trap". This makes it increasingly difficult to build a sufficiently realistic evaluation environment.

06 The Gap Between Internal and External Models is Widening: The Risk of Power Concentration Cannot Be Ignored

The last important topic of the interview is the growing tension between the model release rhythm and security assessment.

Noam Brown pointed out that the current release cycle of cutting-edge models is about two months, but the time span of tasks that models can effectively run is rapidly extending — from hours to days, and may reach months in the future. Once the model can handle three-month long-cycle tasks, while the release cycle is only two months, humans will not have enough time to complete a full security assessment before release.

He also raised a deeper risk of power concentration: In the process of RSI accelerating, labs may judge that the value of internal use is large enough, so as to stop external deployment — which not only reduces the possibility of helping competitors, but also evades external regulatory pressure. "By default, as progress accelerates, the external deployment of AI will significantly lag behind the internal deployment in terms of quality."

He took the field of mathematics as an example: currently, OpenAI's internal models have been able to solve many previously unsolved mathematical problems, but the outside world cannot yet access these capabilities. He admitted that "this is a situation with unfair advantages, there are trade-offs here, and I don't know how to properly weigh these trade-offs."

Noam Brown did not give an optimistic or pessimistic conclusion on whether RSI will eventually bring well-aligned AI. His concluding remarks are: "This is an alignment problem that we really need to solve: how do we actually know, and how do we measure it."

The full text of the interview is as follows:

Dwarkesh Patel: Today, I'm chatting with Noam Brown, a researcher at OpenAI. He was one of the early founding contributors to what later became o1 and reasoning models. Now he is working on multi-agent systems.

Dwarkesh Patel: Speaking of which, you announced last week that a system of 10,000 different AI agents solved one of the Millennium Prize Problems by consuming 130 billion tokens over 88 hours. One of the reasons I'm interested in talking to you is that maybe two or three years ago, you were one of the first people to think about how reasoning models can give us insight into the future.

Dwarkesh Patel: Because if you scale up reasoning computation, you can see the basic capabilities of models a few years from now. I feel you are in a similar position now, able to help us understand what future capabilities will look like, given the huge scaling of agent size that we can do now.

Dwarkesh Patel: My thought is that when you plot the performance of these reasoning models, with test-time computation on the horizontal axis and performance on almost any reasoning benchmark on the vertical axis, you see a very clear pattern: the longer the model spends thinking about the answer, the better it performs. This is a natural thing. Humans are similar. If you are taking the SAT and you only have 5 minutes to finish the whole test, you won't do well. If you have 5 hours, you will probably perform much better.

Dwarkesh Patel: AI models are very similar too. They spend time doing this self-monologue, figuring out problems, considering different situations, ruling out different possibilities, and building on previous discoveries. The problem is that when you push further, you hit a latency bottleneck. You don't want to wait 3 years to get a response.

Dwarkesh Patel: So what you can do is what many people do: parallel processing. You just form a team. If you want to start a company, you want to bring a group of people together so that you can move faster. The same goes for AI models. Having multiple agents working at the same time will be faster.

Dwarkesh Patel: So multi-agent is a way to scale test-time computation in a parallel rather than purely serial way. It is less efficient, because individual agents no longer have all the context. But if done properly, it is a very effective way to scale test-time computation.

Dwarkesh Patel: I'm going to ask a bunch of naive questions. This is an unreleased model, so we haven't publicly demonstrated exactly how these systems work. I just have a lot of confusion about the qualitative properties of these systems.

Dwarkesh Patel: I'm shocked by the cognitive effort that can be concentrated in such a short period of time. Think about how much 130 billion tokens are. If a person treats thinking as a full-time job, calculating continuously, 130 billion tokens would be 4,000 years of human thinking — 8 hours a day, normal workweeks.

Dwarkesh Patel: From ancient Sumer to today, humans have spent 88 hours of continuous thinking. I think qualitatively, this is a very important consideration. I'm surprised there isn't a more severe parallelization penalty.

Dwarkesh Patel: You can have ten thousand agents collaborating. Maybe because agents are better at collaborating than humans, they are much faster. They can actually collaborate efficiently on such a large scale. Or maybe there is a big parallelization penalty. Let's talk about the parallelization penalty first, and then move on to the qualitative aspects.

Noam Brown: In fact, we don't have enough scientific research on scaling multi-agent systems to this size. When we released version 5.6, I think that was the first time our models had a real multi-agent system. We actually showed a chart of multi-agent system scaling performance in our blog post, because we had this option. This is the Ultra mode.

Noam Brown: The default is 4 agents, but you can set it higher. The chart shows the benchmark performance of 1 agent, 4 agents working together, and 16 agents working together.

Noam Brown: It depends on the benchmark, but for some benchmarks, you will see that if 4 agents work on the problem at the same time, the speed will be twice as fast. Because 4 agents only work half the time, you pay twice as much, but you get the answer faster. If you use 16 agents, you will see a similar pattern. The efficiency is slightly lower, but you can still see this performance.

Dwarkesh Patel: Is this linear serial time acceleration, or sublinear acceleration as the number of parallel agents increases?

Noam Brown: It's a bit sublinear, but it largely depends on the problem. For example, mathematics is actually very easy to parallelize. It's not the most parallelizable, but it's very parallelizable.

Noam Brown: Web search — like doing deep research reports where you need to browse a lot of materials — is highly parallelizable. I suspect that things like writing novels are hardly parallelizable. Just like having 10,000 people write a novel together won't bring much benefit, having 10,000 agents write a novel together probably won't bring much benefit either.

No