OpenAI Ex-Co-Founder: Where Exactly Is the Sticking Point for Recursive Self-Improvement?

Dwarkesh Patel, a well-known American podcaster, recently invited three researchers working at cutting-edge AI companies to hold an in-depth discussion on the technical bottlenecks and prospects of AI Recursive Self-Improvement (RSI). The three guests come from different "quasi-open" companies respectively, and are willing to share real technical judgments and divergent views in public.

They are Beren Millidge, Chief Technology Officer of open-source AI company Zyphra, John Schulman, Chief Scientist of Thinking Machines (former co-founder of OpenAI who led the development of Reinforcement Learning from Human Feedback (RLHF) that gave birth to ChatGPT), and Charlie O'Neill, Head of Model Training at model training platform Baseten.

The three guests hold different views on whether RSI will arrive, but they share a fundamental consensus: there are multiple unresolved bottlenecks in the current technical path, and RSI will not happen as rapidly as some people expected. Millidge believes the most likely obstacle comes from regulation rather than technology. Schulman notes that the model's "taste" and long-term judgment capabilities are still significantly weaker than those of humans, while O'Neill argues the key sticking point lies in whether AI can independently set research objectives, instead of only optimizing existing ones.

Regarding the catch-up progress of Chinese labs, the discussions of the three point to several structural factors: distillation itself only requires a small number of trajectories to replicate capabilities, making it the main counterweight to centralization. Cutting-edge labs purchase data from large data companies, and Chinese companies can also obtain the same data. Moreover, real-world user distribution is more important than the RL environment itself, which means the advantages of cutting-edge labs in RL environments may not be as prominent as people assume.

Regarding the effectiveness of Reinforcement Learning (RL), Millidge points out that a large number of successes attributed to RL actually come from intermediate training. RL itself only performs policy fine-tuning, but its signal-to-noise ratio is far higher than that of Supervised Fine-Tuning (SFT). O'Neill finds that the main generalization brought by RL is not horizontal (training in mathematics will not turn the model into the best programmer), but across time spans: the model learns to work continuously on longer tasks.

Regarding the timeline, the guests' judgments vary greatly: it will take 1 to 3 years for AI to become a fully competent remote worker, about 2 years for AI to deliver a 10x productivity boost to AI researchers, and 3 to 10 years for AI to surpass top human experts in all fields.

The following is the full transcript of the interview with Millidge, Schulman and O'Neill, edited by Tencent Technology, with deletions and adjustments made without altering the original meaning:

01 If There Is No Superintelligence in 2036, Where Will the Most Likely Bottleneck Be?

Patel: If by 2036, billions of frantically running superintelligences have not emerged in the world, what is the most likely technical reason? Exclude external factors such as political shocks, wars or bans.

Millidge: There is a possibility similar to Moravec's Paradox. AI may perform excellently on all benchmark tests, but there will always be a stubborn Sim-to-Real Gap that prevents it from achieving true generalization.

If generalized meta-learning is difficult to achieve and continual learning cannot be implemented either, AI may always be extremely powerful on benchmarks but fall short in the real world. To be honest, we have already seen signs of generalization brought by RL in practice, so I think the probability of this scenario is not high. The more likely obstacle comes from regulation.

(Note: Moravec's Paradox is a counterintuitive phenomenon in the AI field. Simply put, it is easy for computers to handle highly complex logical reasoning, but it is very difficult for them to imitate humans' simple perception and movement abilities.)

Schulman: Every time a new model is released, people will exclaim "This is AGI", but after using it for about a month, they will start to think it is stupid. This cycle may continue to repeat. Even if the model can write far more code than humans, it does not mean your productivity has increased by 100 times. You will still be stuck by the model's weak links and its poor judgment.

O'Neill: For me, the core question is how far the current Transformer+RL solution is from the global optimal solution. People imagine that once you have an AI researcher that is 0.1% stronger than all humans, multiplied by hundreds of thousands of parallel instances, you can overcome all bottlenecks.

But if another discontinuous breakthrough similar to the one from the pre-training scaling law to RL in the past is required, the current method may not be able to discover it. If this breakthrough is too far from the current paradigm, no matter how many LLMs you run, you may not find it.

Patel: Do you think this breakthrough will be more difficult than any breakthrough since 2012?

O'Neill: If we knew the answer, we would probably have achieved it already. But the key question is: is this breakthrough cumulative and can be found within the current paradigm, or do we need to completely discard gradient descent and neural networks? If it is the latter, no matter how many LLMs you run, you may not be able to discover it.

Patel: I want to put forward a contrary view. If deep learning continues to develop along the path of the past ten years, but still cannot let AI lead scientific research in the next few years, it would be very strange. Ryan Greenblatt believes that once AI can continuously improve its R&D capabilities through simulation, it may quickly cross human research levels, just like the fault that appeared after chess AI surpassed human Elo ratings.

Millidge: I agree. If AI does not cross that threshold, the most likely explanation is that it hits a capability bottleneck when approaching human level, or is subject to strong regulation. Compared with technical bottlenecks, I think the latter is more realistic.

O'Neill: The key is that there are two types of scientific research. One type has clear goals and can be continuously optimized. The other type requires putting forward completely new questions and paradigms, and these goals themselves cannot be defined in advance. The latter is where AI really struggles to make breakthroughs.

Patel: What is the last link in human AI R&D that will be automated?

Millidge: The hardest part to automate is not executing experiments, but iteratively asking the right questions. AI can do any experiment for you, but it may not know which one to do. When they talk about research, they mostly throw out scattered small steps — they are good at advancing existing directions, but not good at judging which directions are worth promoting.

O'Neill: Choosing the right direction is even rarer than technical discoveries. From DeepMind's "solve intelligence by playing games" to Radford's "try to predict the next word of massive data", there is not only the technology itself, but also the proposal of the scaling law, and the belief that it can be reliably predicted. Even after Radford identified the direction, it took a long time for people to decide to bet on it. Identifying the direction, believing in the direction, and betting on the direction — every step may get stuck.

Schulman: Defining goals, deciding what exactly we want. Even if AI can do all the technical work, we still need to decide how AI assistants should behave, what counts as helpful, and what the goals of RLHF are.

Alignment itself can be split into two parts: goal setting and goal realization. The former will not be automated in the short term. Think about why a post-training team needs so many people — it is because there are too many different fields that need to clarify how the model should behave.

02 Why Are Chinese Labs Catching Up So Fast

Patel: Why has there not been a major consolidation among model providers? Many factors point to centralization.

Schulman: Distillation is the main force against centralization. Anything that can be learned through RL can be easily distilled, because it only requires a small amount of data. If you can obtain trajectories that demonstrate certain behaviors from the model, you can easily distill it. There is another phenomenon worth paying attention to.

Some Chinese companies may be using proxy services that were originally designed to allow Chinese users to access blocked US cutting-edge models. These proxy services collect and sell data. This is a very useful dataset for distillation, because it provides a perfect prompt distribution.

Millidge: Cutting-edge AI labs purchase data from large data companies, and Chinese companies can also buy the same data from the same data companies. They can get exactly the same data, plus distillation, so it is actually quite easy to keep up.

O'Neill: Based on this, a specific prediction can be made. Sonnet 5 and Opus 5 are objectively inferior to GLM-5.3 and Kimi K3 in some aspects, even if they have Mythos' logit distillation available. This shows that the real-world user distribution is more important than the RL environment itself. The advantages of cutting-edge labs in the RL environment may not be as great as imagined.

Patel: So real-world deployment is more important than the environment?

Schulman: Let me offer a different hypothesis. The environment has two dimensions: difficulty and authenticity. It is relatively easy to increase difficulty, you just need to create a large number of high-difficulty puzzle-like tasks.

But authenticity is a different matter, it requires human feedback to define the reward function. If distillation is only carried out on the distribution of "brushing benchmarks", the student model can match the performance of large models on all benchmarks, but perform worse in a wider range of real scenarios.

Large models may have stronger generalization capabilities from narrow tasks to real tasks. This may also explain the problems with some smaller Anthropic models, such as Sonnet 5. Of course, it may also be that they are constantly adjusting the post-training stack, and some parameters are adjusted too high, resulting in quirks that cannot be seen on benchmarks.

03 Training AI Researchers: What Exactly Are We Training

Patel: How will the first model that can automate AI R&D be trained?

Schulman: It will probably be a combination of two approaches. Learn researchers' taste from human feedback, and at the same time create a large number of practice environments involving multi-step research projects. In each iteration, fix the worst parts of the previous round. Researchers will make extensive use of AI, notice their consistent weaknesses, and then fix them by collecting human feedback or creating environments.

O'Neill: The current practical approach is that we go to the forefront of the entire lineage, and turn the bugs that the company found in the training stack in the past few months into the environment. You need to train and progress at the cutting edge, so you lock in all the previous history.

You can imagine a world where you roll back the timeline to before GRPO, and then design an environment for AI to discover the optimal form of RL. But I think we will still be limited by computing power bottlenecks, so everyone will continue to stay at the cutting edge, and diff out the progress of the recent months to turn it into a training environment.

This is essentially distillation. You are always trying to catch up with the progress of the past three months. Of course, AI has contributed to these progresses, but humans are still in the loop. It feels like we are constantly approaching the level of human researchers inch by inch.

Millidge: But there is a key difference. Distillation on trajectories can never surpass the teacher. But the environment can far exceed human capabilities. It is very easy to design an environment that humans cannot solve but AI can try. For example, you say the loss needs to reach 1.3, no human can do it right now, but it is an extremely measurable and verifiable task. This is the path to surpass human AI research.

Schulman: A lot of research is not like climbing a mountain with a clear goal. More often it is like this: we have an intuition about how the model should improve, and there is an algorithm idea roughly moving in this direction. So we design a task to detect whether there are signs of life in this direction. If there are, we gradually build a more realistic version. You are not directly optimizing the goal you ultimately care about, you are relaxing your goal, first find an effective method in a less realistic scenario, and then return to authenticity after the method matures.

04 Will Long-Horizon RL Lead to AGI?

Patel: Are the current bets of labs to expand RL training in millions of environments and hundreds of fields, so that AI can learn general skills such as persistence, information organization, and collaboration, and eventually become a "remote worker" that can work independently for weeks or even months? But this approach only learns in the simulation environment, is it really enough?

O'Neill: I think it is hard to judge now how much of these efforts are for RSI and how much are just for building stronger general models. For the latter, this is indeed likely to be the direction that labs are betting on: start with coding, then expand to finance, PPT and other fields, and gradually cover more work scenarios.

Patel: But if the model is good enough at in-context learning, why do we need to train separately for fields like finance?

Schulman: Because domain-specific training can make the model run more efficiently. Even if the model can learn temporarily, these knowledge and skills can be directly baked into the weights through RL.

Millidge: And the two things can be done at the same time. Domain-specific training itself is not expensive, and the capabilities learned can also be migrated to RSI. Since there is enough computing power and parameters, labs will of course strengthen general capabilities while training high-value domains.

Patel: The longer the time span of the task, the harder it is to simulate in the data center. AI eventually needs to run businesses, conduct lawsuits, and make transactions. If the sim-to-real migration is not good enough, will the huge gap in sample efficiency of the model become a fundamental obstacle?

O'Neill: I divide tasks into two categories: cumulative and non-stationary. RSI is cumulative. Once you discover the attention mechanism, the Mixture of Experts model, and GRPO, you add them to the training stack, and they are yours.

OpenAI does not need to rediscover attention when training 5.6 Sol/5.6 Terra, it basically just calls pre-training.sh and post-training.sh. But real-world work is not cumulative. Imagine an AI agent in a law firm. You need to remember the relationships between all key people in the context, and these relationships are constantly changing. There are various implicit ways of doing things and ways to find information. This is not as clean as RSI.

Patel: It is really unfortunate that RSI is easier than being a legal assistant.

Schulman: The sources of the model's weaknesses are diverse. Some are indeed related to sample efficiency, for example, within a medium-length interval, humans can do weight updates more efficiently than models. But some weaknesses have nothing to do with this, such as thinking diversity is lower than humans, or poor performance in some long-term judgments. What many people call "taste" is actually about what systems are maintainable and can work well in the long run.

Patel: If you give the model a trillion-token context window, and stuff all the experience of a person before doing RLHF into it, will the taste problem be solved?

Millidge: Theoretically, if the model can learn effectively on long contexts, it should be possible. Humans do not need extremely long time to develop taste. A PhD takes about five years, works on 10 to 30 research projects, but their taste develops quite quickly. AI will obviously have far more experience than humans to develop taste. The question is to what extent this can be generalized to truly long-cycle tasks.

O'Neill: Regarding continual learning, the current technical dilemma is as follows. At the macro level, putting deployment data into intermediate training is effective. But when you zoom in to the micro level and try to perform continual fine-tuning on a single model with a small amount of data, all methods collapse a bit. You will see catastrophic forgetting and degradation of the base model's capabilities. Both SFT and on-policy distillation are too destructive. RL is good at injecting capabilities because it only changes a very small part of the model, but this also limits the amount of changes that RL can make.

Millidge: If you keep training the same base model forever, it will eventually asymptote. You can't keep learning new things on the same foundation forever. That's why everyone eventually has to train new models from scratch. Training from scratch