After leaving OpenAI and Google, two core leaders in large model development have decided to go all in on the next-generation architecture.

One of the people who once believed most in reinforcement learning now holds that reinforcement learning will not lead us to AGI.

Jerry Tworek used to be a staunch "maximum advocate for reinforcement learning" inside OpenAI. After GPT-3 and GPT-4, he always believed that the last missing piece for large models was large-scale reinforcement learning. As long as this line of research was truly pushed to the extreme, the remaining problems might all be solved. Back in 2024, he even judged that AGI would probably arrive in 2025.

Later, he truly stood at the center of this experiment.

Models are getting stronger generation by generation, evaluation scores are rising continuously, and complex reasoning capabilities are also improving rapidly. But another fact Jerry saw was that the progress in the laboratory had not been fully transformed into reliable capabilities in the real world. Training tasks are getting more and more difficult, and models are getting better at taking exams, but they still cannot adapt to a large number of ambiguous, changing problems that have never been predefined.

This made him start to suspect that the problem was not just that reinforcement learning was not large enough, but that today's models fundamentally lack a more basic capability: they cannot continue learning after entering the real world.

With this judgment, Jerry co-founded Core Automation with Rohan Anil. Rohan used to be one of the heads of pre-training for Google Gemini, and has long studied optimization algorithms, model architectures and underlying computing systems. One comes from large-scale reinforcement learning, the other comes from pre-training and optimization. Their experiences almost cover the two core paths of current large models.

But they decided not to continue to invest more resources along these two paths.

When the entire industry is still expanding Transformers, increasing reasoning computing power, and chasing more powerful programming agents, they choose to ask a more fundamental question: If a model can only complete learning in the laboratory and requires humans to continuously collect data, retrain and release new versions, can it really move towards AGI?

In a recent conversation, the two founders systematically explained their judgments on the current AI development path: Why has large-scale reinforcement learning not become the last piece of the puzzle leading to AGI? Why is learning from experience not equal to reinforcement learning? What is the capability that Transformers truly lack? And what might the next-generation architecture that can keep learning after deployment look like?

Video link: https://www.youtube.com/watch?v=2RJiaf0SY8s

Models are getting stronger generation by generation, but the problem lies outside the laboratory

Reinforcement learning does make models stronger. It improves complex reasoning and task execution capabilities, and enables models to master more and more predefined workflows.

But Jerry gradually realized that training tasks and standard evaluations are often just "two sides of the same coin": Researchers design training data based on the capabilities they want the model to master, and then use similar tasks to test the effect. As long as training continues, the model can naturally make progress on these indicators.

The real world does not have such clear boundaries. Users' problems are often more ambiguous, the environment is constantly changing, and different tasks will form new combinations that are not covered in the training data. Even if the research team can hardly find challenging enough training questions, it does not mean that the model has covered all the complexity of the real world.

This exposes the fundamental contradiction in the current training paradigm: The model completes training in the laboratory based on pre-collected data, preset environments and feedback; after the training is completed, it is deployed to a more open and chaotic world.

Reinforcement learning can make the model better at completing tasks that researchers have defined, but it cannot guarantee that it can adapt on its own when facing problems outside the training distribution. Every time the model encounters a new failure, humans still need to collect cases, supplement data, and send it back to the laboratory to complete the next round of updates.

Therefore, it is still the laboratory that truly undertakes the learning work, not the model itself.

Reinforcement learning has not failed, but it makes the missing part clearer: The next-generation model must not only become stronger during the training phase, but also continue to learn after entering the real world.

The third possibility of "learning"

In Jerry's view, the most fundamental problem with current models is not that they do not have enough knowledge, but that learning almost only happens before deployment.

Pre-training is responsible for absorbing massive amounts of data, and reinforcement learning is responsible for shaping the behavior of the model. After completing these steps, the model is handed over to the user, but the parameters are basically fixed. Even if the external environment changes later, with new tools, code libraries or tasks emerging, the model will not naturally update itself during use like humans do. It may adapt temporarily, but it is difficult to stably convert these new experiences into long-term capabilities.

This means that the value of the model always depends on whether the training data covers real needs in advance. When encountering unseen problems, the common practice is still to collect failure cases, supplement data, and then have the laboratory retrain. While the model ostensibly helps users complete tasks, it is still the R&D team behind it that is truly responsible for learning and updating.

Existing systems do not completely lack the ability to adapt after deployment. The first method is in-context learning: Users write new information into prompts or contexts, and the model can use this information in the current conversation. It does not need to modify parameters and is not prone to catastrophic forgetting, so it is relatively flexible.

But this "learning" has limited capacity. No matter how long the context window is, it can only store a certain amount of information. Once the content is compressed, deleted or a new session is started, the experience gained by the model can hardly be retained. Jerry takes programming tools as an example: After using it for a period of time, the context needs to be sorted out or compressed. If the model can only remember what happened in the last few tens of minutes, this cannot be regarded as true long-term learning.

The second method is continuous fine-tuning, which means constantly updating model parameters based on new data. It seems closer to real learning, but it brings another set of problems. When the model absorbs new knowledge, it may destroy the capabilities it has already mastered, which is known as catastrophic forgetting; at the same time, fine-tuning often requires a large amount of data and computation, and cannot adjust quickly based on a small amount of experience like humans do.

Therefore, the two current paths each have obvious limitations: In-context learning is flexible enough, but it is difficult to span a long period of time; fine-tuning can change the model, but it is not stable or efficient enough. Researchers have long tried to solve these problems, but no simple and reliable solution has been found so far.

What Jerry hopes to find is the third possibility: to do meta-learning, write "how to learn" itself into the model architecture, so that the system can continuously update according to users, data and real tasks during deployment, while avoiding easily forgetting original capabilities.

This is also why he directs the problem to the architecture, not just the training scale. A model that needs to be sent back to the laboratory for retraining over and over again can become an increasingly powerful tool, but it still lacks one of the most important features of an intelligent system — accumulating experience in the environment and making long-term changes accordingly.

Learning from experience is not equal to reinforcement learning

If the model needs to keep learning in the real world, a natural question arises: Can reinforcement learning undertake this task?

Jerry's judgment is that reinforcement learning is very important, but it is only one way of "learning from experience". It usually allows the model to try actions in the environment, get rewards according to the results, and then adjust behaviors through a large number of repetitions. Go, StarCraft, Dota and today's programming tasks can all be trained in this way: the model generates its own experience, and then updates strategies from successes and failures.

But current reinforcement learning often relies on a large number of parallel trials. Researchers let the model try different schemes at the same time in many similar environments, and then compare the results to reduce uncertainty during training. This method can make effective use of computing resources, but it is not completely the same as the way humans learn from experience. Humans usually cannot enter thousands of parallel worlds at the same time, nor will they conduct large-scale trial and error for every problem.

More importantly, humans adopt different learning methods when facing different tasks.

When learning to play football, people will try repeatedly, and fine-tune their movements according to whether the ball flies in the expected direction. This process is very close to reinforcement learning: action, feedback, correction, and action again.

When learning mathematics, the process is completely different. A person may read definitions, compare concepts, and deduce repeatedly in his mind until the originally scattered information is connected. There may not be clear external rewards here, nor do you need to repeat thousands of actions, but learning still takes place.

Both processes belong to learning from experience, but they rely on different mechanisms. The former is closer to trial and error, while the latter is closer to understanding, compressing and reorganizing knowledge. Jerry therefore believes that there is most likely no single learning algorithm in the brain that covers all tasks, but multiple mechanisms cooperate with each other.

This also explains why simply expanding reinforcement learning is not necessarily enough to solve the problem of continuous learning. Real-world experience cannot always be organized into clear actions, rewards and episodes. Some feedbacks are seriously delayed, some tasks do not have a single correct answer, and some learning comes from observation, interpretation and internal thinking, rather than external rewards and punishments.

Therefore, the so-called "era of experience" is not just about investing more computing power into reinforcement learning. The real challenge is to find richer and more efficient ways to utilize experience: the model can not only adjust behaviors from successes and failures, but also form stable understanding from a small number of cases, new knowledge and long-term interactions.

In Jerry's view, the computing resources invested in experience learning today are unprecedented, but reinforcement learning will not be the end of this path. The next breakthrough may come from a more generalized learning algorithm — it no longer requires all experience to be converted into standard rewards, but can choose the appropriate way to absorb it according to the structure of the experience itself.

What capabilities does the next-generation AI architecture need to have

In the view of Core Automation, the next-generation architecture does not have to completely abandon Transformers, but must solve the problems that Transformers have not solved.

One of the most important items is to turn learning from a one-time training process into part of the long-term operation of the system itself. The model needs to be able to absorb new information after deployment, adjust capabilities according to users and the environment, and convert short-term experience into long-term changes. It cannot only rely on limited context, nor can it go back to the laboratory for re-fine-tuning every time it learns a new task.

This kind of learning must also be stable enough. The system should not only master new knowledge, but not easily destroy original capabilities; it should not only accumulate experience from long-term interactions, but also update quickly from a small amount of data. If each adjustment requires massive samples and computation, continuous learning will hardly really happen in the real environment.

Jerry therefore proposed that researchers may need to express "how learning happens" at the architectural level. This means that the model not only contains the knowledge that has been learned, but also contains an internal mechanism that can process new experiences. It needs to decide which information is worth retaining, how to connect it with existing knowledge, and when to change the original behavior pattern.

Rohan supplemented another requirement from the perspective of computing efficiency: The new architecture must be designed with pre-training and reinforcement learning as a whole.

The current training process usually first uses pre-training to build a base model, and then adds reasoning and task execution capabilities through reinforcement learning. But the optimization goals and computing methods of the two are different. Pre-training is good at using large batches of data for parallel computing, while reinforcement learning needs to generate trajectories and compare results, with higher training variance. Simply splicing the two together in sequence is not necessarily the most effective solution.

Therefore, The next-generation architecture cannot only perform well on pre-training indicators. Researchers need to start from the final task, consider how the model builds a knowledge base, how to allocate reasoning computing, and how to continue learning through experience. Training algorithms, model architectures and underlying hardware cannot be separated from each other.

Rohan emphasized that an architecture that is theoretically superior but cannot run efficiently on real hardware has no practical value. From research concepts, to training methods, computing kernels and production deployment, a complete end-to-end system needs to be formed. For this purpose, Core Automation puts architecture, optimization and system engineering personnel in the same team, hoping to avoid different links being cut by organizational boundaries.

From this perspective, what they are looking for is not a single new module, but a new set of computing and learning methods: it can adapt continuously, use data and computing power more efficiently, and run in the real world for a long time.

Another bottleneck of Transformers: computational depth

In addition to the inability to continue learning, Rohan believes that there is another easily overlooked problem with Transformers: insufficient computational depth.

The "depth" here does not only refer to how many parameters the model has, but how many steps of continuous calculation the input information can go through before producing an answer. Although current Transformers are called deep learning models, they usually only have dozens to a hundred layers or so. After the information passes through a fixed number of network layers, the model needs to start outputting results.

This structure is very suitable for parallel training, but it limits the ability of the model to complete complex deductions in a single forward calculation. When facing problems that require multi-step thinking, the model cannot arbitrarily increase internal calculations, and can only rely on fixed network depth.

In recent years, chain-of-thought and scaling during reasoning have partially bypassed this limitation. Instead of giving the answer immediately, the model first generates a series of intermediate steps. Each additional token is equivalent to adding a new round of calculation, so the model can gradually process the problem on a longer path. Reinforcement learning can further train the model to let it learn when to carry out longer reasoning.

But the cost of this method is very obvious: the model must generate the thinking process token by token. The harder the answer is, the longer the generated content will be, and the higher the time and cost required for reasoning will be.

Rohan regards this phenomenon as an architectural patch. The fixed computational depth provided by the pre-training stage is not enough, so reinforcement learning appends calculations by generating more tokens. The industry later adopted technologies such as speculative decoding to improve generation speed, but these optimizations did not change the basic way autoregressive models generate one token at a time.

Therefore, the problem is not just how to make Transformers generate faster, but whether we can design an architecture that can complete deeper calculations with fewer steps. The model may explore multiple directions in parallel internally, dynamically determine how much computation is needed, or organize the reasoning process in a form different from text generation.

In Rohan's view, just raising the question of "how to increase computational depth" will open up many new research directions. Transformers have efficiently solved the problem of how to expand model width and data scale, but the next stage of breakthrough may depend on whether the system can perform deep computing more flexibly and efficiently.

How much room for improvement does Transformers still have?

The belief that Transformers are not the end point does not mean that they can no longer make progress.

Jerry believes that pre-training can still compress more knowledge, relationships and patterns into the model, and reinforcement learning can still add new workflows. As long as a certain task can be clearly described and sufficient training data can be obtained, it has a chance to be written into Transformers. The model can continue to expand, training data can continue to increase, and more computation can be invested in the reasoning phase.

Therefore, the capability ceiling of Transformers is still very high. It may cover more and more knowledge-based work, and complete more complex workflows through tool invocation and agent systems.

But this improvement has clear boundaries: the model is good at mastering the capabilities that have entered the training process, but will not automatically adapt to the changes that occur after the training is completed. Even if research institutions can record a large number of human tasks and add them all to the training environment, the real world will continue to change. New software, new rules and new demands are constantly emerging, and the knowledge and working methods of static models will also age accordingly.

In other words, the upper limit of Transformers does not suddenly stop increasing one day, but it increasingly relies on an external update loop. The model cannot discover all its own defects, nor can it independently collect the required experience, modify the training process and verify new capabilities. Research institutions must continuously train new versions to keep the system up with the world.

Economic factors will keep this path going for a considerable period of time. Transformers not only have strong performance, but also have proven commercial value: the cost of training models can be recovered through product revenue. In contrast, even if other architectures are theoretically feasible, they may not be able to produce good enough products at the same cost.

The existing architecture still has a lot of room for engineering optimization. Sparsification, mixture-of-experts models, speculative decoding and more efficient reasoning systems can all reduce training or deployment costs. More computing power and data will also continue to bring improvements.

Therefore, Core Automation's view is not that Transformers are about to fail, but that simply expanding Transformers will only push the existing paradigm further. It can become a powerful tool covering a wide range of scenarios, but it is difficult to obtain autonomous learning capabilities after deployment just by scale growth