A former OpenAI big shot made "extreme remarks": Large models will eventually collapse and become dumber as they learn, and the same goes for humans.

It's a curse for AI to have a photographic memory. Reinforcement learning is stupid and crazy, and the code is a complete mess. We'll have to wait another decade for AGI...

According to a report by Zhidongxi on October 21st, recently, Andrej Karpathy, the former director of AI at Tesla and a founding member of OpenAI, conducted a in - depth interview lasting 2.5 hours, systematically sharing his profound insights into the current situation and future of AI. Many of his views challenge the mainstream perception and are highly inspiring.

While the industry is generally discussing the "Year of Agents", Karpathy calmly proposed that this will be the "Decade of Agents". He pointed out that to create an AI agent that can truly work as reliably as an intern, we still need to overcome core bottlenecks such as multi - modality, continuous learning, and the use of computers, which will take approximately a decade.

Taking his recently open - sourced nanochat project as an example, Karpathy revealed the "cognitive deficiencies" of large language models. In innovative programming tasks that require precise architectures, AI assistants, due to their excessive reliance on common patterns in the training data, will instead misunderstand customized code and add unnecessary complexity, ultimately "bloating the codebase and making it a complete mess".

Regarding reinforcement learning, the mainstream method for iterating large models currently, Karpathy's criticism is particularly sharp. He vividly compared it to "sucking in supervision signals with a straw", that is, laboriously allocating a single signal of the final result to the entire complex action process. He believes this is both inefficient and absurd.

By comparing with the way humans learn, Karpathy pointed out two key deficiencies in the current training of large models: "model collapse" leads to the exhaustion of the diversity of generated content, and "over - memorization" hinders them from discovering general patterns like humans.

This is like a vicious cycle: training a new model with data generated by the previous model will only make the results more and more narrow. Interestingly, Karpathy believes that humans also "collapse" and stick to the established rules, and the brain may be introducing randomness (entropy) through "dreaming" to counter this trend.

When measuring the progress of AI, Karpathy believes that we should pay more attention to its actual performance in specific economic tasks rather than abstract indicators. He predicts that the promotion of AGI to macro - economic growth will be slow and continuous, and it will, like technologies such as computers and mobile phones, quietly integrate into and continue the existing GDP growth trend of about 2%.

Although the prospect is long, Karpathy firmly believes that we are already in the "intelligence explosion". He elaborated from a broader perspective: from a historical scale, this transformation is like a "fireworks feast", and we are witnessing it in slow - motion.

The content of this interview has spread widely on foreign networks. Many netizens on the social platform X commented that it is a "must - watch", "every minute is precious", and "absolutely worth watching"... Where do these subversive views come from and what do they point to? Zhidongxi intercepted the core content of the 2.5 - hour interview and edited it without changing the original meaning.

01. AGI Still Needs a Decade, and Multi - modality and Continuous Learning Abilities Are the Bottlenecks

While the industry is discussing the "Year of Agents", Karpathy proposed that this is the "Decade of Agents". So what bottlenecks need to be overcome in a decade?

The main content of the conversation is as follows:

Dwarkesh Patel: Andrej, why do you say it's the decade of agents instead of a year of agents?

Andrej Karpathy: This is actually a response to a famous quote before. I'm not sure who said it, but they implied that in terms of large language models and their future development, now is the decade of agents. This quote touched me because there are some over - predictions in the industry.

Some early agents are impressive. For example, Claude and Codex have been widely used, but I still think there is a lot of work to be done. We'll be collaborating with these things for a decade.

Dwarkesh Patel: What do you think will take a decade to achieve? What are the bottlenecks?

Andrej Karpathy: To really make it work. We imagine agents as hired employees or interns, but currently, they obviously can't do that because they are simply not good enough - not intelligent enough, not multi - modal enough, unable to use computers, and lacking continuous learning ability. Their cognitive ability is insufficient, so they are useless. It will take about a decade to solve all these problems.

Dwarkesh Patel: Why a decade instead of one year or fifty years?

Andrej Karpathy: This involves some of my intuition, and it's also a bit of inference based on my experience in this field. I've been in the AI field for nearly twenty years, which is not particularly long. My past experiences make me think that the current problems are solvable and can be overcome, but the difficulty is still significant. If I weigh it all up, I feel it will take about a decade to overcome them.

02. Large Language Models Still Have Cognitive Deficiencies, and Programming Models Make a "Complete Mess"

On October 14th, Karpathy open - sourced nanochat, claiming that it can train a "simplified ChatGPT" with less than $100 (about 711.5 RMB). Once released on GitHub, it received 5.6k stars.

However, when building the nanochat codebase, Karpathy found that the current AI programming assistants have obvious limitations and are hardly helpful. Why is this?

The main content of the conversation is as follows:

Dwarkesh Patel: You mentioned on Twitter that the programming models were hardly helpful when you were building the (nanochat) codebase. Why?

Andrej Karpathy: I spent about a month building this codebase. I think there are mainly three ways for people to interact with code currently: completely rejecting large language models, using the auto - completion function in the models to assist in writing (which is my current stage), and ambient programming. I use them in specific situations. But these are all tools you can use. You must understand what they are good at, what they are not good at, and when to use them.

Nanochat is not a suitable scenario because it is a rather unique codebase. It is almost intellectually intensive code, and everything has to be written very precisely. These programming models have many cognitive deficiencies. For example, they always misunderstand the code because they memorize too many typical ways of doing things on the Internet, which I don't use at all.

Dwarkesh Patel: Can you give an example?

Andrej Karpathy: I'm using eight GPUs, all of which are performing forward and backward calculations. The method to synchronize gradients between them is to use PyTorch's Distributed Data Parallel (DDP) container. When you perform backward calculations, it will automatically start communication and synchronize gradients. I don't think it's necessary to use the DDP container, so I removed it. But the programming models tried to make me use the DDP container and kept trying to mess up the code style.

They are too defensive and always try to build a production - ready codebase, but I don't need these extra things. So I think they bloat the codebase and increase the complexity. They keep misunderstanding and use a lot of deprecated APIs. It's a complete mess and hardly useful.

03. Reinforcement Learning Is Terrible, Like "Sucking in Supervision Signals with a Straw", Stupid and Crazy

Reinforcement learning is a mainstream method for iterating large models currently. However, Karpathy sharply pointed out that in essence, this method is "sucking in supervision signals with a straw" - laboriously allocating a single result signal to the entire complex action process, which is both inefficient and absurd, and is completely different from the way humans learn through detailed review.

The main content of the conversation is as follows:

Dwarkesh Patel: Let's talk about reinforcement learning. How can we understand that humans can build a rich world model just by interacting with the environment, and it has little to do with the reward at the end of the scenario?

Andrej Karpathy: Humans don't use reinforcement learning. Reinforcement learning is much worse than I thought. Of course, other methods are even worse. In reinforcement learning, it almost assumes that every little detail of getting the correct answer is correct, but this is not the case. You may be on the wrong track before finding the correct answer. Every mistake you make, as long as you finally get the correct answer, will be weighted as "keep doing this". This is terrible. It's just noise.

You do so much work and only get a single number in the end. Based on this number, you add or subtract weights to the entire trajectory. I like to say that you are "sucking in supervision signals with a straw". You do so much work that could have been done in a minute, but now you are sucking in the supervision information of the final reward signal bit by bit and spreading it across the entire trajectory to add or subtract weights. This is stupid and crazy. Humans will never do this.

First of all, humans will never deploy hundreds of times. Secondly, when a person finds a solution, they will go through a rather detailed review process: "Well, I think I did well in these parts and not so well in these parts. I should do it this way or that way." They will think carefully. There is no such thing in the current training of large models. But I do see some papers trying to do this.

Dwarkesh Patel: Since this is obvious, why can't process - based supervision, as an alternative, successfully improve the model's ability? What's preventing us from using this alternative paradigm?

Andrej Karpathy: Process - based supervision means that we don't set up a reward mechanism at the end. After you finish 10 minutes of work, I won't tell you whether you did well or not. I'll tell you how well you did at each step. The reason we don't do this is that it's tricky to do it correctly. You only have a partial solution and don't know how to assign credit.

In fact, whenever you use a large language model to assign rewards, these behemoths with billions of parameters are easy to be manipulated. If you perform reinforcement learning on them, you will almost certainly find adversarial examples as the criteria for your model evaluation. So you can't do this for too long. You may only do 10 or 20 steps and maybe succeed, but you can't do 100 or 1000 steps. It will find all these false things in every corner and crevice of this giant model and find ways to deceive it.

04. Compared with Human Learning, AI Faces the Crisis of "Silent Collapse", and Photographic Memory Becomes a Pain Point

By making an analogy with the way humans learn, Karpathy talked about two important deficiencies in the learning of large models.

One is the model collapse problem. Currently, AI only passively predicts data and lacks human - like active thinking, resulting in the exhaustion of the diversity of generated content, that is, silent collapse.

The other is the over - memorization problem. Humans are good at finding general patterns because of their imperfect memory; while AI is trapped in precise memory and is weak in generalization. Karpathy believes that to enhance AI's thinking ability, we need to balance memory and algorithms, which is the core challenge.

The main content of the conversation is as follows:

Dwarkesh Patel: In the field of machine learning, what are the concepts corresponding to human daydreaming, sleeping, or simple reflection?

Andrej Karpathy: Indeed, we are missing some key dimensions in this regard. For example, take reading a book. Currently, large language models "read books" by passively predicting the next word and extracting knowledge from it. But this is not how humans learn - books are more like a set of "prompts" for us to stimulate thinking, discussion, and internalization. Humans truly master knowledge through this active processing, while AI completely lacks this mechanism. I hope that in the future, we can integrate in - depth reflection and digestion of materials into the pre - training of models, but this is still a research topic to be solved.

This is not easy. For example, why don't we train models with synthetic data? The core problem is that the samples generated by the model have "silently collapsed" - at first glance, the samples seem normal, but in fact, they only cover a very small range of content thinking, and the data distribution is too limited (for example, ChatGPT only tells a few jokes). Such samples lack the richness, diversity, and high entropy of human content and are difficult to support effective training. How to maintain entropy while avoiding collapse and make synthetic data work is still an unsolved research topic.

I think there may not be a fundamental solution to this problem. I also think humans will collapse over time, or rather, break down. That's why children are not overly adapted yet. They will say things that shock you because they haven't broken down, but we have. We will eventually repeat the same ideas, say more and more the same things, the learning rate will decline, the collapse will continue to worsen, and then everything will get worse.

Dwarkesh Patel: Have you read this super - interesting paper that says dreaming is a way to prevent this over - fitting and collapse? It can put you in strange situations completely different from daily reality, thus preventing this over - fitting.

Andrej Karpathy: This is an interesting idea. I think when you generate things in your mind and process them, you are training with your own samples, training with synthetic data. If you do this for too long, you will deviate from the right track and eventually collapse. You always need to find entropy in life. Talking to others is an important source of entropy, and so on. Maybe the brain has also established some internal mechanisms to increase entropy in this process. This is an interesting idea.

Dwarkesh Patel: Here's an immature thought. Toddlers with poor memory and a tendency to forget are actually good at learning new languages and exploring the world. Large language models can accurately repeat information but have difficulty quickly grasping abstract concepts. Adults are in between. Is there something worth exploring here?

Andrej Karpathy: Humans are more likely to "miss the forest for the trees" than large language models. We are not so good at memory, but this is a feature rather than a defect.

Large language models have extremely strong memory. They are troubled by the memory of pre - training documents. In a sense, this may be very distracting for them. Compared with large language models, humans are not so good at memory, so we are forced to find patterns in a more general sense. This is a human characteristic rather than a defect because it forces you to only learn the generalizable parts.

So, when I talk about the core of cognition, I hope large language models will do less memorization and only retain the algorithms for thinking, ideas for experiments, and all this cognitive glue for action.

Dwarkesh Patel