Latest judgment from Ilya: Scaling Laws are approaching their limits, marking the end of the brute-force aesthetics in AI.

In the narrative system of Silicon Valley, Ilya Sutskever is one of the few names that can be called a "totem".

As an important driving force behind the ImageNet revolution, a co - founder of OpenAI, and a key creator of the GPT series, he has almost defined the forward direction of deep learning in the past decade.

But just as the global industry is betting on "stacking more GPUs and building larger models" and trying to break through the door of AGI through scaling, this technological founder has chosen another path: founding SSI.

In the early morning of November 26th, Ilya Sutskever, the CEO of Safe Superintelligence (SSI) and co - founder of OpenAI (hereinafter referred to as Sutskever), in an interview with podcast host Devaksh Patel, rarely and systematically discussed his core judgments on the current AI path:

1. The breakthrough in the pre - training era lies in that it provides a reusable "recipe" that almost surely works: prepare sufficient data, computing power, and a model structure that can support the scale, and the performance will continue to rise.

Now, the law of scale is approaching its limit. Reinforcement learning consumes a huge amount of computing power, but it cannot be regarded as a true "scaling". At the same time, the line between scaling and wasting computing power has become blurred.

This means that the industry is returning from "scale - driven" to "research - driven". Scientific research does require computing power, but it does not need the absolute maximum amount of computing power. What is really needed is the right questions and new methods.

2. During the stagnation era, AI companies will converge, but can still earn huge revenues. Even if innovation slows down, companies will still make significant progress and obtain high revenues. Differentiation may become more difficult, but "stagnation" does not mean "decline".

If models are limited to certain fields, they can still be extremely powerful. You can have many super - intelligent entities with narrow functions.

3. Humans can learn quickly because evolution has built - in powerful general - purpose learning mechanisms. Humans learn fast not because we are pre - installed with a large amount of knowledge, but because evolution has selected a small number of extremely useful priors for us.

The reason why we can master skills in an extremely short time is that the brain is a highly general - purpose and extremely efficient learning system. This provides inspiration for artificial intelligence: future breakthroughs will come from stronger learning methods, rather than simply scaling up.

4. When AI is strong enough, many social behaviors that do not exist today will emerge. Powerful AI is likely to bring "universal high income", greatly improve productivity, and reshape political structures, systems, and governance methods.

In such a world, everyone will have an AI acting for them. The real risk is that humans may gradually change from participants to bystanders. One of the answers to maintaining subjectivity is to establish a deeper coupling with AI, so that humans can continue to understand and participate in the situations where AI is located. Brain - computer interfaces are a feasible solution.

5. What is most worrying about superintelligence is not its intention, but its power: when a system is powerful enough to determine everything, even if its goals are benevolent, humans may still not like the way it achieves its goals. Power itself is the risk, not the motivation.

6. True research taste comes from the persistence in "aesthetic beliefs": beauty, simplicity, and elegance.

Research taste comes from the persistence in simple and clear beliefs: artificial intelligence should conform to the essential structure of the brain, but understand the brain in the right way.

What is important is not the shape of the brain, but the number of neurons, the plasticity of connections, and the ability to learn from experience through local rules. These are the foundations of "the way things should be".

When these elements appear simultaneously, you will have real confidence in a method. This confidence is a top - down belief: when the data does not support your conjecture temporarily, it can still support you to move forward, because you believe in the inherent correctness behind it.

The following is the transcript of Sutskever's interview:

01 The model gets full marks in exams but fails in practice

Ilya Sutskever: Do you know what's the most incredible thing? All of this is real.

Devalkesh Patel: What do you mean?

Ilya Sutskever: Don't you think so? All this AI stuff, and everything in the San Francisco Bay Area... All of this is happening. Isn't this like a plot from a science - fiction novel?

Devalkesh Patel: And there's something else that's incredible, which is that the slow start of AI feels so normal. Think about when we planned to invest 1% of GDP in AI. I thought it should be a big deal, but now it feels...

Ilya Sutskever: It turns out that we can get used to some things very quickly. But at the same time, it's also a bit abstract. What does this mean? It means that you'll see in the news that a certain company has announced a certain amount of financing. That's all you see. So far, you haven't really felt any other form of impact.

Devalkesh Patel: Should we really start from here? I think this is a very interesting discussion.

Ilya Sutskever: Sure.

Devalkesh Patel: I think your point, that from an ordinary person's perspective, everything doesn't seem much different, even after the singularity arrives, still holds.

Ilya Sutskever: No, I don't think so.

Devalkesh Patel: Okay, interesting.

Ilya Sutskever: The feeling I mentioned before, that it doesn't feel much different, for example, a certain company announces an incredibly large investment. I think no one knows how to deal with this investment.

But I think the impact of AI will eventually be felt. AI will penetrate into all areas of the economy. This will be driven by very powerful economic forces, and I think the impact will be very significant.

Devalkesh Patel: When do you expect the impact to occur? I think these models seem more intelligent than the economic impact they imply.

Ilya Sutskever: Yes. This is one of the most confusing things about these models at present. How to explain the fact that they perform so well in evaluations? When you look at those evaluation results, you'll think "these evaluations are quite strict". They really perform extremely well. But the economic impact seems to lag far behind. It's hard to understand. On the one hand, the models can make such amazing predictions, and on the other hand, in some cases, they repeat the same mistakes twice?

For example, suppose you use Vibe Code to do something. You go to a certain place and then encounter a bug. You tell the model: "Can you fix this bug?" The model says: "My goodness, you're absolutely right. I did encounter a bug. Let me fix it."

Then it introduces a second bug. Then you tell it: "You've encountered a second bug." It says again: "My goodness, how could I make such a mistake? You're right again." Then it puts the first bug back, and you can alternate between these two bugs. How is this possible? I'm not sure, but it does indicate that something strange is happening.

I have two possible explanations. The more whimsical explanation is that reinforcement learning training may make the model too single - minded, too narrow, and lack a certain sense of perception, although it also improves the model's perception in some aspects. Precisely because of this, they can't complete some basic tasks.

But there's another explanation. Previously, when people carried out pre - training, the problem of training data selection already had an answer, because the answer was all the data. Pre - training requires all the data, so you don't need to consider which data to use.

However, when people conduct reinforcement learning training, they do need to think. They'll say: "Okay, we want to conduct this kind of reinforcement learning training for this thing, and that kind of reinforcement learning training for that thing."

As far as I know, all companies have teams responsible for creating new reinforcement learning environments and adding them to the training portfolio. The question is, what exactly are these environments? The degree of freedom is so high. You can create a wide variety of reinforcement learning environments.

One thing you can do is that people will draw inspiration from evaluation results. You'll think: "Hey, I hope our model performs extremely well when it's released. I hope the evaluation results look great. What kind of reinforcement learning training can help us accomplish this task?" I think this situation does exist, and it may explain a lot of what's happening.

If you combine this with the situation where the model has insufficient generalization ability, it may be possible to explain a lot of the phenomena we see, that is, the disconnection between evaluation performance and real - world performance, and we don't even understand what this disconnection means today.

Devalkesh Patel: I like the idea that it's the human researchers who overly focus on evaluation results that are misusing the real reward mechanism.

I think there are two ways to understand or think about the point you just raised. The first way is that if a model only performs like a superman in programming competitions, it won't automatically become more tasteful or make better judgments on how to improve the codebase. Then you should expand the test environment so that it's not just limited to the best performance in programming competitions. It should also be able to develop the best applications for specific needs (such as X, Y, or Z).

The other view, perhaps this is what you want to express, is: "Why can't achieving super - human results in programming competitions make you a better programmer?" Perhaps the right way is not to blindly increase the number and variety of competition environments, but to find a way to enable you to learn from one environment and improve your performance in other environments.

Ilya Sutskever: I have an analogy that may be helpful to you. Since you mentioned algorithmic competitions, let's take it as an example. Suppose there are two students. One student is determined to become the best algorithmic competition player, so he devotes ten thousand hours to practicing in this field. He'll solve all the problems, remember all the proof techniques, and be able to implement all the algorithms quickly and accurately. In this way, he finally becomes one of the top players.

The second student thinks: "Oh, programming competitions are really cool." Maybe he practices for 100 hours, or even less, but he also achieves very good results. Which student do you think will develop better in his future career?

Devalkesh Patel: The second one.

Ilya Sutskever: Exactly. I think it's basically like this. These models are more like the first student, or even more extreme. Because we'll say that this model should be good at algorithmic competitions, so we'll collect all the algorithmic competition questions that have ever appeared. Then we do some data augmentation, so that we have more algorithmic competition questions, and train with these questions. Now, you get a very good algorithmic competition player.

I think it's easier to understand with this metaphor. Yes, if it's trained so well that it can easily master all different algorithms and proof techniques. But more intuitively, even if it has this ability, it may not be able to generalize to other fields.

Devalkesh Patel: Then, what about the things the second student did before the 100 - hour fine - tuning? What's the analogy for that?

Ilya Sutskever: I think they have that "trait". That "trait". I remember when I was an undergraduate, there was a student like that, so I know this trait exists.

Devalkesh Patel: I think it's interesting to distinguish "it" from the role of pre - training itself. Understanding what you said about pre - training not needing to select data, it can actually be understood as being very similar to ten thousand hours of practice. It's just that this ten - thousand - hour practice is free because it already exists in the pre - training data distribution. But maybe what you mean is that the generalization ability of pre - training is actually not strong. Although the amount of pre - training data is large, its generalization ability may not be better than that of reinforcement learning.

Ilya Sutskever: The main advantages of pre - training are: A, the amount of data is extremely large; B, you don't need to worry about which data to use for pre - training. This data is very natural and contains many behavioral characteristics of people: what people think and many personal characteristics. It's like a process in which people project the whole world onto text, and pre - training is trying to capture this projection using massive data.

Pre - training is difficult to understand because it's hard to figure out how the model depends on the pre - training data. When the model makes mistakes, could it be because some information just isn't fully supported by the pre - training data? "Supported by pre - training" may just be an inaccurate statement. I don't know what other useful information I can add. I think some human behaviors don't have a direct correspondence with pre - training.

02 Emotion, is a value function

Devalkesh Patel: Here are some analogies people have made about human "pre - training". I'm curious to hear your views on why these analogies may be problematic. One analogy is to think about the first 18, 15, or 13 years of a person's life. During this time, they may not be economically productive, but what they do allows them to better understand the world, etc. Another analogy is to imagine evolution as an exploration that has lasted for 3 billion years, ultimately forming a human life.

I'm curious whether you think these two are similar to pre - training. If not pre - training, how would you view human lifelong learning?

Ilya Sutskever: I think these two are somewhat similar to pre - training. Pre - training tries to play the role of both. But I think there are also some big differences. The amount of pre - training data is extremely large.

Devalkesh Patel: Yes.

Ilya Sutskever: Somehow, even if humans have a small part of the pre - training data, after 15 years of training, the knowledge they master is still far less than that of artificial intelligence. But whatever they master, their understanding is deeper. At that age, you won't make the mistakes that artificial intelligence makes.

There's another point. You may ask, could this be related to evolution? The answer is maybe. But regarding this matter, I think the theory of evolution may have more advantages. I remember reading relevant cases. One way for neuroscientists to understand the brain is to study people with damage to different parts of the brain. Some people will have strange symptoms that you can't imagine. This is really interesting.

I think of a relevant case. I read about a person who had brain damage, perhaps due to a stroke or an accident, which led to the loss of his emotional processing ability. So he can no longer feel any emotions. He can still talk eloquently, solve some simple puzzles, and his test scores are all normal. But he can't feel any emotions. He doesn't feel sad, angry, or excited. Somehow, he becomes extremely bad at making any decisions. It even takes him several hours to decide which pair of socks to wear. He also makes very bad decisions in terms of finance.

What does this show about the role that our innate emotions play in making us qualified agents? When it comes to the pre - training you mentioned, if you can fully utilize the advantages of pre - training, perhaps you can achieve the same effect. But this seems... Well, it's hard to say whether pre - training can really achieve this effect.

Devalkesh Patel: "That" what? Obviously, it's not just emotions. It seems to be something like a value function that tells you what the final reward of any decision should be. Do you think this isn't implicitly in pre - training to some extent?

Ilya Sutskever: I think it's possible. I'm just saying it's not 100% certain.

Devalkesh Patel: How do you view emotions? What's the analogy of emotions in machine learning?

Ilya Sutskever: <