Oppose Agent, oppose RSI, Yann LeCun's new model rejects all current AI narratives.

There is a scene in Avengers: Endgame where Captain America whispers "Hail Hydra" in an elevator with S.H.I.E.L.D. agents.

With just this one line, the Hydra members disguised as S.H.I.E.L.D. agents hand over the scepter embedded with the Mind Stone to Captain America.

That is not all. Precisely because the Hydra members mistook Captain America for one of their own, the plan of Iron Man and other members to steal the Space Stone at the same time was not exposed. In the end, the Avengers successfully eliminated Thanos and saved the universe.

You see, even a single sentence can have an impact on the entire universe. Yann LeCun's new world model H-JEPA aims to infer subsequent developments through the underlying logic of hierarchical reasoning.

For a small action, what result will it cause in the short term? What about the medium term? What about the long term?

This is exactly the case with the famous butterfly effect: a butterfly flapping its wings in Brazil may eventually trigger a tornado in Texas.

What on Earth is H-JEPA

The starting point of H-JEPA is Yann LeCun's JEPA, whose full name is Joint-Embedding Predictive Architecture.

Different from the video generative models we are using now, it does not draw the next frame. JEPA compresses the picture into a set of abstract vectors (the so-called latent representations), and then predicts the future in this abstract space.

It is like two people playing chess with their eyes closed. At this time, there is no need for a chessboard at all. You only need to say which move you are going to make, such as moving the general out of the palace and advancing the pawn one step. The other party can conceive the trend of the chessboard in their mind based on what they hear, and then respond.

After understanding JEPA, let's look at this H-JEPA. Its approach is to stack multiple such JEPAs layer by layer.

Each layer has its own observation encoder, action encoder and latent predictor, and each layer predicts the state after different time spans in its own representation space. The bottom layer manages the finest details and sees the nearest, while the upper layers see further and more macroscopically.

When planning, the order is reversed. The high-level layer first gives a rough target state, which becomes the sub-goal of the next lower layer, and the lower layer refines it until the bottom layer outputs the real action.

As early as 2022, Yann LeCun proposed the hierarchical JEPA in his paper A Path Towards Autonomous Machine Intelligence. It was only four years later that a reproducible complete implementation was available for the first time.

The underlying logic of hierarchy is a concept that Yann LeCun once proposed, called "going from New York to Paris".

For example, if you want to go from Beijing to Shanghai, the planning in your mind must be hierarchical. First think about "taking the high-speed rail", then think about "how to get to Beijing South Railway Station", and finally think about "which foot to step first".

You will never ponder how high the steps under your feet are when booking a ticket.

Different abstraction layers are supposed to retain different information. The layer for booking tickets only needs to know "time, price, train number"; the layer for stepping needs to know "center of gravity, steps, ground".

This is exactly the most troublesome point for Transformer. If you make predictions by each token, it is equivalent to thinking about booking tickets, going to the station, and going down the stairs together. The more steps there are, the more prone to errors.

To achieve a longer-term goal, you have to go through all the details in the middle.

The paper mentions that the flat single-layer model needs to expand many fine-grained steps in long tasks, the prediction error will accumulate, and the action search space will also become larger.

Regarding the hierarchical structure, the paper takes the Visual AntMaze maze task as the reference standard. The success rate of the single-layer LeWM is only 18%, and after adding three layers of H-JEPA, the success rate reaches 73%.

At this time, there is a question. Since adding three layers can achieve such a result, why doesn't Yann LeCun add up to 100 layers, or even 10000 layers?

In the Push-T task, the two-layer model performs better, and after adding three or four layers, the performance drops instead.

The paper speculates that the task trajectory is inherently short, which limits the training data available for the high-level layer. The three-layer model can only use about 58% of LeWM's transferable training data, and the four-layer model only has about 14% left.

In other words, the more layers, the better.

Why Does Yann LeCun Oppose Agents?

The most interesting part of H-JEPA is that its concept of hierarchy is completely opposite to the current mainstream Agent narrative.

The current approach of Agent is to split a long task into more steps and spend more tokens to try.

That is, plan first, then call tools, then check, and then start over. The finer the split and the more attempts, the closer it seems to completing the task.

However, Yann LeCun's paper mentions that adding hierarchical layers is more effective than simply adding steps. After hierarchization, the high-level layer only needs to decide "where to go" in a few steps, and the details are handed over to the lower layers, so the computational cost of planning is also reduced.

Top giants like OpenAI and Anthropic improve the capabilities of their models by expanding outward at the same granularity. Therefore, each iteration of the new model has longer context, more tokens, and more attempts. This is why after they have been shouting about price cuts for a long time, you find that it is getting more and more expensive when you actually use it.

What H-JEPA proposes is to expand upward. It increases the level of abstraction, allowing the model to use different representations at different time scales.

At Davos in January this year, Yann LeCun said that LLMs can pass the bar exam and write code, but they cannot really deal with the complex problems of the real world. Because there are no household robots or L5-level autonomous driving today.

He also said: "I can't imagine that a system that can't predict the consequences of its actions in advance can be made into an agent." By the way, when Yann LeCun said this, Demis Hassabis was right next to him.

H-JEPA is a world model. Before each action is executed, it will "perform once" in the latent space to see what state the short-term, medium-term, and long-term will lead to, and then select the action.

Therefore, Yann LeCun advocates such an idea: a real Agent does not "say a word, look at the reaction, and change it if it doesn't work". It should roughly know what kind of chain reaction it will cause before saying that word.

Another point is that today's Agents, especially programming Agents, mainly rely on reward training, that is, "add points if you do it right, deduct points if you do it wrong".

One of the important reasons why code has become the main battlefield of AI is that there are ready-made right and wrong signals for writing code. For example, whether the test can pass, whether there is a compilation error, and whether the benchmark score has increased.

An official article of Anthropic once mentioned that Claude can already run experiments and optimize code under the premise of preset goals and evaluation criteria, but it still has gaps in "choosing goals and judging what problems are worth studying". Therefore, the clearer the verifiable standard is, the better the Agent works.

H-JEPA takes the opposite path in this regard. The paper repeatedly emphasizes that its training "uses no reward and no reconstruction". It neither needs external scores, nor does it need to restore pixels bit by bit.

H-JEPA learns to predict the next state in the representation space from beginning to end.

The famous "Wet Monkey Theory" talks about exactly this matter.

There are 5 monkeys in the cage, with bananas hanging above. As soon as a monkey climbs the ladder to get the bananas, all the monkeys will be punished.

The monkeys don't know what the water gun is, but after being sprayed with water many times, they know they can't touch the bananas.

Why Does Yann LeCun Oppose RSI?

In May this year, a company called Recursive Superintelligence ended its stealth mode. It received more than 650 million US dollars in financing, with a pre-money valuation of about 4.65 billion US dollars.

One of the co-founders of this company is Tian Yuandong, the former research director of Meta FAIR. He also has another identity: an old subordinate brought out by Yann LeCun during the FAIR era.

What Recursive Superintelligence advocates is to let AI recursively improve itself. After all, the name of the company is exactly that, which is like the answer is hidden in the riddle.

RSI is so popular that even the people he brought out have gone to do it, but Yann LeCun didn't mention a single word about it in H-JEPA.

Because Yann LeCun believes that what is missing in today's world models, or even in his ideal Agent, cannot be made up by RSI.

Yann LeCun's point of view is that if you let AI improve itself repeatedly, what is improved is still the basic capability. For example, if you let the language model do RSI, in the end what it improves is still the language capability. The multi-modality and reasoning that the language model lacks will not grow by itself just because of more improvements.

Therefore, Yann LeCun thinks that RSI is just an amplifier, which amplifies what already exists.

From JEPA to H-JEPA, even if RSI works wonders, it won't help.

At the same time, to make RSI work, verifiable judgment basis is needed.

As mentioned earlier, whether the code test passes and whether the benchmark score rises are such signals, and H-JEPA clearly does not use rewards during training.

Yann LeCun believes that there is no ready-made judgment standard in the physical world. For example, if a robot pushes a cup crookedly, how many points should it get? In a world where there are no scores to refer to, how can "improvement" be measured?

Yann LeCun also dissed RSI in his paper incidentally.

In the DROID experiment, the data comes from the teleoperation videos of real robots, and the scenes, lights, and objects are constantly changing between different clips.

If you only let the model predict, it will learn "which scene this is" and ignore the manipulator that is actually moving in the picture. The paper calls this slow-feature collapse.

Researchers must artificially add an inverse dynamics constraint, requiring the model to infer the intermediate action based on the states before and after. Without this constraint, the model performance drops directly to zero. With it, the planning capability is restored.

In other words, letting the model learn "what action causes what" relies on a constraint designed by humans, not the model getting better and better by itself.

In addition, it is not the number of improvements that determines success or failure.

As the old saying goes, after reading a book a hundred times, its meaning reveals itself. But Yann LeCun believes that what really determines the success or failure of the model is not how many rounds of RSI have been done, but what should be retained and what should be discarded in each layer, and whether the deep model has enough data.

The aforementioned Push-T task, where the three-layer and four-layer models do not progress but regress, is the most intuitive example. Recursively deepening cannot automatically bring about growth in model capabilities.

In Yann LeCun's roadmap, compared with RSI, AI must first learn to represent the world correctly, and then it makes sense to "improve itself".

This article is from the WeChat official account "Letter AI", author: Miao Zheng, published with authorization from 36Kr.