In the era of self-evolution, reinforcement learning is likely to require a brand new set of algorithms.

In the previous article of this column Jeff Dean and His Peers, Jump Overboard Before Bayesian AI Arrives, when we talked about self-evolution, we mentioned that one of the critical bottlenecks is the extremely narrow search space of the model, where all solutions obtained after repeated attempts are nothing more than combinations and bridging work.

Current self-evolution research has begun to guard against such convergence of solutions, but most efforts implement modifications outside the model itself.

For example, the Darwin Gödel Machine developed by the Sakana AI and UBC team maintains a continuously expanding Agent archive. It generates variants from different ancestors and allows multiple lineages to develop in parallel. In experiments, the system's performance on SWE-bench increased from 20.0% to 50.0%, and on Polyglot from 14.2% to 30.7%. The authors repeatedly emphasize the role of stepping stones: a certain modification may fall behind temporarily, but later becomes the stepping stone required for a breakthrough.

Varun Gumma et al. proposed IDEAgent, which directly defines scientific research ideation as Quality-Diversity Search, that pursues both quality and diversity when selecting differentiated outputs after search.

The search process of a self-evolution system actually goes through two rounds of selection. The first round happens inside the model, where the trained model determines which candidates are worth generating. The second round takes place in the external loop, which decides which candidates output by the model can survive to the next iteration.

All the explorations mentioned above are focused on the second round, but there has been very little prior work on making the model itself produce more exploration diversity.

The more open the outer loop is, the more glaring this contradiction becomes: a carefully maintained evolutionary tree is managed at the back end, while the RL at the front end only provides the exact same single seed.

Long-horizon tasks will exponentially amplify the defects at the entry point. When solving a math problem, if one alternative route is not selected, the only loss is one chance to solve the problem. For a 200-round program optimization task, a data structure that does not appear in the 10th round will cause the parallelization scheme that could have been generated based on it in the 50th round to disappear entirely.

Every subsequent round can operate very smartly, but they can only keep climbing on a tree that has already been heavily pruned.

So how do we solve this first step problem?

At present, the solution is far from satisfactory.

01. The Dilemma of Reinforcement Learning

Let's first break down the problems faced by RL.

In early 2025, researchers used quite optimistic language when discussing reinforcement learning, and they all believed that RL could enable models to learn new capabilities.

On January 22, DeepSeek released the technical report of R1. The most eye-catching part is the R1-Zero model, which uses the GRPO algorithm that only obtains rewards based on whether the final answer is correct, but gradually develops behaviors such as extended thinking, going back to check, and trying alternative routes during the training process.

The paper describes these changes as reasoning capabilities that emerge naturally through reinforcement learning. As a result, everyone believed that RL allowed the model to learn reasoning methods that it had not mastered before.

A few days later, a paper including a research team from Google, titled SFT Memorizes, RL Generalizes, further reinforced this impression. The researchers made the SFT model and RL model start from the same base model, used the same training tasks, and then tested whether they could adapt to unseen rule changes.

The results showed that SFT was more likely to learn closely following demonstration answers, while RL was better at adapting to changes in rules and visual conditions.

This constituted the main evidence for the narrative of "RL learns reasoning through training and brings generalization" in early 2025.

RL not only allows the model to solve more difficult problems, but also suppresses low-probability solutions

However, the traditional academic circle has long held the view that RL is unlikely to bring generalization. In the past, in the CV field, RL methods were widely questioned precisely because of their poor generalization performance, and the high-performing AlphaGo could only play chess.

Therefore, reflections naturally followed. The April 2025 paper from Peking University titled Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? was the first shot fired in this discussion.

In the past, the most common metric for evaluating reasoning models was pass@1, which only let the model answer once to see if it could get the correct result. RL models far outperformed base models on this metric, so people naturally assumed that RL models had stronger reasoning capabilities.

However, this paper continuously increased the number of sampling times and observed the change of pass@k. The results showed that when the number of samplings was small, the RL model was significantly stronger. After the number of samplings was sufficiently large, the base model could catch up, and even surpass the RL model in some cases.

The researchers further compared the generation trajectories and found that the correct reasoning paths commonly used by RL models could often be sampled from the base model, albeit with a much lower probability.

This provides another explanation for the effectiveness of RL.

Assume that when a base model faces a problem, it may generate dozens of ideas. Correct ideas only account for a very small part of them, so the first answer is often wrong. RL training does not necessarily need to invent new ideas. As long as it continuously increases the probability of correct paths and suppresses the probability of wrong paths, it can achieve a huge improvement in pass@1.

From a product perspective, this is of course a capability improvement. Answers that originally required hundreds of samplings to encounter now appear in a single attempt, making the model more reliable and cheaper to run.

But if the research question is "whether the model has acquired capabilities that the base model did not originally possess", the conclusion is not so straightforward. It is more like improving the invocation efficiency of existing capabilities.

This paper thus proposed a highly influential explanation: what RLVR mainly does is distribution reshaping, that is, rearranging the probability of the base model's existing outputs. It pushes high-reward paths to the center of the distribution while squeezing other paths to the edges. The model does not simply "know more", but "trusts some of the paths more".

The narrative that "RL mainly sharpens the distribution" began to spread from this point.

On May 28, 2025, a research paper by a Chinese team titled The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models studied how the model distribution changes during RL training.

The authors observed that a large amount of RL training would experience a rapid drop in "policy entropy" in the early stage. The model's outputs become more and more deterministic, its exploration capability weakens accordingly, and the performance gradually enters a plateau at this stage.

The "entropy" here can be understood as how many choices the model retains at each step. High entropy means that multiple tokens have the opportunity to be selected, while low entropy means that the model is increasingly confident in only a few tokens.

The authors' analysis shows that policy gradient updates form a kind of self-reinforcement: tokens that already have high probability and positive advantage are more likely to be further enhanced. The higher their probability, the more opportunities they have to be sampled and reinforced in subsequent steps. Low-probability choices gradually lose the chance to enter the training samples.

This makes sharpening no longer just a guess on the pass@k curve, but has an explanation in training dynamics. RL samples from the current policy, and then uses these samples to update the current policy. Once a route takes the lead, it is more likely to appear in the next round and more likely to continue leading.

Training seems to still be in free exploration, but the actual candidate set used may become narrower and narrower.

During this period, many papers found in tests that RL-trained models might produce more diverse answers, thus casting doubt on the sharpening conclusion.

But in July 2025, the Stanford paper titled The Invisible Leash: Why RLVR May or May Not Escape Its Origin attributed this phenomenon.

The fact that the model has many tokens to choose from at each step does not mean that it will eventually generate many different solutions. Several reasoning trajectories that seem very different in language expression may eventually fall into the same solution and the same answer. Conversely, a rare route may only be triggered by a token with very low probability in the early stage, but can enter a completely different reasoning area.

This paper found that after RLVR training, token-level entropy sometimes even rises, but answer-level entropy still decreases. That is to say, the model's output seems more uncertain locally, but eventually converges to fewer answer types.

The authors further compared the empirical support sets of the base model and RL model under large-scale sampling, and found that the number of new answers added by RL is usually less than the number of answers it loses. pass@1 is improved, but some results that the base model could occasionally reach no longer appear.

Long-horizon tasks will further amplify this loss. A solution often needs to pass through multiple low-probability forks consecutively to form. As long as the alternative direction of the first step is suppressed, all subsequent search branches will disappear together.

Slight sharpening on a single token, propagated through dozens of decision steps, may lead to the exit of an entire family of solutions.

By October, the paper released by the Hunyuan team titled Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward proved this point.

Researchers found that there are some tokens in pre-trained models with very low probability, but which can lead to effective new reasoning. They call these tokens reasoning sparks.

Ordinary RLVR will impose excessive penalties on them, making them gradually disappear during training. By specifically protecting these low-probability but valuable tokens, the authors extended the RL training process and achieved higher final accuracy.

Then how does the sharpened RL show the possibility of generalization?

The answer is that sharpening and generalization do not necessarily occur at the same level.

The paper ProRL released by the NVIDIA team at the end of May last year may be one of the most powerful counterattacks to date against the claim that the model is simply sharpening. The authors did not settle for conventional short-horizon RL, but extended the training to more than 2000 steps, while adding KL constraints, reference policy reset, and more diverse training tasks.

The results showed that RL does not continue to sharpen in all fields. On mathematical tasks, pass@1 still increased while pass@128 decreased, but on code and some logical tasks, both pass@1 and high-k metrics could grow simultaneously.

The most prominent example is BoxNet, a class of planning tasks that require the agent to move in a grid, carry boxes, and avoid action conflicts.

Under the authors' experimental settings, the base model sampled 256 times for 100 test questions, but did not generate a single solution accepted by the verifier. Although the single success rate of the ProRL model was only around 7.7%, as the sampling volume increased, it eventually covered all test questions at pass@256.

This shows that RL, at least in long-horizon tasks, can push a batch of tasks that were originally below the limited sampling resolution into the solvable range.

From an external perspective, this is capability boundary expansion and "generalization": capabilities that the base model did not demonstrate in tens of thousands of generations can be stably demonstrated by the RL model.

In February 2026, a study by Peking University and other teams titled New Skills or Sharper Primitives? provided a mechanistic explanation for this. The researchers constructed four sets of artificial algebraic systems, only allowing the model to learn one-step operations during the training phase, while requiring the model to perform two to five consecutive steps during the testing phase.

RL has never seen these combined problems, but after training, some problems that the base model could not solve even after 128 samplings became problems that could be solved stably.

This is because assuming that a problem requires five consecutive steps to complete, and the correct rate of each step is 0.3, the probability of success of the entire reasoning chain will be so low that even 128 samplings often fail to encounter a successful case.

If RL increases the correct rate of each step to 0.7, the success rate of the entire problem will be greatly improved.

Here the model did not learn a sixth operation, nor did it see the complete solution in the test, but the entire problem has changed from being almost impossible to appear to being stably invocable.

In the experiment, the correlation coefficient between the joint correct rate of atomic steps and the final success rate of combined tasks reached 0.69 to 0.96, indicating that the emergence of complex capabilities can largely be explained by the underlying steps becoming more reliable.

This resolves the contradiction of "why sharpening can still bring generalization".

RL can perform sharpening on atomic capabilities, suppress wrong operations, and increase the probability of correct operations. These local changes, multiplied along the long reasoning chain, will open up previously unreachable areas at the task level.

It does not expand the set of basic knowledge and reasoning primitives, but expands the set of tasks that these primitives can stably combine to produce. Therefore, this is still a real form of generalization.

It is not that new knowledge is obtained outside of training experience, but rather compositional generalization. The model learns a set of more reliable, repeatedly invocable underlying operations, and then applies them to unseen rule combinations, longer reasoning chains, and new problem instances.

This also shows that RL is still indispensable. For complex problems, it is not enough for the model to only vaguely grasp each local step. As long as several links are unstable, the entire reasoning chain will quickly collapse as the task length increases. The role of RL is to train these potential capabilities to a level sufficient for continuous execution, turning combinations that originally remained in the low-probability tail into usable capabilities.

However, the problem that RL improves a batch of atomic capabilities while sacrificing other skills and low-frequency paths remains severe in the era of self-evolution.

It seems to be