New Breakthrough in Reinforcement Learning: Tsinghua Team Proposes "Single Rollout Asynchronous Optimization" Method Enabling Stable 1000-Step Training

In long-horizon Agent and coding tasks, improving the efficiency of Reinforcement Learning (RL) remains a core challenge in current research.

At present, most existing LLM RL pipelines adopt synchronous, batch-based workflows, which often require waiting for an entire batch of rollouts to complete before performing updates, limiting efficiency gains. While asynchronous RL can alleviate the synchronous waiting problem, existing methods mostly focus on improving system throughput, with less consideration for training stability and task performance; furthermore, the GRPO method relies on grouped sampling, making it difficult to directly adapt to asynchronous Agent training.

To address this issue, a research team from the KEG Laboratory at Tsinghua University proposed the Single-rollout Asynchronous Optimization (SAO) method. SAO replaces the grouped sampling of GRPO with single-rollout sampling, meaning each input task generates only one rollout that can be immediately used for training, thereby reducing off-policy impact and enhancing generalization capabilities.

Paper link: https://arxiv.org/abs/2607.07508

Research results show that SAO can stably train for 1000 steps, and consistently outperforms GRPO and its variants on Agent coding and mathematical reasoning benchmarks; at the same time, single-rollout RL also demonstrates adaptability to dynamic environments in simulated online learning tasks.

This work provides a feasible approach for efficient RL training, and has been deployed in the Agent RL training pipeline of GLM-5.2 (750B-A40B).

Figure | Performance of SAO on reasoning and code benchmarks.

How does SAO stabilize asynchronous training?

In terms of overall framework, SAO adopts single-rollout asynchronous optimization: each input task generates only one training rollout, which immediately enters training once completed. To ensure smooth operation of the workflow, SAO designs Direct Bilateral Importance Sampling, value model enhancement, and multi-turn Agent trajectory processing mechanisms, and expands the scale of value model pre-training data. The specific process is as follows:

1. Direct Bilateral Importance Sampling (DIS): Used to reduce off-policy deviation in asynchronous training. Since rollout generation and model training are not synchronized, SAO does not need to save a large number of historical models, and directly uses the token log probabilities recorded during the rollout phase to calculate the importance sampling ratio, with tokens exceeding the set interval not participating in gradient updates.

2. Increase the update frequency of the value model: Used to reduce the gradient estimation variance in single-rollout optimization. The gradient variance of single-rollout training is higher, so a more reliable value model is needed to estimate the advantage signal. SAO decouples the update frequencies of the policy model and the value model: for every one update of the policy model, the value model is updated twice, thereby reducing training variance.

3. Fix the parameters of the attention module: Responsible for stabilizing value model training. The research team found that the fluctuation of the value model mainly comes from the full attention layer, while the Mixture of Experts (MoE) layer is relatively stable. Therefore, when training the value model, SAO fixes the parameters of the attention module and only optimizes the MoE projection layer to reduce fluctuations in value model training.

4. Token-level GAE that skips observation information: Used to reduce environmental feedback noise in multi-turn Agent trajectories. In multi-turn trajectories, model actions and environmental feedback appear alternately; if environmental observation information directly participates in advantage calculation, noise will be introduced. Therefore, SAO skips the observation information tokens of environmental feedback and only propagates the advantage signal between the actions generated by the model.

Figure | Overview of SAO with single-rollout design.

What are the experimental results of SAO?

Experimental results show that SAO performs better in mathematical reasoning, coding tasks, and simulated online learning; on multiple benchmarks, it outperforms the corresponding baseline models and GRPO series methods, and can maintain stability during long training processes. The specific results are as follows:

Mathematical Reasoning and Coding Tasks

The evaluation of SAO covers scenarios such as Olympiad-level mathematical reasoning with Python tools, complex mathematical question answering, and real-world code repair. The results show that SAO outperforms the corresponding baseline models and GRPO methods in these tasks. On AIME2025, SAO achieves an accuracy of 97.3%, which is higher than GRPO's 84.2%.

Figure | Experimental results on mathematical reasoning benchmarks (accuracy %).

On the coding task SWE-Bench Verified, SAO achieves an accuracy of 29.8%, which is also higher than the baseline model's 23.0% and GRPO (w/ DIS)'s 27.0%.

Figure | Experimental results on SWE-Bench Verified.

Training Stability

From the perspective of training curves, SAO basically maintains a leading position on different benchmarks, and the training process is also more stable. Ordinary GRPO experiences performance collapse relatively early; after adding DIS, GRPO can train stably, indicating that Direct Bilateral Importance Sampling can filter out token updates that deviate too much; in contrast, SAO maintains stability while its evaluation performance continues to improve in the middle and late stages of training.

Figure | Performance comparison between SAO and GRPO (w/ DIS) during the training process.

Training Dynamics

Experimental results show that the stability of SAO is related to its framework design. More frequent value model updates make the value estimates more consistent with the true returns; after fixing the parameters of the attention module, value model training becomes more stable; DIS controls the off-policy impact by filtering out token updates that deviate too much. In contrast, the VAPO control method based on the value model will experience training collapse at about 90 steps if DIS is not used.

Figure | Training dynamics of asynchronous single-rollout reinforcement learning.

Ablation Experiments

The research team further examined the role of each framework design. They adjusted the key components of SAO respectively: reducing the number of value model updates, switching to full-parameter value model training, and adding the VAPO control without DIS and the moving average baseline. The results show that the performance of these settings is lower than that of the full SAO; on BeyondAIME, the accuracy of the full SAO is 74.8%, and after removing the faster value updates, the accuracy drops to 69.8%.

Figure | Ablation results of value model training strategies and critic update frequencies.

Online Learning Simulation

The research team tested SAO's adaptability in non-stationary environments through a simulated online writing task. In the task, reward preferences switch between cute style, chuunibyou style, and classical style. The results show that SAO can quickly align with the new target style after the reward preference switches, and maintain a higher level on training rewards; in contrast, the moving average baseline has obvious adaptation lag, and the convergence level is also lower.

Figure | Online learning simulation under changing writing style preferences.

Limitations and Future Directions

However, the research team pointed out that although SAO performs stably in multiple types of Agent tasks, there are still limitations in terms of application scope, deployment infrastructure, and real-world online applications. To expand to more general scenarios, the following issues still need to be addressed:

1. Verify broader applicability

Current experiments mainly focus on large-scale Agent reasoning, coding tasks, and simulated online writing tasks based on Qwen3-30B-A3B. In the future, it is still necessary to verify whether SAO can be transferred to smaller models, non-Agent RLHF scenarios, and task environments with denser rewards and shorter rollouts.

2. Improve deployment infrastructure

SAO relies on a trained value model and rollout log probabilities. For practical deployment in the future, the training infrastructure needs to reliably save these probability information during the asynchronous generation process.

3. Improve safety guarantees for real-world online adaptation

Online learning experiments are still limited to controlled simulation environments. The research team pointed out that for future use in real user scenarios, stronger safety protection, monitoring mechanisms, and privacy review are still required.

4. Strengthen release and monitoring mechanisms

The research team pointed out that if the system lacks appropriate data filtering, access control, and evaluation mechanisms, the capabilities related to SAO may be used to optimize harmful objectives. In the future, SAO-based systems still require responsible release and continuous monitoring.

For more technical details, please refer to the original paper.

This article is from the WeChat public account "Academic Headline" (ID: SciTouTiao), written by Xia Qiansi, and published with authorization by 36Kr.