Ranked second on the daily list, open-sourced by Peking University, the reinforcement learning data strategy features easier access and fairer comparison.
Which data should be fed to the model in each round of reinforcement learning?
This question is becoming increasingly important with the development of reasoning models and RLVR. As the role of reinforcement learning in the post-training of large models continues to grow, what the model is trained on and how it is trained also requires more careful arrangement. A problem may be very difficult at first, but the model can answer it stably after a period of training. Which data should be retained, and how to adjust the learning weight and domain proportion, all need to be reconsidered in combination with the training state of the model.
There is often more than one data strategy method, and there are still some practical problems in how to clearly compare different methods. Different methods have their own implementations, and changing the filtering rule may require modifying the trainer; when comparing effects, the model, reward function and training budget are not necessarily consistent. Even if the score improves, it is difficult to judge whether the benefit comes from the data strategy or other training settings.
To solve the troublesome problems in data strategy access and effect comparison, Peking University DCAI Team, in collaboration with UCAS, Shanghai Algorithm Innovation Research Institute and Zhongguancun College, has newly released DataFlex-RL.
GitHub Open Source Repository: https://github.com/haolpku/DataFlex-RL
Paper Link: https://huggingface.co/papers/2609.06107
Spend less time repeatedly modifying the trainer and leave more energy to verify whether the data strategy is effective, which is the most direct value of DataFlex-RL.
DataFlex-RL reuses the verl training process, and makes data selection, sample reweighting and domain proportion adjustment configurable components, allowing developers to configure three types of actions in the same set of RLVR/GRPO processes:
- Select which rollouts participate in the current update;
- Assign what learning weight to samples or tokens;
- Decide which domains to sample the next batch of data from.
These strategies directly reuse the signals already generated by the RL loop, such as rewards, advantages, token probabilities and prompt grouping, and the code for rollout, verification and strategy update remains basically unchanged.
After the paper was released, it received attention from the community and ranked as #2 Paper of the day on HuggingFace. 591 experiments were organized in the paper to test the data strategy on multiple models and tasks of mathematics, logic and science.
Three dimensions of dynamic data strategy: Selection, Weighting and Mixing
In GRPO training, the model will generate multiple responses to the same question, the verifier will give rewards, and then the model will be updated according to the reward difference within the group. DataFlex-RL divides data strategies into three categories: selection determines which responses participate in the current update, weighting adjusts their contribution to the update, and mixing changes the domain composition of subsequent batches based on historical feedback.
Data Selection: Control which responses participate in strategy update
The learning signals provided by responses generated for the same question may be very different. In GRPO based on the within-group reward difference, if all responses in a group are correct or all wrong, a discriminative advantage usually cannot be formed.
DataFlex-RL supports filtering by within-group solution rate, advantage ranking, reward variance and response efficiency.
difffilter is a specific example: generate five responses for each question, and only keep groups with a solution rate strictly between 0.2 and 0.8, that is, cases where two or three responses are correct, to focus the update on questions of intermediate difficulty. maxvar retains the subset with the largest reward variance in the group, gfpo selects according to the ratio of reward to response length, and topk retains the responses with top-ranked advantage amplitudes.
What is filtered here are the generated responses, and the current implementation uses 0/1 weights to control the policy loss, so it will not reduce the generation cost that has already occurred. When evaluating efficiency, it is also necessary to pay attention to the amount of data that actually participates in the update.
Dynamic Reweighting: Adjust the learning intensity of different samples
Sometimes researchers want to keep all responses in the current batch and let some samples bear greater update weights, and reweighting is used for this scenario.
DataFlex-RL supports response-level and token-level weights: softmax and per adjust the response contribution according to the advantage amplitude, diffband weights samples whose rewards fall within the specified quantile interval, and ar uses the existing token probability to apply relatively low weights to low-probability tokens.
These methods normalize the mean of the weights to 1, so as to control the change of the overall loss scale as much as possible.
Dynamic Domain Mixing: Adjust the proportion according to historical performance
The learning progress of mathematics, logic and science tasks may be different, and a fixed proportion may not be suitable for each stage. DataFlex-RL uses a sliding window to accumulate training feedback of each domain, and then the mixer updates the target proportion of subsequent batches.
The mixing strategy will first accumulate feedback in the warmup stage, and then update the proportion at the configured interval. The current plug-in selects samples from available samples by domain through a custom replay buffer, and the data that actually enters training is also limited by the number of candidates in each domain, so it is necessary to observe both the target proportion and the actual data distribution at the same time.
Unified Technical Architecture: Enable flexible access of data strategies to training
The three types of strategies act at different positions, and all need to access the training process, read feedback, and then convert the judgments into specific operations. DataFlex-RL accesses training through the verl plugin, reuses existing signals, and separates scoring from execution, so that policy components can be replaced independently.
Plug-in Access: Reuse the verl training process
To compare different data strategies, it is first necessary to let them run in the same set of training processes. DataFlex-RL accesses verl in the form of a plugin and provides its own trainer through a registration mechanism.
For selection and weighting, the plugin reads the signal after the original advantage calculation is completed, runs the data strategy, and then writes the result to the existing rollout_is_weights field of verl. The selection result is converted to 0/1 weight, and the weighting strategy returns continuous weight, both of which are processed by the existing policy loss interface. Domain mixing uses a separate trainer and a custom replay buffer to accumulate feedback and select available samples according to the updated domain proportion.
Different strategies share the model generation, reward verification and model update processes, and the modifications are concentrated at the access position of the data strategy. When developers add new methods, they can continue to use these training capabilities and focus their main work on the policy components. The current plugin is oriented to the verl v1 trainer interface, and it is necessary to confirm that the host version is compatible and the plugin can be registered normally before use.
Share Training Signals: Reduce repeated scoring
After accessing the training, the strategy also needs the basis for judgment. Rewards reflect response performance, advantages provide policy update signals, and token log-prob records token probabilities. These data have already been generated along with the training, and the scorer can directly read and calculate the required scores.
The strategy does not need to perform an additional model forward pass for scoring, and can reuse the same signal processing code. What is shared is the signal source and calculation logic, which does not require all strategies to use the same score; domain mixing also needs to accumulate feedback into historical statistics by domain.
The training process will also record the retention ratio, weight statistics, domain target proportion and feedback, which is convenient for checking whether the strategy runs as expected.
Decouple Scoring from Execution: Replace strategies through components
With the scores, the next step is to decide how to use them. DataFlex-RL assigns these two steps to Scorer and Actuator respectively. The Scorer calculates scores according to the input signals, and the actuator converts the scores into retention indices, continuous weights or domain proportions.
With the scores, the next step is to decide how to use them. The same advantage amplitude score can be given to topk for filtering, or to softmax to generate continuous loss weights.
Components are registered by name and then combined through configuration: when adding a new filtering method based on existing signals, usually only the execution rule needs to be implemented; when a new judgment basis is required, the scorer can be extended.
Experimental Results: From basic training gain to dynamic data scheduling
A total of 591 experiments were carried out in the paper, covering different base models, as well as mathematics, logic and science tasks. The core experiments were carried out on Qwen2.5-7B-base, including 13 configurations, each using 12 paired random seeds, and a total of 156 runs were completed. The experiments first confirmed that GRPO can effectively train the model, and then observed how the three types of data strategies participate in subsequent training.
On Qwen2.5-7B-base, the average score of the untrained model is 42.01, which rises to 49.77 after training with the standard GRPO baseline of uniform sampling, with an average improvement of 7.76 percentage points. On Llama-3.1-8B-base, the model also increased from 10.87 to 21.12, with an average improvement of 10.25 percentage points.
This set of results establishes the reference frame for subsequent experiments: GRPO brings basic training gains, and data strategies continue to arrange responses, learning weights and domain proportions.
The selection and reweighting experiments cover difffilter, maxvar, gfpo, topk, ar, per, softmax and diffband. They adjust training signals from the perspectives of response filtering, sample weight and token weight, and are compared under the same model, data and training budget.
In the domain mixing experiment, reward_gap, dump_ucb and tscl update the sampling proportion of subsequent batches according to the domain reward gap, exploration feedback and training change speed. Compared with the fixed equal-proportion mixing, the average improvements of the three methods are 0.45, 0.61 and 0.16 percentage points respectively, demonstrating the way to adjust the domain proportion according to training feedback.
On Llama-3.1-8B-base, the paper continues to test the configuration of selection, reweighting and domain mixing, verifying that the same set of components can be migrated to different models and follow the same training and evaluation process.
In addition, Table 8 supplements this comparison from the evaluation side: when covering the three domains of mathematics, logic and science, the results obtained by different full aggregation methods are generally relatively close; after adjusting the evaluation coverage, the ranking of methods will also change.
These experiments demonstrate the working closed loop of DataFlex-RL: during training, existing signals such as rewards, advantages and token probabilities are read to dynamically adjust the sample selection and learning weight of the current batch, or update the domain proportion of subsequent batches; during verification, strategies are switched through configuration, the same set of training and evaluation processes are followed, execution processes are recorded and results are compared.
DataFlex-RL not only provides a dynamic training framework, but also provides an effective verification interface, which is convenient for researchers to observe the actual role of data scheduling in different models, tasks and training stages.
From the perspective of the larger AI+Data ecosystem, the Peking University DCAI team has previously open sourced DataFlex, which supports seamless integration with LLaMA-Factory and covers data operations such as Select, Mix, Reweight and Reorder. DataFlex-RL further extends this flexible data processing idea to the RLVR/GRPO training process, dynamically scheduling rollout and domain data according to training feedback. The two sets of tools complement each other, covering more links from data preparation to reinforcement learning updates, making it more convenient for data strategies to enter the actual training process.
About the Authors
Liang Hao, Ph.D. of the Big Data Science Research Center of Peking University, has won the National Scholarship, the Peking University President's Scholarship for two consecutive years, and published more than 10 CCF-A papers/journals as the first author.
He led the design and development of the Data-Centric AI series of open source projects, which have accumulated tens of thousands of GitHub Stars in total. Among them, the DataFlow project won the championship of the ICML SeePhy competition and the championship of the Zhiyuan LIC Challenge. At the same time, he led the