Ranked second on the daily ranking list, open-sourced by Peking University, the reinforcement learning data strategy features easier access and fairer comparison.
What data should be fed to the model in each round of reinforcement learning?
This question is becoming increasingly important with the development of inference models and RLVR. As the role of reinforcement learning in the post-training of large models continues to grow, what the model trains on and how it trains also requires more careful arrangement. A problem may be very difficult at first, but the model can answer it stably after a period of training. Which data should be retained, and how to adjust the learning weight and domain proportion, all need to be reconsidered in combination with the training state of the model.
However, there is often more than one data strategy method, and there are still some practical challenges in clearly comparing different methods. Different methods have their own implementations, and changing the screening rule may require modifying the trainer; when comparing effects, the model, reward function and training budget may not be consistent. Even if the score improves, it is difficult to judge whether the benefit comes from the data strategy or other training settings.
To solve the troublesome problems in data strategy access and effect comparison, The DCAI Team of Peking University, in collaboration with UCAS, Shanghai Algorithm Innovation Research Institute and Zhongguancun College, has newly released DataFlex-RL.
GitHub Open Source Repository: https://github.com/haolpku/DataFlex-RL
Paper Address: https://huggingface.co/papers/2609.06107
Spend less time repeatedly modifying the trainer and leave more energy to verify whether the data strategy is effective, which is the most direct value of DataFlex-RL.
DataFlex-RL reuses the training process of verl, and makes data selection, sample reweighting and domain proportion adjustment into configurable components, so that developers can configure three types of actions in the same set of RLVR/GRPO process:
- Select which rollouts participate in the current update;
- Assign what learning weight to samples or tokens;
- Determine from which domains the next batch of data is sampled.
These strategies directly reuse the signals already generated by the RL loop, such as reward, advantage, token probability and prompt grouping, and the code for rollout, verification and strategy update remains basically unchanged.
After the paper was released, it attracted the attention of the community and was listed as the #2 Paper of the day on HuggingFace. 591 experiments were organized in the paper to test data strategies on a variety of models and tasks including mathematics, logic and science.
Three Dynamic Data Strategy Dimensions: Selection, Weighting, and Mixing
In GRPO training, the model will generate multiple responses to the same question, the verifier will give rewards, and then the model will be updated according to the reward difference within the group. DataFlex-RL divides data strategies into three categories: selection determines which responses participate in the current update, weighting adjusts their contribution to the update, and mixing changes the domain composition of subsequent batches based on historical feedback.
Data Selection: Control Which Responses Participate in Strategy Update
The learning signals provided by responses generated for the same question can be very different. In GRPO based on intra-group reward difference, if all responses in a group are correct or all are wrong, it is usually impossible to form a discriminative advantage.
DataFlex-RL supports filtering based on intra-group solution rate, advantage ranking, reward variance and response efficiency.
difffilter is a specific example: generate five responses for each question, and only retain groups with a solution rate strictly between 0.2 and 0.8, that is, cases where two or three answers are correct, so as to focus the update on questions of medium difficulty. maxvar retains the subset with the largest intra-group reward variance, gfpo selects according to the ratio of reward to response length, and topk retains the responses with top-ranked advantage amplitude.
What is filtered here are the already generated responses. The current implementation controls the strategy loss through 0/1 weight, so it will not reduce the generation cost that has already occurred. When evaluating efficiency, it is also necessary to pay attention to the amount of data that actually participates in the update.
Dynamic Reweighting: Adjust the Learning Intensity of Different Samples
Sometimes researchers want to keep all responses in the current batch, while making some samples bear a larger update weight, and reweighting is used for this scenario.
DataFlex-RL supports response-level and token-level weights: softmax and per adjust the response contribution according to the advantage amplitude, diffband weights samples whose rewards fall within the specified quantile interval, and ar uses the existing token probability to impose relatively lower weights on low-probability tokens.
These methods normalize the average weight to 1, and try to control the change of the overall loss scale.
Dynamic Domain Mixing: Adjust Proportion According to Historical Performance
The learning progress of mathematics, logic and science tasks may be different, and the fixed proportion may not be suitable for each stage. DataFlex-RL uses a sliding window to accumulate training feedback of each domain, and then the mixer updates the target proportion of subsequent batches.
The mixing strategy will first accumulate feedback during the warmup phase, and then update the proportion at the configured interval. The current plug-in selects samples from available samples by domain through a custom replay buffer. The data that actually enters training is also limited by the number of candidates in each domain, so it is necessary to observe both the target proportion and the actual data distribution at the same time.
Unified Technical Architecture: Allow Data Strategies to Flexibly Access Training
The three types of strategies act in different positions, all of which need to access the training process, read feedback, and then convert the judgment into specific operations. DataFlex-RL accesses training through verl plugins, reuses existing signals, and separates scoring from execution, so that strategy components can be replaced independently.
Plug-in Access: Reuse the verl Training Process
To compare different data strategies, it is first necessary to make them run in the same set of training processes. DataFlex-RL accesses verl in the form of a plugin, and provides its own trainer through the registration mechanism.
For selection and weighting, the plugin reads the signal after the original advantage calculation is completed, runs the data strategy, and then writes the result to the existing rollout_is_weights field of verl. The selection result is converted to 0/1 weight, and the weighting strategy returns continuous weight, both of which are processed by the existing strategy loss interface. Domain mixing uses a separate trainer and a custom replay buffer to accumulate feedback and select available samples according to the updated domain proportion.
Different strategies share the model generation, reward verification and model update processes, and the changes are concentrated at the access position of the data strategy. When developers add new methods, they can reuse these training capabilities and focus their main work on the strategy components. The current plugin is oriented to the verl v1 trainer interface. Before use, you need to confirm that the host version is compatible and the plugin can be registered normally.
Share Training Signals to Reduce Repeated Scoring
After accessing the training, the strategy also needs judgment basis. Rewards reflect response performance, advantage provides strategy update signals, and token log-prob records token probabilities. These data have been generated along with training, and the scorer can directly read and calculate the required scores.
The strategy does not need to perform an additional model forward pass for scoring, and can reuse the same signal processing code. What is shared is the signal source and calculation logic, which does not require all strategies to use the same score; domain mixing also needs to accumulate feedback into historical statistics by domain.
The training process will also record the retention ratio, weight statistics, domain target proportion and feedback, which is convenient for checking whether the strategy runs as expected.
Decouple Scoring from Execution, Replace Strategies Through Components
With the scores, the next step is to decide how to use them. DataFlex-RL assigns these two steps to Scorer and Actuator respectively. The Scorer calculates the score according to the input signal, and the actuator converts the score into retention index, continuous weight or domain proportion.
With the scores, the next step is to decide how to use them. The same advantage amplitude score can be used for topk filtering, or it can be used for softmax to generate continuous loss weights.
Components are registered by name and then combined through configuration: when adding a new filtering method based on existing signals, usually only the execution rule needs to be implemented; when a new judgment basis is required, the scorer is extended.
Experimental Results: From Basic Training Gain to Dynamic Data Scheduling
A total of 591 experiments were carried out in the paper, covering different base models, as well as mathematics, logic and science tasks. The core experiments were carried out on Qwen2.5-7B-base, including 13 configurations, each configuration used 12 paired random seeds, and a total of 156 runs were completed. The experiment first confirmed that GRPO can effectively train the model, and then observed how the three types of data strategies participate in subsequent training.
On Qwen2.5-7B-base, the average score of the untrained model is 42.01, which rises to 49.77 after training with the standard GRPO baseline of uniform sampling, with an average improvement of 7.76 percentage points. On Llama-3.1-8B-base, the model also increased from 10.87 to 21.12, with an average improvement of 10.25 percentage points.
This set of results establishes the reference frame for subsequent experiments: GRPO brings basic training gains, and data strategies continue to arrange responses, learning weights and domain proportions.
The selection and reweighting experiments cover difffilter, maxvar, gfpo, topk, ar, per, softmax and diffband. They adjust training signals from the perspectives of response filtering, sample weight and token weight, and are compared under the same model, data and training budget.
In the domain mixing experiment, reward_gap, dump_ucb and tscl update the sampling proportion of subsequent batches according to the domain reward gap, exploration feedback and training change speed. Compared with the fixed equal proportion mixing, the average improvements of the three methods are 0.45, 0.61 and 0.16 percentage points respectively, which demonstrates the way to adjust the domain proportion according to training feedback.
On Llama-3.1-8B-base, the paper continues to test the selection, reweighting and domain mixing configurations, to verify that the same set of components can be migrated to different models and follow the same training and evaluation process.
In addition, Table 8 supplements this comparison from the evaluation side: when covering the three domains of mathematics, logic and science, the results obtained by different full aggregation methods are generally close; after adjusting the evaluation coverage, the ranking of methods will also change.
These experiments demonstrate the working closed loop of DataFlex-RL: during training, existing signals such as reward, advantage and token probability are read, dynamically adjust the sample selection and learning weight of the current batch, or update the domain proportion of subsequent batches; during verification, the strategy is switched through configuration, the same set of training and evaluation process is followed, the execution process is recorded and the results are compared.
DataFlex-RL not only provides a dynamic training framework, but also provides an effective verification interface, which is convenient for researchers to observe the actual role of data scheduling in different models, tasks and training stages.
From the perspective of the larger AI+Data ecosystem, the DCAI team of Peking University has previously open sourced DataFlex, and supports seamless integration with LLaMA-Factory, covering data operations such as Select, Mix, Reweight and Reorder. DataFlex-RL further extends this flexible data processing idea to the RLVR/GRPO training process, dynamically scheduling rollout and domain data according to training feedback. The two sets of tools complement each other, covering more links from data preparation to reinforcement learning updates, making data strategies more convenient to enter the actual training process.
About the Authors
Liang Hao, Ph.D. of the Big Data Science Research Center of Peking University, has won the National Scholarship, the President's Scholarship of Peking University for two consecutive years, and has published more than 10 CCF-A papers/journals as the first author.
He leads the design and development of the Data-Centric AI series of open source projects, which have accumulated tens of thousands of GitHub Stars. Among them, the DataFlow project won the championship of ICML SeePhy competition and the championship of Zhiyuan LIC Challenge. At