Abstract
Multi-Task GRPO algorithm improves balanced performance across diverse reasoning tasks by dynamically adapting task weights and using a ratio-preserving sampler to ensure equitable optimization.
RL-based post-training with GRPO is widely used to improve large language models on individual reasoning tasks. However, real-world deployment requires reliable performance across diverse tasks. A straightforward multi-task adaptation of GRPO often leads to imbalanced outcomes, with some tasks dominating optimization while others stagnate. Moreover, tasks can vary widely in how frequently prompts yield zero advantages (and thus zero gradients), which further distorts their effective contribution to the optimization signal. To address these issues, we propose a novel Multi-Task GRPO (MT-GRPO) algorithm that (i) dynamically adapts task weights to explicitly optimize worst-task performance and promote balanced progress across tasks, and (ii) introduces a ratio-preserving sampler to ensure task-wise policy gradients reflect the adapted weights. Experiments on both 3-task and 9-task settings show that MT-GRPO consistently outperforms baselines in worst-task accuracy. In particular, MT-GRPO achieves 16-28% and 6% absolute improvement on worst-task performance over standard GRPO and DAPO, respectively, while maintaining competitive average accuracy. Moreover, MT-GRPO requires 50% fewer training steps to reach 50% worst-task accuracy in the 3-task setting, demonstrating substantially improved efficiency in achieving reliable performance across tasks.
Community
We propose a novel technique for multitask learning with GRPO without forgetting about worst-case tasks.
arXivLens breakdown of this paper 👉 https://arxivlens.com/PaperView/Details/multi-task-grpo-reliable-llm-reasoning-across-tasks-2314-93336a46
- Executive Summary
- Detailed Breakdown
- Practical Applications
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraints (2026)
- Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoning (2026)
- Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR (2026)
- AWPO: Enhancing Tool-Use of Large Language Models through Explicit Integration of Reasoning Rewards (2025)
- FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewards (2026)
- Clipping-Free Policy Optimization for Large Language Models (2026)
- Stable and Efficient Single-Rollout RL for Multimodal Reasoning (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Thanks for sharing this interesting work! I may be missing something, but I have a question about the transition from Eq. (4) to Eq. (5) and the derivation in Appendix B.1.
Since Eq. (5) minimizes over z ∈ Δ_K, the term −εΩ(z) seems to reward, rather than penalize, deviations from uniform weighting. In fact, for fixed θ:
min_{z ∈ Δ_K} [∑_{k=1}^K z_k J_k(θ) − εΩ(z)] = min_k J_k(θ) − ε(K−1).
This follows because ∑_k z_k J_k(θ) ≥ min_k J_k(θ) and Ω(z) ≤ K−1, with both bounds attained at a simplex vertex corresponding to a worst-performing task. Thus, ε appears to change only the objective value by a constant, rather than interpolate toward average-performance optimization.
Would the intended sign perhaps be +εΩ(z)? I also noticed that Appendix B.1 establishes ∑_k z_k = 1, but does not appear to establish z_k ≥ 0. In addition, the Lagrangian sign for a maximization problem with constraints J_k − J_j − ε ≤ 0 may need clarification.
Could you please clarify whether Eq. (5) is intended as a soft surrogate rather than a strictly equivalent reformulation of Eq. (4)?
Thanks again—I would be very interested to hear your interpretation.
Hi! Thanks for the question. The sign was indeed an error. I spotted it 3 weeks ago and updated the final version of the paper recently and also the poster we presented at the conference. But this doesn't affect the results as the update rule we designed did not directly stem from the equation, but ofc it was motivated from it. And yes, the simplex constraint indeed makes it a soft surrogate. I have clarified both in the updated version which I will upload online today. Kindly get back to us if you still have further questions after viewing the updated pdf.
Thank you so much for the interest and the question.
The pdf is accessible here till the arxiv is updated. Thanks again.
Thanks for the quick and thoughtful response, and for sharing the updated PDF.
Get this paper in your agent:
hf papers read 2602.05547 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper