arxiv:2607.29209

SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

Published on Jul 31

· Submitted by

dingyi on Aug 3

Upvote

Authors:

Yifan Ding ,

Yoshua Y. Li ,

Abstract

Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.

View arXiv page View PDF Add to collection

Community

dingyii

Paper author Paper submitter about 3 hours ago

Hi everyone! We're excited to introduce our new work, Stable Advantage Fusion (SAF) — a lightweight advantage-fusion framework for jointly training reinforcement learning (RL) with on-policy distillation (OPD). Instead of naively combining GRPO and OPD with a fixed mixing coefficient, SAF separately regulates the magnitude and timing of the teacher signal, fully exploiting dense token-level guidance while preserving room for continued exploration driven by verifiable rewards.
🚀 Key Highlights:
✦ Diagnosing why 1+1<2: We find that fixed-weight fusion causes two types of mismatch: a small number of OPD tokens can have advantage values far larger than those from GRPO, dominating the update; meanwhile, sustained full-strength distillation pulls the student too close to the teacher, causing entropy collapse and a surge in response length early in training, while limiting further exploration later on.
✦ A four-stage stable fusion framework: SAF applies Top-k sparsification → tanh-bounded compression → KL-triggered warm-up → linear annealing solely to the OPD advantage values, jointly resolving both the token-level magnitude mismatch and the training-phase timing mismatch.
✦ Lightweight, modular, and easy to integrate: SAF requires no additional models, auxiliary losses, or extra forward passes. All four stages can be toggled independently, making it a plug-and-play replacement for existing GRPO+OPD pipelines.
💡 Additional finding: Staying closer to the teacher does not necessarily lead to better final performance. Training dynamics show that fixed fusion achieves the lowest student–teacher KL divergence, yet plateaus in accuracy earlier. By dynamically adjusting the trust placed in the teacher signal, SAF achieves a better balance between teacher guidance and autonomous exploration driven by verifiable rewards.