Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Abstract
TreeOPSD enables long-horizon reinforcement learning for language model agents without critics by using tree-structured self-distillation with bounded updates derived from terminal rewards.
Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing approaches obtain denser credit via learned critics, process reward models, or on-policy distillation, but they rely on a value network, step-level annotations, or a stronger external teacher that increase training complexity. Meanwhile, importance sampling and clipping mitigate policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each credit to a closed-form, bounded teacher distribution parameterized by the likelihood ratio relative to the frozen behavior policy. Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting.
Community
We are excited to share TreeOPSD (Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning).
Training long-horizon LLM agents with verifiable rewards faces two practical bottlenecks: terminal rewards alone make turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing remedies either add extra components (learned critics, process annotations, or stronger external teachers) or rely on importance-ratio clipping, which restricts step size without defining a finite target where updates should stop.
TreeOPSD addresses both challenges in a critic-free, self-distillation framework. Treating each interaction turn as a tree node, it branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each backed-up credit to a closed-form, bounded teacher distribution parameterized by the turn-level likelihood ratio relative to the frozen behavior policy. Minimizing the Forward KL divergence between this teacher and the student policy recovers the exact tree policy-gradient direction as its tangent at the behavior policy, while bounding updates toward a finite target across sequential mini-batches.
Across Multi-Hop QA and Deep Research benchmarks (from 1.7B to 30B-A3B models), TreeOPSD consistently outperforms GRPO, GSPO, and fixed-credit tree policy gradients under matched rollout budgets, improving Qwen3-30B-A3B-Thinking-2507 by +4.97 points on average in the Deep Research setting.
Get this paper in your agent:
hf papers read 2610.23181 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper