Abstract
Multiscale Policy Optimization improves long-horizon agent training by aligning credit assignment and policy updates at the turn scale.
Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.
Community
Excited to share Multiscale Policy Optimization (MPO), a reinforcement learning framework designed for long-horizon LLM agents!
We observe that critic values vary primarily across assistant turns, motivating turn-level optimization with fine-grained token credit assignment. MPO combines Multiscale Advantage Estimation (MAE) with turn-level importance sampling to better align credit assignment and policy updates.
Experiments on search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent improvements over strong RL baselines.
Most of my bad rollouts are one rotten turn — a tool call with a malformed argument — while the rest of that turn is the model narrating correctly around it. Token-level credit smears that blame across every token in the step, so the policy gets punished for the parts it got right. What I'd want measured before trusting the gains: turn-level ratios when turn lengths are wildly unequal. A 40-token function call and a 6000-token file read in the same trajectory have completely different variance, and long-horizon agents live entirely in that regime. If clipping needs retuning per task to keep the long-turn ratio from exploding, that's a deployment cost, not a detail.
The more useful finding for me is the near-constant critic value within a turn — it means the critic is already a runtime signal, not just a gradient source. I'd take that value exposed at inference as a cheap per-turn anomaly score to trigger a retry or a human check over another RL recipe I can't afford to run. Does it stay calibrated as the policy drifts, or does it need the same on-policy refresh the training loop gives it?
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper