Experience-Handoff Reinforcement Learning for Recursive Self-Improvement
Abstract
Language-model agents trained with reinforcement learning can improve by learning to create and curate experience artifacts that help future rollouts, not just by optimizing actions from scalar rewards.
It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering () harness and train one model to act, reflect, and curate task-local playbooks. Our method, , evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.
Community
Why not train an agent's weights inside the harness it runs in?
Reinforcement learning (RL) is the standard way to train LLM agents, and it boils a whole attempt down to one reward number. Everything the agent saw along the way is thrown out: error logs, tool outputs, constraints it discovered.
Harnesses like ACE (Agentic Context Engineering) keep that information. The model reflects on each attempt and writes notes in a playbook, which its next attempt reads. But the model was never trained to write those notes, so a small model gains little from its own edits.
In our new paper, Experience-Handoff Reinforcement Learning for Recursive Self-Improvement, we combine the two. We keep the ACE harness fixed and train one model in three roles: the actor that does the task, the reflector that draws lessons from an attempt, and the curator that edits the playbook.
Our method, ACE-RL, scores each playbook edit by running the next attempt with it and checking the task reward. Each role is compared only against other outputs of the same role, which keeps its learning signal stable. We credit each rollout's contribution to the final task reward to get reward group for each component.
- We found that ACE-RL reaches the highest held-out reward of all methods we tested: PPO, GRPO, ERL, and end-to-end PPO through the harness.
- The gains last even after the cutoff that we give during the train. We trained with 3 attempts per task, and ACE-RL stays ahead of GRPO up to 8 attempts.
- Training only the actor isn't enough. A GRPO-trained model placed in the same harness after 8 attempts ends up below where it gets by simply sampling 8 times.
Our takeaway: agent training should improve the agent's actions, and also the notes it leaves for its future attempts.
Training one model to act, reflect and write the playbook, so the handed-off notes are judged by how much they help the next attempt, is a lovely idea. Error logs usually get thrown away. Did the learned playbooks end up looking different from hand-written ones?
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper