Abstract
Real-world agent learning is often constrained by costly environment interactions, such as running time-consuming experiments or obtaining human feedback. In-context learning offers a highly sample-efficient way for agents to learn from their own interaction histories, but its gains disappear once that experience is removed from the context. Separately, context distillation provides a mechanism for internalizing contextual information into model weights. However, applying it to agents' interaction histories without sacrificing environment sample efficiency remains underexplored. We term this problem Experience Distillation and develop an implementation that requires no further environment interaction beyond the collected experience. Experiments on 749 curated software-engineering tasks and six text-adventure games show that it retains at least 64.8\% of the gains from in-context learning across both domains, whereas direct supervised fine-tuning on the collected experience recovers only 3.8\%. Compared with classical reinforcement-learning baselines, in-context learning from trial-and-error experience followed by Experience Distillation matches their performance with at least \(9.6\times\) fewer environment samples.
Community
Real-world agent learning is often constrained by costly environment interactions, such as running time-consuming experiments or obtaining human feedback. In-context learning offers a highly sample-efficient way for agents to learn from their own interaction histories, but its gains disappear once that experience is removed from the context. Separately, context distillation provides a mechanism for internalizing contextual information into model weights. However, applying it to agents' interaction histories without sacrificing environment sample efficiency remains underexplored. We term this problem Experience Distillation and develop an implementation that requires no further environment interaction beyond the collected experience. Experiments on 749 curated software-engineering tasks and six text-adventure games show that it retains at least 64.8% of the gains from in-context learning across both domains, whereas direct supervised fine-tuning on the collected experience recovers only 3.8%. Compared with classical reinforcement-learning baselines, in-context learning from trial-and-error experience followed by Experience Distillation matches their performance with at least (9.6\times) fewer environment samples.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- In-Context Reinforcement Learning under Non-Stationarity: A Survey (2026)
- EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learning (2026)
- EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments (2026)
- InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning (2026)
- Learning While Acting: A Skill-Enhanced Test-Time Co-Evolution Framework for Online Lifelong Learning Agents (2026)
- Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses (2026)
- Joint Learning of Experiential Rules and Policies for Large Language Model Agents (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
What's the practical overhead of running context distillation every N episodes vs. just keeping the last K trajectories in the prompt window? I've been running agents that accumulate interaction history and the context window fills up fast — long before the model's behavior stabilizes. The idea of distilling that experience into weights is appealing, but the paper's setup (teacher-student with replay buffers) adds infrastructure I'd want to see costed out. How many environment steps do you need before the distillation step pays for its own compute? If I'm running a 7B model on a single GPU, is this viable between customer-facing turns, or does it need a separate offline pass? The sample-efficiency numbers look good in the paper's setting, but I'd want to know how brittle the approach is to distribution shift — if the agent encounters a novel task variant mid-deployment, does the distilled policy adapt faster than a context-only baseline, or does it need to re-distill from scratch?
Thanks for the thoughtful question. The practical cost depends heavily on the information density of the environment interactions, so the break-even point may vary across tasks. Our goal is indeed for the distilled model to adapt faster than a context-only baseline after absorbing prior experience, and the paper includes results in this direction. In particular, we evaluate whether distilled experience improves behavior without carrying the original experience context at test time, which helps separate persistent adaptation from simply using a longer prompt.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper