arxiv:2610.19376

DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents

Published on Oct 14

· Submitted by

Tianle Chen on Oct 22

Google

Upvote

Authors:

Tianle Chen ,

Abstract

DeLoop is a self-supervised preference optimization method that reduces repetitive tool-use loops in language-model agents by learning from their own rollouts, improving efficiency and success under context compression.

Generated by thinkingmachines/Inkling-Small

Language-model agents that run long tasks often redo work they have already done: they re-issue a tool call they made earlier and get back exactly the same observation, a step that adds no new information. We identify this failure as a loop. Loops appear natively in Qwen, Llama, and Gemini agents and become far more common once the agent's history is compressed to fit its context window. To reduce these loops, we propose DeLoop, a self-supervised training method. DeLoop builds on a simple observation: the agent's own rollout can provide the supervision needed to learn from its loops. We use a looping action as a negative example and the first later non-looping action in the same rollout as a positive example. DeLoop learns from these pairs through direct preference optimization and does not require task rewards, gold answers, or LLM judges. On CodeGym under compression, DeLoop reduces the share of Qwen3-8B tasks containing loops from 57.3% to 6.1%. It also uses about one third as many steps per task, with similar task success. On Qwen3-32B, DeLoop reduces loops and steps while raising task success from 37.7% to 44.8%.

View arXiv page View PDF Project page GitHub 0 Add to collection

Community

Tianle

Paper author Paper submitter about 12 hours ago

•

edited about 12 hours ago

TL DR: LLM agents often re-issue a tool call they already made and get the same observation back. We call this a loop. DeLoop turns each loop into a free training pair from the agent's own rollout: the repeated call is the rejected action, and the agent's first later loop-breaking action is the chosen one. We train on these pairs with DPO. No task reward, gold answer, or LLM judge needed.

Why it matters: loops get much worse when the history is compressed. On CodeGym, Qwen3-8B loops in 29.0% of runs with the full history and in 57.3% under summarization. Summaries that explicitly track past actions don't fix it, because the agent often ignores a record it already has.

Results

Qwen3-8B on CodeGym: loop rate 57.3% → 6.1% (9.4×), steps per task 15.0 → 4.7, tokens −47%, with similar task success
Qwen3-14B / 32B: task success +7.2 / +7.1 points (45.1 → 52.3, 37.7 → 44.8)
With cross-domain pairs (DeLoop-Mix): Qwen3-8B success 25.1 → 30.3. Loops also drop under 7 memory policies, in ALFWorld, Open-AgentRL, AppWorld, ScienceWorld, and BrowseComp-Plus (22.4% → 1.2%)

🔗 Project page (with an interactive example of a loop forming): https://cskyl.github.io/deloop/
📄 arXiv: https://arxiv.org/abs/2610.19376
💻 Code and data: coming soon at https://github.com/cskyl/deloop