Durable execution and replay are treated as first-class loop infrastructure. Multi-file refactoring tasks that require tracking and carrying state across many steps, isolating the durable-state weakness that breaks long agent loops.

Use RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code to measure progress and gate completion with repeatable evidence.

[2503.07832] RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code

Recent advances in language model (LM) agents and function calling have enabled autonomous, feedback-driven systems to solve problems across various digital domains. To better understand the unique limitations of LM agents, we introduce RefactorBench, a benchmark consisting of 100 large handcrafted multi-file refactoring tasks in popular open-source repositories. Solving tasks within RefactorBench requires thorough exploration of dependencies across multiple files and strong adherence to relevant instructions. Every task is defined by 3 natural language instructions of varying specificity and is mutually exclusive, allowing for the creation of longer combined tasks on the same repository. Baselines on RefactorBench reveal that current LM agents struggle with simple compositional tasks, solving only 22% of tasks with base instructions, in contrast to a human developer with short time constraints solving 87%. Through trajectory analysis, we identify various unique failure modes of LM agents, and further explore the failure mode of tracking past actions. By adapting a baseline agent to condition on representations of state, we achieve a 43.9% improvement in solving RefactorBench tasks. We further extend our state-aware approach to encompass entire digital environments and outline potential directions for future research. RefactorBench aims to support the study of LM agents by providing a set of real-world, multi-hop tasks within the realm of code.

Dhruv Gautam; Spandan Garg; Jinu Jang; Neel Sundaresan; Roshanak Zilouchian Moghaddam


Verification is promoted from a final check to a loop-control signal. Scores planning, verification coverage, recovery, abstention, and atomic transitions (not just whether code passes), measuring the loop discipline that separates reliable agents from reckless trial-and-error.

Use RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents to measure progress and gate completion with repeatable evidence.

Benchmark or leaderboard source with repeatable tasks or scores; compare systems only after checking setup and scope.

Gate progress with tests, evals, and evidence.

[2606.22678] RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents

Agentic coding harnesses - such as Agent-Skills, Superpowers, and Agent-Rigor - are increasingly deployed to augment underlying LLMs for real-world software engineering tasks. Existing benchmarks evaluate these agents almost exclusively on outcome correctness: whether generated code passes tests or resolves issues. We argue that this outcome-only lens is insufficient: an agent that arrives at a correct solution through reckless trial-and-error, without planning, verification, or graceful recovery, is fundamentally less reliable than one that follows sound engineering discipline. We introduce RigorBench, the first benchmark designed to measure process discipline in AI coding agents. RigorBench evaluates these harnesses across five pillars: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity. A composite RigorScore aggregates these dimensions into a single metric via a weighted sum. We curate a suite of 30 tasks spanning five categories - Plan-Then-Build, Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, and Don't Break the Build-and evaluate leading harnesses in a controlled with/without experimental design against baseline coding assistants. Our results show that structured process discipline not only improves process quality scores by an average of 41% but also raises downstream outcome correctness by 17%, providing the first quantitative evidence that how agents code matters as much as what they produce. We release the full benchmark, scoring rubrics, and trajectory analysis tools as open-source artifacts.

Meher Bhaskar Madiraju; Meher Sai Preetam Madiraju


Checkpointed state makes long-running agent work recoverable across failures. Quantifies structural erosion and verbosity creep across iteration checkpoints in native harnesses like Claude Code and Codex, evidence for why loops need verification and budgets.

Use SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks to measure progress and gate completion with repeatable evidence.

[2603.24755] SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.

Gabriel Orlanski; Devjeet Roy; Alexander Yun; Changho Shin; Alex Gu; Albert Ge; Dyah Adila; Nicholas Roberts; Frederic Sala; Aws Albarghouthi

Code and Leaderboards are located at https://www.scbench.ai


The work turns loop quality into a measurable task or score. Long-horizon CLI tasks where most runs stall below 30% completion, mapping where unattended loops break down.

Use LongCLI-Bench: A Preliminary Benchmark for Long-horizon Agentic Programming in Command-Line Interfaces to measure progress and gate completion with repeatable evidence.

[2602.14337] LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces

Recent advances in AI-assisted programming have empowered agents to execute complex workflows via command-line interfaces, however, existing benchmarks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics, fail to rigorously evaluate the long-horizon planning and execution capabilities essential for realistic software engineering. To address these gaps, we introduce LongCLI-Bench, a comprehensive benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks. We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring. We propose a dual-set testing protocol for LongCLI-Bench, which measures requirement fulfillment (fail-to-pass) and regression avoidance (pass-to-pass), and incorporates step-level scoring to pinpoint execution failures. Extensive experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench. Step-level analysis further indicates that the majority of tasks stall at less than 30% completion, highlighting that critical failures often occur in the early stages. Although self-correction offers marginal gains, human-agent collaboration through plan injection and interactive guidance yields significantly higher improvements. These results highlight that future research must emphasize the development of synergistic human-agent workflows alongside advances in agents' planning and execution capabilities to overcome key challenges in long-horizon task performance.

Yukang Feng; Jianwen Sun; Zelai Yang; Jiaxin Ai; Chuanhao Li; Zizhen Li; Fanrui Zhang; Kang He; Rui Ma; Jifan Lin; Jie Sun; Yang Xiao; Sizhuo Zhou; Wenxiao Wu; Yiming Liu; Pengfei Liu; Yu Qiao; Shenglin Zhang; Kaipeng Zhang

Published in Findings of the Association for Computational Linguistics: ACL; the linked arXiv record remains available for open access.


Evaluation data is used as the feedback signal for improving loop behavior. Benchmark of 2,458 instances across research and coding domains measuring how reliably LLM judges verify rubrics on agent outputs, finding substantial noise even in strong models and quantifying the trade-offs of prompt design, batched evaluation, and majority voting.

Use Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios? to measure progress and gate completion with repeatable evidence.

[2606.29920] Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric scoring remains underexplored. This concern is especially pronounced in agentic scenarios, where long, complex outputs further challenge reliable scoring. To address this, we conduct a systematic meta-evaluation of LaaJ reliability for rubric verification. We introduce RuVerBench, the first benchmark for assessing LaaJ reliability in rubric verification for agentic scenarios. RuVerBench covers two prevalent agentic domains, deep research and agentic coding, with 2,458 instances, each containing a model-generated output, a rubric, and a human-annotated label indicating whether the output satisfies the rubric. Using RuVerBench, we evaluate numerous frontier LLMs and find that even the most advanced models achieve strong performance but still exhibit substantial noise. We further analyze the impact of key LaaJ strategies, including prompt design, batching, and majority voting, on rubric verification. We find that weaker models are more sensitive to prompt variations, batched verification presents a trade-off between accuracy and efficiency, and majority voting yields effective but diminishing returns. We have released our dataset and code to facilitate future research: https://github.com/THU-KEG/RuVerBench.

Yangda Peng; Yunjia Qi; Hao Peng; Haotian Xia; Guanzhong He; Xintong Shi; Richeng Xuan; Songyuanyi Lu; Yixian Liu; Zhichao Hu; Yuhong Liu; Lei Hou; Bin Xu; Juanzi Li


The work turns loop quality into a measurable task or score. Microsoft Research benchmark of 100 tasks across 10 synthetic web environments that evaluates long-running monitoring agents on whether they wait or act appropriately, scoring task completion, response speed, and resource efficiency.

Use SentinelBench: A Benchmark for Long-Running Monitoring Agents to measure progress and gate completion with repeatable evidence.

[2606.05342] SentinelBench: A Benchmark for Long-Running Monitoring Agents

AI agents are increasingly asked to carry out work that spans minutes, hours, or longer. Yet the default model of agent behavior is continuous action: issuing tool calls, refreshing pages, searching for alternatives, or otherwise trying to force progress. This is the wrong approach for many long-running tasks, which are better served by a strategy of sustained attention. Instead, agents should monitor an environment, notice when an external event makes progress possible, then respond promptly without wasting resources while waiting. To measure progress on this class of tasks, we introduce SentinelBench, an open-source benchmark for time-evolving monitoring tasks. SentinelBench contains 100 tasks across 10 synthetic web environments, including email, calendars, finance, professional networking, and entertainment. Each environment exposes a live web interface and replays a scripted sequence of events, requiring agents to navigate and reason about web pages whose state shifts underfoot. SentinelBench measures task completion, reaction time, and resource use, exposing the tradeoff between responsiveness and cost. We report results across three models and two browser-agent harnesses, establishing performance baselines for future comparison and demonstrating how agent design choices can dramatically impact key metrics. Together, these results show that SentinelBench distinguishes meaningful differences in agent behavior.

Matheus Kunzler Maldaner; Adam Fourney; Amanda Swearngin; Hussein Mozannar; Gagan Bansal; Maya Murad; Rafah Hosn; Saleema Amershi


The work turns loop quality into a measurable task or score. Multi-session coding benchmark of 109 repository-level tasks reconstructed from 11,260 recorded user-agent sessions, replayed with an LLM user simulator and scored on final correctness and the number of corrective feedback turns.

Use SWE-Together: Evaluating Coding Agents in Interactive User Sessions to measure progress and gate completion with repeatable evidence.

[2606.29957] SWE-Together: Evaluating Coding Agents in Interactive User Sessions

Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.

Yifan Wu; Zhuokai Zhao; Songlin Li; Ho Hin Lee; Jiacheng Zhu; Shirley Wu; Tianhe Yu; Serena Li; Lizhu Zhang; Xiangjun Fan; Shengzhi Li


The work turns loop quality into a measurable task or score. Cross-domain diagnostic benchmark that scales task horizon through depth and breadth extension, then attributes failures across 3,100+ agent trajectories to a seven-category taxonomy via a trajectory-grounded LLM judge validated against human annotation.

Use The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break to measure progress and gate completion with repeatable evidence.

[2604.11978] The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break

Large language model (LLM) agents perform strongly on short- and mid-horizon tasks, but often break down on long-horizon tasks that require extended, interdependent action sequences. Despite rapid progress in agentic systems, these long-horizon failures remain poorly characterized, hindering principled diagnosis and comparison across domains. To address this gap, we introduce HORIZON, an initial cross-domain diagnostic benchmark for systematically constructing tasks and analyzing long-horizon failure behaviors in LLM-based agents. Using HORIZON, we evaluate state-of-the-art (SOTA) agents from multiple model families (GPT-5 variants and Claude models), collecting 3100+ trajectories across four representative agentic domains to study horizon-dependent degradation patterns. We further propose a trajectory-grounded LLM-as-a-Judge pipeline for scalable and reproducible failure attribution, and validate it with human annotation on trajectories, achieving strong agreement (inter-annotator \kappa=0.61; human-judge \kappa=0.84). Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures and offer practical guidance for building more reliable long-horizon agents. We release our project website at \href{https://xwang2775.github.io/horizon-leaderboard/}{HORIZON Leaderboard} and welcome contributions from the community.

Xinyu Jessica Wang; Haoyue Bai; Yiyou Sun; Haorui Wang; Shuibai Zhang; Wenjie Hu; Mya Schroder; Bilge Mutlu; Dawn Song; Robert D Nowak


The work targets tasks that exceed a single context window or prompt session. Reliability metrics for long-horizon agents (reliability decay, variance amplification, graceful degradation, meltdown onset) measured over 23,392 episodes across 10 models, showing capability and reliability rankings diverge as tasks lengthen.

Use Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2603.29231; inspect its method and evaluation before treating results as production evidence.

[2603.29231] Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents

Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to this divergence. We introduce a reliability science framework for long-horizon LLM agents with four metrics: Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP). We evaluate 10 models across 23,392 episodes on a 396-task benchmark spanning four duration buckets and three domains. Key findings: (1) reliability decay is domain-stratified -- SE GDS drops from 0.90 to 0.44 while document processing is nearly flat (0.74 to 0.71); (2) VAF bifurcates by capability tier -- high VAF is a capability signature, not an instability signal; (3) capability and reliability rankings diverge substantially, with multi-rank inversions at long horizons; (4) frontier models have the highest meltdown rates (up to 19%) because they attempt ambitious multi-step strategies that sometimes spiral; and (5) memory scaffolds universally hurt long-horizon performance across all 10 models. These results motivate reliability as a first-class evaluation dimension alongside capability.

Aaditya Khanal; Yangyang Tao; Junxiu Zhou


Evaluation data is used as the feedback signal for improving loop behavior. Evaluation environment that measures whether a self-evolving agent's modifications to prompts, memory, and tools generalize to held-out tasks, using train, validation, and test splits and cost metrics on Terminal-Bench 2.0 and HLE.

Use SEAGym: An Evaluation Environment for Self-Evolving LLM Agents to measure progress and gate completion with repeatable evidence.

[2606.17546] SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

Self-evolving LLM-based agents improve mainly by changing their agent harness: the structured execution layer around a base model, including prompts, memory, tools, middleware, runtime state, and the model-tool interaction loop. Existing evaluations often reduce this process to isolated task scores or a single sequential curve, obscuring whether an update produces reusable improvement, overfits recent tasks, increases cost, or harms older behavior. We introduce SEAGym, an evaluation environment for measuring agent harness updates across training, validation, test, replay, and cost records. SEAGym turns Harbor-compatible benchmarks into dynamic self-evolution task sources with train batches, frozen update-validation, held-out ID and OOD transfer views, replay diagnostics, and saved snapshot and metric records. Instantiating SEAGym on Terminal-Bench 2.0 and HLE, we compare ACE, TF-GRPO, and AHE under a shared epoch/batch protocol. The results show that these evaluation views provide complementary signals about the evolution process: frequent updates may fail to improve held-out performance, useful intermediate snapshots may collapse later, and source diversity and model backend can affect harness reliability.

Congjie Zheng; Chuanyi Xue; Bin Liang; Jun Yang; Changshui Zhang


Evaluation data is used as the feedback signal for improving loop behavior. Benchmark of 26 evolving coding tasks across 227 evaluation rounds using cumulative executable tests to check that agents keep prior requirements working as specifications change, with top agents reaching only about 50% on multi-turn success metrics.

Use EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions to measure progress and gate completion with repeatable evidence.

[2605.24110] EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions

Coding agents are increasingly used as iterative development partners, but most benchmarks still evaluate one specification followed by one final assessment. This leaves out a basic question: can an agent keep its own codebase working as requirements change? We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, states requirements through observable behavior, and uses cumulative executable tests to check new requirements and still-active prior ones. We evaluate 13 coding agents with two metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state. For most agents, SR exceeds MT@4 by 22-40 points. The gap also changes rankings: the highest-SR agent (78.9) ranks only third in persistent execution (44.0 MT@4). Even the strongest agents achieve only about 50% success on multi-turn metrics, and aggregate pass rate drops below half of round-1 performance by round 5. Failure analysis shows tier-dependent behavior: weaker agents fail early, while stronger agents survive long enough to expose specification-tracking and regression failures. We release the benchmark data and Harbor multi-turn infrastructure.

Haiyang Shen; Xuanzhong Chen; Wendong Xu; Yun Ma; Liang Chen; Kuan Li

Work in Progress; 32 pages, 10 figures, preprint


Links loop design to measurable tasks where progress and failure can be compared. Repeated-execution study on OSWorld decomposing why computer-use agents fail tasks they previously completed, separating execution stochasticity, task-specification ambiguity, and behavioral variability as distinct causes of unreliability.

Use On the Reliability of Computer Use Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2604.17849; inspect its method and evaluation before treating results as production evidence.

[2604.17849] On the Reliability of Computer Use Agents

Computer-use agents have rapidly improved on real-world tasks such as web navigation, desktop automation, and software interaction, in some cases surpassing human performance. Yet even when the task and model are unchanged, an agent that succeeds once may fail on a repeated execution of the same task. This raises a fundamental question: if an agent can succeed at a task once, what prevents it from doing so reliably? In this work, we study the sources of unreliability in computer-use agents through three factors: stochasticity during execution, ambiguity in task specification, and variability in agent behavior. We analyze these factors on OSWorld using repeated executions of the same task together with paired statistical tests that capture task-level changes across settings. Our analysis shows that reliability depends on both how tasks are specified and how agent behavior varies across executions. These findings suggest the need to evaluate agents under repeated execution, to allow agents to resolve task ambiguity through interaction, and to favor strategies that remain stable across runs.

Gonzalo Gonzalez-Pumariega; Saaket Agashe; Jiachen Yang; Ang Li; Xin Eric Wang


Evaluation data is used as the feedback signal for improving loop behavior. Grades 2,614 SWE-agent trajectories across eight models to show that 10.7% of passing trajectories in its 1,815-trajectory evaluation subset are lucky trial-and-error successes, replacing binary pass/fail with process-quality tiers that shift model rankings.

Use AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation to measure progress and gate completion with repeatable evidence.

Research source arXiv:2605.12925; inspect its method and evaluation before treating results as production evidence.

[2605.12925] AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false. We evaluate 2,614 OpenHands trajectories from eight model backends on 60 SWE-bench Verified tasks. Of these, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification. We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and define AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens builds PTA references by merging multiple passing solutions for the same task, and uses a context-sensitive intent labeler to assign actions to Exploration, Implementation, Verification, or Orchestration based on trajectory history rather than tool identity alone. On AgentLens-Bench, the quality score separates passing trajectories into Lucky, Solid, and Ideal tiers and further decomposes Lucky Passes into five recurring mechanisms. Across the eight model backends, Lucky rates range from 0.5% to 23.2%, and some models move by as many as five rank positions when ranked by quality score instead of pass rate. We plan to release the project repository soon, including AgentLens-Bench artifacts, the AgentLens SDK, and the analysis tooling.

Priyam Sahoo; Gaurav Mittal; Xiaomin Li; Shengjie Ma; Benjamin Steenhoek; Pingping Lin; Yu Hu


Verification is promoted from a final check to a loop-control signal. Formalizes infeasible-model debugging as a solver-in-the-loop process where each action triggers solver re-execution and infeasibility recomputation, giving deterministic verification for iterative repair in operations research.

Use ORLoopBench: Solver-in-the-Loop Benchmarks for Self-Correction to measure progress and gate completion with repeatable evidence.

[2601.21008] ORLoopBench: Solver-in-the-Loop Benchmarks for Self-Correction and Behavioral Rationality in Operations Research

Operations Research practitioners debug infeasible models through an iterative process: inspecting Irreducible Infeasible Subsystems ( IIS), identifying constraint conflicts, and repairing formulations until feasibility is restored. Existing LLM benchmarks mostly treat OR as one-shot translation from problem descriptions to solver code, omitting this diagnostic loop. We formalize infeasible-model repair as a solver-in-the-loop Markov Decision Process in which each action triggers solver re-execution and IIS recomputation, yielding deterministic, verifiable feedback. We introduce ORLoopBench, a benchmark suite with two components: OR-Debug-Bench releases 5,362 LP/MILP repair instances, while OR-Bias-Bench evaluates closed-form operational decision rationality across inventory settings. Solver-verified RLVR training enables an 8B model to surpass frontier APIs on LP repair (95.3% vs 92.4% RR @5), improves diagnostic behavior, and transfers to MILP repair. The same evaluation exposes semantic drift in whole-model code regeneration: feasible regenerated MILPs can solve the wrong problem. Process-level evaluation with solver oracles enables targeted training for reliable OR self-correction.

Ruicheng Ao; David Simchi-Levi; Xinshang Wang

Published in Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306; the linked arXiv record remains available for open access.


The work turns loop quality into a measurable task or score. Benchmark of 68 real-world data-analysis tasks built from Kaggle notebooks spanning 2,225 interactive turns, finding that long-horizon errors account for 52-69% of agent failures and that maintaining a correct analytical state is the core bottleneck.

Use LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis to measure progress and gate completion with repeatable evidence.

[2605.30434] LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

Real-world data analysis is inherently iterative, yet existing benchmarks mostly evaluate isolated or short interactive tasks, leaving agents' ability to track evolving analytical context over long horizons untested. We introduce LongDS, a benchmark for long-horizon, multi-turn data analysis where agents must maintain, update, restore, and compose evolving analytical states. LongDS comprises 68 tasks constructed from real-world Kaggle notebooks, spanning 2,225 turns across six domains including Geoscience, Business, and Education. Tasks are designed around state-evolution patterns (e.g., counterfactual perturbation, rollback, multi-state composition), with an average dependency span of 11.3 turns. Evaluating five state-of-the-art models, we find that the best model reaches only 48.45% average accuracy, performance drops nearly 47 points from early to late turns, and long-horizon errors account for 52%--69% of failures. Further analysis shows that additional agent steps do not necessarily improve performance, suggesting that the key bottleneck is maintaining a correct analytical state rather than increasing interaction budget. We release LongDS to support research on reliable long-horizon agentic data analysis. Code and data will be released at https://github.com/zjunlp/DataMind.

Kewei Xu; Xiaoben Lu; Shuofei Qiao; Zihan Ding; Haoming Xu; Lei Liang; Ningyu Zhang


The work turns loop quality into a measurable task or score. Multi-session benchmark of interdependent agentic tasks where agents must distill earlier sessions into memory and use it to guide later actions, showing that near-saturated scores on long-context memory benchmarks fail to transfer.

Use MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks to measure progress and gate completion with repeatable evidence.

[2602.16313] MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

Existing evaluations of agents with memory typically assess memorization and action in isolation. One class of benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how memory is used to guide future decisions. Another class focuses on agents acting in single-session tasks without the need for long-term memory. However, in realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks. To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops. The benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where agents must learn from earlier actions and feedback by distilling experiences into memory, and subsequently use that memory to guide later actions to solve the overall task. MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory.

Zexue He; Yu Wang; Churan Zhi; Yuanzhe Hu; Tzu-Ping Chen; Lang Yin; Ze Chen; Tong Arthur Wu; Siru Ouyang; Zihan Wang; Jiaxin Pei; Julian McAuley; Yejin Choi; Alex Pentland


The work turns loop quality into a measurable task or score. Benchmark for persistent, tool-mediated task completion across multiple sessions, finding that agents fail by treating prior-session history as current context instead of stale state that needs re-validation.

Use Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversations to measure progress and gate completion with repeatable evidence.

[2606.00832] Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversations

Recent advances in agentic AI have enabled agents to complete complex tasks through tool use, reasoning, and multi-step planning. Yet existing benchmarks evaluate agents within a single session, ignoring past actions, stated preferences, and prior decisions that agents must integrate to fulfill personalized user goals. We introduce Momento, a benchmark for persistent agentic task completion in multi-session service environments, requiring agents to take consequential, tool-mediated actions while resolving temporal dependencies and evolving user goals across sessions. Experimental results reveal that current agents fail primarily through misestimation of user state, treating prior session history as a reliable proxy for current context rather than stale information requiring re-validation, highlighting a substantial gap between current agent capabilities and realistic long-horizon human-agent interaction.

Adril Putra Merin; David Anugraha; Ayu Purwarianti; Genta Indra Winata


The work turns loop quality into a measurable task or score. Benchmark of 100 multi-turn tasks across 5 user personas with hidden intents, inter-task dependencies, and cross-session continuity, measuring agent proactivity separately from task completion in long-horizon trajectories.

Use π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows to measure progress and gate completion with repeatable evidence.

[2605.14678] $π$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce $\pi$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, $\pi$-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.

Haoran Zhang; Luxin Xu; Zhilin Wang; Runquan Gui; Shunkai Zhang; Haodi Lei; Zihao He; Bingsu He; Chicheng Qin; Tong Zhu; Xiaoye Qu; Yang Yang; Yu Cheng; Yafu Li


The work targets tasks that exceed a single context window or prompt session. A 132-month CFO simulation where agents repeat a monthly cycle of liquidity management, financial closings, and financing decisions with compounding state, and only 15.4% of trials survive the full horizon.

Use Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation to measure progress and gate completion with repeatable evidence.

[2603.23638] Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation in an Uncertain Enterprise Environment

Large language model (LLM) agents are increasingly tested on complex tasks, but their ability to allocate scarce resources over long horizons remains unclear. Unlike reactive tasks with immediate feedback, this setting requires agents to make binding commitments under partial observability, delayed consequences, hard resource budgets, and shifting dynamics. We introduce EnterpriseArena, a 132-month CFO simulator that evaluates long-horizon resource allocation under uncertainty in a FinTech lending firm. Agents must manage liquidity, close books, gather costly signals, and request equity or debt financing across changing macroeconomic regimes. The simulator is built from transformed firm-level financial data, anonymized business documents, decade-scale macroeconomic and industry signals, and expert-validated operating rules. Experiments across 23 LLMs and four agent frameworks show that current agents remain far from robust: only 15.4% of trials survive the full horizon, larger models do not reliably outperform smaller ones, and failures cascade across observation, action timing, and capital sizing. These findings establish long-horizon resource allocation under uncertainty as a distinct capability gap for LLM agents.

Yi Han; Yan Wang; Lingfei Qian; Haohang Li; Yupeng Cao; Yueru He; Xueqing Peng; Nanhan Shen; Yitao Xu; Yankai Chen; Dongji Feng; Jimin Huang; Xue Liu; Jian-Yun Nie; Sophia Ananiadou


Control flow is represented as an inspectable graph rather than an opaque prompt loop. Benchmark isolating whether agents transfer reusable procedures such as searching, debugging, and verification across episodes in four long-horizon domains linked by ability graphs, finding curated experience transfers but no automatic method yields consistent gains.

Use EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer to measure progress and gate completion with repeatable evidence.

[2607.05202] EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer

Agent self-evolution in long-horizon LLM systems is largely procedural: useful experience is not merely stored information, but reusable procedures for searching, debugging, and verification. Yet current evaluations do not isolate this form of transfer. Agent benchmarks test single-episode task solving; memory benchmarks target information retention rather than procedural reuse. We introduce EvoAgentBench, a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work. EvoAgentBench extracts trace-grounded Abilities from agent executions, canonicalizes them into operational units, and builds domain-specific Ability Graphs linking tasks that share procedural overlap. By design, every test task is backed by verified training-side Ability support. Across a 528/267 train/test split, two scaffolds, and three backbones, curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. EvoAgentBench shifts self-evolution evaluation from aggregate accuracy comparison to fine-grained diagnosis of experience encoding, routing, and uptake. The benchmark is publicly available at https://huggingface.co/datasets/EverMind-AI/EvoAgentBench.

Xingze Gao; Chuanrui Hu; Hongda Chen; Pengfei Yao; Zhao Wang; Yi Bai; Zhengwei Wu; Yunyun Han; Xiaofeng Cong; Jie Gui; Yafeng Deng; Teng Li


Persistent memory is treated as an external runtime artifact. Bounded-memory testbed built on Slay the Spire 2 where every agent decision is made from a fresh prompt assembled by typed retrieval over recorded state, keeping prompt size bounded across runs of any length, with 298 documented trajectories released.

Use AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents to measure progress and gate completion with repeatable evidence.

[2607.02255] AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see. The simplest contract appends past observations, tool calls, and reflections to every prompt, which makes prior context easy to access but also turns it into a jumbled mixture in which the effect of any single memory component is hard to isolate. We introduce and instrument an alternative bounded contract: every decision is made from a fresh user message assembled by typed retrieval, with no raw cross-decision transcript appended. The prompt thus stays bounded across runs of any length, and any single layer can be ablated in isolation. We instantiate the contract in Slay the Spire 2, a closed-rule stochastic deck-building game whose runs require hundreds of tactical and strategic decisions. A public online benchmark of frontier LLMs on the same game reports zero wins at the lowest difficulty across five configurations, and the developer-reported human win rate at the same difficulty is 16%; the task is hard but not saturated. Within our harness, a fixed-A0 ablation shows the largest observed difference when triggered strategic skills are enabled: the no-store baseline wins 3/10 games and adding the skill layer 6/10. At this sample size the comparison is directional rather than statistically decisive (Fisher exact p\approx0.37); a cross-backbone probe and public accumulating-context baselines are reported as operational comparisons rather than controlled tests of the contract variable itself. We release a reproducible testbed: 298 completed trajectories with condition tags, frozen memory/skill snapshots, prompt records, and analysis scripts -- an agent design and a validated, reusable methodology for studying how explicit memory layers shape long-horizon LLM-agent decisions.

Xiangchen Cheng; Yunwei Jiang; Jianwen Sun; Zizhen Li; Chuanhao Li; Xiangcheng Cao; Yihao Liu; Fanrui Zhang; Li Jin; Kaipeng Zhang


Evaluation data is used as the feedback signal for improving loop behavior. Empirical evaluation of iteration budgets for generate-validate-repair loops across code generation, test generation, and translation, finding the first three to four iterations capture most gains and that orchestration and feedback design matter more than the model.

Use Is Three the Magic Number? An Empirical Evaluation of LLM-Based Repair Loops to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.05197; inspect its method and evaluation before treating results as production evidence.

[2607.05197] Is Three the Magic Number? An Empirical Evaluation of LLM-Based Repair Loops

Iterative repair loops have become a core design pattern in LLM-based software engineering systems. These workflows repeatedly generate, validate, and repair artifacts using feedback such as compiler errors or test failures. Despite their widespread use, the impact of repair-loop iteration limits remains poorly understood, as most prior work adopts fixed, often arbitrary, repair budgets. We study repair-loop effectiveness across multiple software engineering tasks, including code generation, test generation, and code translation. Across several representative workflows, datasets, and contemporary low-cost LLMs, we observe a consistent pattern of diminishing returns: the first three to four repair iterations account for most achievable gains, while later iterations contribute only marginal improvements. We further find that repair behavior is influenced more strongly by workflow orchestration and feedback design than by the underlying model itself. These results suggest that repair budgets should be treated as an explicit experimental variable, as they directly affect evaluation outcomes, computational cost, runtime, and reproducibility in LLM-based software engineering research.

Tobias Kiecker; Eik Reichmann; Hosung Kang; Gabin An; Lars Grunske


The work targets tasks that exceed a single context window or prompt session. 113 from-scratch, contamination-free long-horizon software-engineering tasks with custom verifiers that accept any correct implementation, built to sidestep SWE-bench-style pretraining recall and better differentiate frontier coding agents.

Use DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks to measure progress and gate completion with repeatable evidence.

[2607.07946] DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.

Wenqi Huang; Charley Lee; Leonard Tng; Serena Ge

32 pages, 10 figures. Code and data: https://github.com/datacurve-ai/deep-swe ; https://deepswe.datacurve.ai/


Verification is promoted from a final check to a loop-control signal. Benchmarks the profile-diagnose-edit-verify loop where the verifier is a profiler rather than a test suite: agents must deliver measured, reproducible speedups without breaking correctness, and across seven agent configurations the framework choice shifts results even with identical models.

Use PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization to measure progress and gate completion with repeatable evidence.

[2607.07744] PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization

Coding-agent benchmarks have largely measured whether agents can produce functionally correct patches, but production software also demands measurable speedups on real execution targets. Performance optimization is a distinct agentic task: agents must profile executions, diagnose cross-layer bottlenecks, edit code without breaking correctness, and verify that gains are reproducible rather than measurement artifacts. We introduce PERFOPT-Bench, a benchmark for evaluating this full performance-engineering loop. Each task provides a correct but deliberately suboptimal codebase and asks the agent to improve a target performance metric; scoring requires hidden correctness tests, verified-speedup measurement, and trajectory-level audit. We evaluate 7 agent stacks with different LLMs and agent frameworks on 7 long-horizon optimization tasks. The results show that optimization performance is workload-dependent rather than determined by model identity alone: no single stack dominates, and changing the agent framework can materially change the same LLM's per-task speedup profile. We further find that raw speedup is unsafe as a benchmark score, since some large gains arise from benchmark-specific shortcut exploitation; an exploratory relay pilot suggests that restarting from an externalized optimization summary can recover additional headroom after an initial session stops. The benchmark and our evaluation are available at: https://anonymous.4open.science/r/Dataset-D3CC.

Yingyun Cui; Yi Xie; Piaohong Wang; Jiawei Ma; Bo Liu; Liangliang Cao


Verification is promoted from a final check to a loop-control signal. Databricks engineering post (July 8, 2026, authors including Matei Zaharia and Patrick Wendell) on an internal benchmark built from real merged PRs with test-suite verification, finding that models cluster into three capability tiers, token price is a poor proxy for end-to-end task cost, and harness choice matters, with their Pi harness sending about 3x less context per turn at equal quality.

Use Benchmarking coding agents on Databricks' multi-million line codebase to measure progress and gate completion with repeatable evidence.

Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase | Databricks Blog

Databricks shares results from its internal coding benchmark, evaluating coding agents on a multi-million line codebase to optimize engineering cost and performance.


Checkpointed state makes long-running agent work recoverable across failures. Capability-driven benchmark of 400 bilingual tasks for proactive agents operating everyday tools in live Docker environments with step-level checkpoints, decomposed into five foundational capabilities, skill usage, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination, so failures localize to a root-cause capability instead of mixing capabilities per task, with closed-loop evaluation using multiple agent roles to simulate human feedback without leaking grading criteria.

Use UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks to measure progress and gate completion with repeatable evidence.

[2607.08768] UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.

Zhekai Chen; Chengqi Duan; Kaiyue Sun; Bohao Li; Yuqing Wang; Manyuan Zhang; Xihui Liu

Project Page: https://uniclawbench.github.io | GitHub Repo: https://github.com/HKU-MMLab/UniClawBench


The work turns loop quality into a measurable task or score. Benchmark for whether agent loops respect the logical relations inside skill files (preconditions, constraints, fallbacks): 70% of 5,000+ public skills contain at least one such relation, and on 86 executable cases leading coding agents show unsafe-behavior rates up to 70%, with an inference-time scaffold cutting violations by 63%.

Use SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills to measure progress and gate completion with repeatable evidence.

[2607.09016] SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills

Agent skills extend LLM agents with reusable procedures, tools, and domain-specific workflows, but their safety depends on resolving dependencies among interacting instructions. We introduce SkillLogic, a framework for analyzing logical relations in skill files and constructing executable tests from them. Our taxonomy covers eight relation types, including preconditions that gate valid actions, constraints that limit how allowed actions may be performed, and fallbacks that specify recovery behavior after failure. Using SkillLogic, we scan over 5000 public skills and find that 70% contain at least one logical relation. We then construct SLBench, an 86-case executable benchmark from high-confidence, high-impact, and locally testable relations. Evaluating Codex and Claude Code across six LLM backbones shows unsafe rates up to 70%, with violations leading to privacy leaks, unsafe configuration changes, and incomplete cleanup. The human audit attributes failures to both agent capability gaps and low-salience skill text. We further show that SLGuard, a lightweight inference-time scaffold, reduces violations by 63% on targeted cases. Our results establish logical-relation following as a distinct reliability challenge for skill-guided agents.

Xuan Chen; Chengpeng Wang; Lu Yan; Xiangyu Zhang


Links loop design to measurable tasks where progress and failure can be compared. Commit-history-derived milestone task streams where agents must preserve system integrity across successive runs - frontier-model scores collapse from >80% on isolated tasks to at most 38% in continuous settings, quantifying the error-accumulation gap loop engineering targets.

Use SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution to measure progress and gate completion with repeatable evidence.

[2603.13428] SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution

Real-world software must continuously evolve to meet ever-changing and open-ended requirements. AI agents, increasingly deployed as long-running systems, are now entrusted to drive this evolution. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as functionally cohesive development goals. These executable sequences enable SWE-Milestone, a benchmark that evaluates agents on streams of milestone-level tasks, requiring them to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to 38.03% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation.

Gangda Deng; Zhaoling Chen; Zhongming Yu; Haoyang Fan; Yuhong Liu; Yuxin Yang; Dhruv Parikh; Rajgopal Kannan; Le Cong; Mengdi Wang; Qian Zhang; Viktor Prasanna; Xiangru Tang; Xingyao Wang

Accepted at Proceedings of the 43rd International Conference on Machine Learning (ICML); the linked arXiv record is the available paper version.

Current arXiv acceptance note and official project record


The work turns loop quality into a measurable task or score. Benchmark measuring whether agents correctly abstain from acting when a task is underspecified, unsafe, or impossible, rather than proceeding anyway, a capability every unattended loop depends on for safe exits.

Use AgentAbstain: Do LLM Agents Know When Not to Act? to measure progress and gate completion with repeatable evidence.

[2607.10059] AgentAbstain: Do LLM Agents Know When Not to Act?

Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain. This gap poses real risks: under ambiguity, conflicting constraints, or tool failures, agents may execute unintended and irreversible actions. To close this gap, we present the first systematic evaluation framework for agentic abstention: the calibrated ability of tool-using LLM agents to recognize when not to act. At its core, AgentAbstain is a paired-task benchmark built on an agent-native taxonomy of 8 abstention scenarios across pre-execution reasoning and runtime discovery. It contains 263 paired tasks across 42 executable sandbox environments, where each pair consists of a should-act task and a should-abstain variant produced through a controlled perturbation to the instruction, tool, or environment state. To scale this paired design and resist data contamination, we propose AbstainGen, a fully automated pipeline that synthesizes sandbox environments and generates paired tasks end-to-end, validated by deterministic replay and semantic LLM judges; fresh task instances can be regenerated on demand, and three independent annotators rate 94-98% of sampled tasks as well-designed. Across 17 frontier LLMs in 4 agent harnesses, the best agent (Gemini 3.1 Pro) achieves only 59.5% paired accuracy (correct on both the act and abstain sides of each paired task). More importantly, abstention capability is largely independent of general task-solving capability, indicating that scaling task-solving alone will not close this gap. We further identify failure modes such as post-hoc abstention, in which agents execute irreversible actions before recognizing abstention triggers. Our code and dataset are open-sourced at agentabstain.github.io.

Xun Liu; Yi Evie Zhang; Vira Kasprova; Parisa Rabbani; Pardis Sadat Zahraei; Tianyu Zhang; Ali Ebrahimpour-Boroojeny; Varun Chandrasekaran


The work turns loop quality into a measurable task or score. Benchmark showing that once a stateful personal agent is nudged into a sycophantic stance, it persists across later sessions through memory, so single-turn sycophancy tests understate the risk in long-running agents.

Use Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy to measure progress and gate completion with repeatable evidence.

[2607.10526] Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy in Stateful Personal Agents

Stateful personal agents increasingly maintain long-term user profiles, episodic memories, and reusable skills. This persistence turns conversational sycophancy into a state-writing failure: accepted user-centric claims can be committed as lasting preferences, background facts, or workflows and later reused after the original conversation is gone. We call this persistent sycophancy and introduce the Personal Agent Sycophancy Benchmark (PASB), a 1,600-task benchmark that traces whether a conversational claim is accepted, written into durable agent state, and reused in a later neutral query. Unlike prior benchmarks that provide pre-written memories, PASB evaluates real agents (Hermes-Agent and OpenClaw) that decide what to store. It isolates the write process by combining four scenario framings with four temporal delivery patterns and separating a five-turn persist stage from a cleared three-turn query stage, ensuring downstream effects arise only from durable state. Across twelve models, the commit boundary is the key inflection point: downstream failure increases from 45.0% in session-only episodes to 71.9% after commitment, a consistent increase of 27.0 percentage points. Committed claims exhibit three write-time patterns: status promotion, attribution removal, and scope broadening. These patterns become stronger under memory-like or procedural framing, repeated reinforcement, and even across domain boundaries. These results show that agent sycophancy is fundamentally a state-writing governance problem. Once user content is committed to durable memory, safety must govern what agents write, not only what they say. PASB identifies the write-time controls needed to gate risky commits while preserving the source, role, and scope of stored content beyond response-level mitigations.

Xutao Mao; Liangjie Zhao; Leyao Wang; Rui Qian; Qiang Huang; Wentao Wang; Bo Han; Xiang Zheng; Cong Wang


Links loop design to measurable tasks where progress and failure can be compared. Tests whether harnessed agents adapt when hidden tool reliability changes, paired with no-shift controls that separate genuine adaptation failures from ordinary task errors and expose routine lock-in.

Use Set-shifting Behavioral Test for Harnessed Agents to measure progress and gate completion with repeatable evidence.

[2607.13396] Set-shifting Behavioral Test for Harnessed Agents

What happens to an LLM agent's tool choice when the reliable tool silently changes within an ongoing session? We borrow set-shifting from cognitive psychology to study how well agents adapt to hidden reliability shifts. Our benchmark mounts tool-skill libraries with redundancies, where many tools solve the same task but differ in hidden reliability. In our evaluation framework, a branched schedule shifts the reliable tool group at hidden boundaries and pairs every shift with a no-shift control. We find that agents, by default, settle on a small recurring routine within a few turns of each boundary, with call shares concentrating on a few discrete values after each reliability shift. We score the set-shifting accuracy for each agent trajectory: the joint probability of routing to the target tool group in every post-shift window. We test open-weight LLMs in an open-source agentic harness and find qualitatively distinct failure modes across the same set of routines. We also find that set framing, how the toolset presents the alternatives as competing or complementary, shifts the routing dynamics.


Links loop design to measurable tasks where progress and failure can be compared. Applies 11 server-evolution mutations to 123 MCP servers and evaluates 12 LLMs; reported performance drops include 13.7% for GPT-5.4 and 14.4% for Claude Sonnet 4.6, quantifying tool-interface drift.

Use MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers to measure progress and gate completion with repeatable evidence.

[2607.14642] MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers

As Model Context Protocol (MCP) servers emerge as the core infrastructure for connecting LLMs with external tools, existing benchmarks leverage real-world MCP servers to evaluate LLM agents' tool-using capabilities. However, these benchmarks overlook the continuous evolution of tool interfaces and functionalities within MCP servers, resulting in flawed assessments that fail to capture the agent's adaptability in changing tool landscapes. To bridge this gap, we introduce \textbf{MCPEvol-Bench}, a novel benchmark for evaluating the task-solving capabilities of LLM agents under dynamic toolset evolution. Inspired by large-scale empirical study, we propose 11 mutation operators to simulate realistic tool evolution within 123 MCP servers. We benchmark 12 state-of-the-art LLMs on multiple versions of MCP servers, revealing that even frontier models struggle to adapt to evolving tools. For instance, GPT-5.4 and Claude-Sonnet-4-6 exhibit performance declines of 13.7\% and 14.4\% in evolved MCP servers, respectively, accompanied by substantial increases in planning and reasoning errors. These findings highlight the vulnerability of LLM-driven workflows, establishing MCPEvol-Bench as a standard for evaluating agent adaptability in dynamic tool environments.

Huanxi Liu; Kun Hu; Jiaqi Liao; Qiang Wang; Pengfei Qian; YuanZhao Zhai; Dawei Feng; Bo Ding; Huaimin Wang


The work turns loop quality into a measurable task or score. Provides 652 issue-PR instances across 23 languages, seven image categories, and four relevance levels; the strongest evaluated agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, leaving substantial room for visual-evidence-aware repair loops.

Use MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization to measure progress and gate completion with repeatable evidence.

[2607.15205] MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.

Shaoxiong Zhan; Shi Hu; Boyu Feng; Hai Lin; Andrew Gong; Zhengda Zhou; Jiaying Zhou; Yunyun Hou; Hao Su; Hai-Tao Zheng


Control flow is represented as an inspectable graph rather than an opaque prompt loop. Builds executable long-horizon training environments from nearly 400 MCP servers and about 4,500 tools, generates tasks from tool-dependency graphs, and introduces turn-aware credit assignment for agentic reinforcement learning in large, dynamic tool spaces.

Use ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning to measure progress and gate completion with repeatable evidence.

[2607.15660] ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning

While LLM agents demonstrate strong reasoning abilities in compact and well-defined scenarios, they struggle to maintain robustness and effectiveness when faced with large-scale, diverse, and dynamic real-world environments that demand seamless tool integration. To address this gap, we introduce ToolVerse, a comprehensive framework that scales up agentic RL environments and enables agents to perform complex long-horizon reasoning in Tool-Integrated Reasoning (TIR) tasks. First, ToolVerse automatically builds the massive executable agent training environments from nearly 400 real-world Model Context Protocols (MCPs) that contain about 4500 tools. Second, we propose a task design strategy based on a tool dependency graph, utilizing Dynamic Unlocking Sampling Algorithm to generate long-horizon tasks, and produce GUST (Graph Unlocking Sampling Tasks) dataset. Third, to alleviate the credit assigment problem in long-horizon agentic RL, we propose a fine-grained Turn-Aware Relative Advantage algorithm. We conduct extensive Agentic RL training using ToolVerse and evaluate our framework on serveral agentic benchmarks. Experimental results demonstrate that our framework significantly strengthens LLMs' capabilities in long-horizon tool use, achieving a marked performance boost and showcasing robust reasoning within dynamic environments.

Shuaiyu Zhou; Fengpeng Yue; Zengjie Hu; Yuanzhe Shen; Chenyang Zhang; feng hong; Cao Liu; Ke Zeng


Verification is promoted from a final check to a loop-control signal. Evaluates long-horizon mobile work that requires clarification and MCP tool use in reproducible self-hosted environments with deterministic backend, storage, and callback verification.

Use MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interaction and MCP-Augmented Tasks to measure progress and gate completion with repeatable evidence.

MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments - ACL Anthology

Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, Zhidan Liu, Steven Hoi, Yue Wang. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Quyu Kong; Xu Zhang; Zhenyu Yang; Nolan Gao; Chen Liu; Panrong Tong; Chenglin Cai; Hanzhang Zhou; Jianan Zhang; Liangyu Chen; Zhidan Liu; Steven Hoi; Yue Wang


Context is managed as durable loop state rather than a single prompt payload. Provides 138 high-fidelity tasks across six capabilities whose average rollout exceeds one million tokens and 90 tool calls, exposing context-retention and long-horizon execution limits.

Use AGENCYBENCH: Benchmarking the Frontiers of Autonomous Agents to measure progress and gate completion with repeatable evidence.

AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts - ACL Anthology

Keyu Li, Junhao Shi, Yang Xiao, Mohan Jiang, Jie Sun, Yunze Wu, Dayuan Fu, Shijie Xia, Xiaojie Cai, Tianze Xu, Weiye Si, Wenjie Li, Dequan Wang, Pengfei Liu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Keyu Li; Junhao Shi; Yang Xiao; Mohan Jiang; Jie Sun; Yunze Wu; Dayuan Fu; Shijie Xia; Xiaojie Cai; Tianze Xu; Weiye Si; Wenjie Li; Dequan Wang; Pengfei Liu


The work targets tasks that exceed a single context window or prompt session. 46 long-horizon terminal tasks across nine categories (experiment reproduction, software engineering, scientific computing, and more) graded with dense intermediate rewards and partial credit instead of binary pass/fail; runs average 231 episodes and 85 minutes, and the strongest of 15 frontier models reaches only 15.2% pass@1 at a 0.95 reward threshold and 10.9% at perfect completion.

Use Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading to measure progress and gate completion with repeatable evidence.

[2607.08964] Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.

Zongxia Li; Zhongzhi Li; Yucheng Shi; Ruhan Wang; Junyao Yang; Zhichao Liu; Xiyang Wu; Anhao Li; Yue Yu; Ninghao Liu; Lichao Sun; Haotao Mi; Leowei Liang


Verification is promoted from a final check to a loop-control signal. 67 long-horizon workplace-workflow tasks across commerce, knowledge work, legal, localization, and manufacturing, graded by a hybrid of structural checks, executable verification, and LLM judging. Shows state-of-the-art agents degrade sharply when workflows mix languages versus monolingual runs, a multilingual coverage axis long-horizon benchmarks otherwise ignore.

Use PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents to measure progress and gate completion with repeatable evidence.

[2607.06008] PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents

Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external environments. However, most existing benchmarks implicitly assume a monolingual setting, where the entire execution process, including reasoning, tool invocation, and output generation, is conducted within a single language. In contrast, real-world applications often involve multilingual inputs and outputs within a unified workflow, yet the interaction between multilinguality and agentic execution remains underexplored. In this work, we introduce PolyWorkBench, a benchmark for evaluating LLM agents on multilingual long-horizon workplace workflows. PolyWorkBench consists of 67 tasks across five domains, including commerce, knowledge work, legal analysis, localization, and manufacturing, where agents must process heterogeneous multilingual inputs, perform iterative reasoning, invoke external tools, and produce structured outputs. To enable comprehensive evaluation, we propose a hybrid framework that combines structural grading, executable verification, and LLM-based semantic assessment. This design allows us to capture both functional correctness and linguistic consistency across complex workflows. Empirical results show that state-of-the-art LLM agents suffer significant performance degradation in multilingual workflow settings compared to monolingual counterparts. Our analysis suggests that multilinguality introduces compounding effects across reasoning and execution steps, highlighting the importance of jointly modeling language variation and procedural decision-making in agent evaluation.

Hongliang Li; Yijin Liu; Zhiwei Zhang; Zihe Liu; Xinyue Lou; Jinan Xu; Fandong Meng; Kaiyu Huang


Evaluation data is used as the feedback signal for improving loop behavior. Evaluates agent memory by the operations that maintain it, remembering, forgetting, updating, reflecting, with structured operation traces rather than final-answer accuracy, exposing lifecycle failure modes that answer-level evals miss. Finds current memory systems remain far from uniformly reliable.

Use MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations to measure progress and gate completion with repeatable evidence.

[2607.12893] MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations

Long-term memory has become a foundational capability for LLM-based agents that accompany users across extended, multi-session interactions. Existing benchmarks, however, evaluate such memory almost exclusively through downstream question answering, scoring only the correctness of a final answer. This black-box formulation conflates the heterogeneous causes of memory failure, such as missing the introduction of a relevant fact, binding an operation to the wrong target, or relying on stale values after a correction. As a result, it can credit correct answers despite their reliance on inconsistent or unsafe memory states. In this paper, we argue that, in dynamic long-horizon interactions, memory is not a static collection of facts but a lifecycle of explicit operations, including remembering, forgetting, updating, reflecting, and their compositions. We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations and represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence. A controllable generation pipeline embeds these operations into long, task-oriented conversations and produces gold operation traces together with six categories of operation-level probes, evaluated under both adjacent-evidence and long-context settings. Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals, revealing that current systems remain far from uniformly reliable. For instance, session-level retrieval outperforms turn-level retrieval, and long-context models remain notably weak at reconstructing ordered memory-state trajectories. These results move long-term memory evaluation from final-answer scoring toward interpretable, operation-level diagnosis.

Xixuan Hao; Zeyu Zhang; Zehao Lin; Yihang Sun; Ziliang Guo; Xichong Zhang; Yuxuan Liang; Feiyu Xiong; Zhiyu Li


Evaluation data is used as the feedback signal for improving loop behavior. Open-source evaluation infrastructure that factors agent evaluation into three independent components, Benchmark, Harness, and Environment, with a fault-tolerant asynchronous runtime and failure-mode diagnostics across 20+ benchmarks, making the harness a swappable, measurable variable rather than a confound.

Use AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.13705; inspect its method and evaluation before treating results as production evidence.

[2607.13705] AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.

Kai Chen; Zichen Ding; Jiaye Ge; Shufan Jiang; Mo Li; Qingqiu Li; Zehao Li; Zonglin Li; Tianhao Liang; Shudong Liu; Zerun Ma; Zixin Shang; Wenhui Tian; Zun Wang; Liwei Wu; Zhenyu Wu; Jun Xu; Bowen Yang; Dingbo Yuan; Qi Zhang; Songyang Zhang; Peiheng Zhou; Dongsheng Zhu


The work turns loop quality into a measurable task or score. Controlled study on the 104-task XBOW benchmark showing plain coding agents already solve a large share of tasks attributed to specialized security harnesses, a methodological warning to establish model-matched baselines before crediting the harness/loop architecture for performance gains.

Use Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.13085; inspect its method and evaluation before treating results as production evidence.

[2607.13085] Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing

Recent autonomous penetration testing papers report high benchmark scores while adding multi-component security harnesses around frontier LLMs. Because these systems often change both architecture and backbone model, it is difficult to tell how much performance comes from the harness rather than from the underlying model. This paper presents a controlled study on the 104-task XBOW benchmark using default coding CLI agents as plain-agent baselines. We first run Codex, OpenCode, and Pi with the same GPT-5 model, budget, target interface, and scoring rule. This phase identifies the strongest same-model baseline and tests whether security-specific prompt variants improve its observed score. We then compare the default Codex scaffold with published MAPTA and PentestGPT V2 results under the closest available model matches. Finally, we repeat the plain-agent experiment with GPT-5.2 and GPT-5.5 to measure model scaling inside the same scaffold. The results show a mixed but practical picture. Specialised harnesses can add measurable benchmark lift and may improve cost efficiency, but plain coding agents already solve a large share of the benchmark; repeated plain-agent runs can match or exceed some published architecture scores in union coverage, and newer models substantially improve the same scaffold. Future evaluations should report model-matched plain-agent baselines before attributing benchmark gains to architecture design alone.

Ananda Dhakal; Krish Neupane; Aarjan Chaudhary


Packages the evidence as queryable CSV and JSONL rather than only a rendered page. Tencent's coding-agent evaluation suite spanning Code, Web, Office, and Security domains, where every task is reverse-engineered from a real commit, PR, or business scenario and rewritten as a short colloquial role-played request, contamination resistance comes from this construction plus dataset versioning rather than secrecy, so the full task directories, environment images, tests, and solutions ship publicly for third-party audit, alongside a cross-model leaderboard (per-subset scoring, not comparable across subsets).

Use Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction to measure progress and gate completion with repeatable evidence.

[2607.20911] Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction

We introduce Tencent WorkBuddy Bench, a multi-domain evaluation suite for coding agents; this report documents its construction methodology, scoring protocol, and a cross-model leaderboard. At its core is a unified evaluation framework for constructing and running distribution-informed coding-agent tasks across four work domains - Code, Web, Office, and Security. Rather than adapting public issue text, every task is reverse-engineered from a real commit, pull request, or business scenario and rewritten as a short, colloquial, role-played request, so that a task's prompt is not recoverable by web-searching the underlying issue, pull request, or commit thread. Because the dataset is released openly - task directories, environment images, evaluation harness, tests, and reference solutions - contamination resistance rests on this construction together with dataset versioning rather than on secrecy. The four subsets - repository-level engineering, front-end development, office and business workflows, and red-/blue-team security - probe complementary facets of real work, each with its own verification style. All are packaged in a uniform task-directory format and run, under a uniform and reproducible protocol, on two agent harnesses (CodeBuddy Code and Claude Code); the full open release makes the benchmark reproducible end to end and directly auditable, since any third party can re-run each task and inspect its content. Because each subset uses a different scoring instrument, scores are not comparable across subsets and the suite reports no suite-wide average. We report a cross-model leaderboard across several model families.

Tencent WorkBuddy Bench Team; Siqi Cai; Shaopeng Chen; Xiang Fei; Yong Mao; Zihan Xu; Zhiheng Lyu; Zhijian Shao; Yuchen Shi; Shuwen Zhang; Chaofan Qiu; Linjie Che; Xiaoxi Zhao; Feng Wu; Kai Zhang; Chaofan Zhu; Yubin Qi; Xiaoyun Liang; Peijie Dong; Yunhao Zhang; Yuanjie Zhu; Ling Jiang; Xianjun Zhang; Zhehang Chu; Anyuan Sang; Zhen Feng; Sen Nie; Shi Wu; Yuanzhen Xu; Xin Li; Ning Yang; Zhiqiang Dong; Hande Dong; Qiang Lin; Yi Liu; Yunsheng Wu; Ke Li; Xing Sun

30 pages, 9 figures. Project page: https://workbuddybench.com/ ; code: https://github.com/Tencent/workbuddy-bench ; dataset: https://huggingface.co/datasets/tencent/workbuddy-bench


The work turns loop quality into a measurable task or score. Benchmark that evaluates coding agents as interactive project builders: agents must turn ambiguous product intent into working software through planning, requirement clarification, tool use, debugging, and repository-level construction across multi-turn sessions with simulated users, graded on functional correctness, structural fidelity, and interaction quality rather than static fully specified tasks.

Use ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders to measure progress and gate completion with repeatable evidence.

[2607.21217] ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are increasingly expected to transform incomplete product intent into working software by combining various abilities including planning, requirement clarification, tool use, debugging, and repository-level construction. Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings. The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent. To make this setting both realistic and evaluable, ICAE-Bench introduces three key designs. First, to avoid the ambiguity of unconstrained fuzzy requirements, each task derives ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interaction through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to evaluate open-ended repositories fairly, ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.

Zhongyuan Peng; Dan Huang; Chuyu Zhang; Caijun Xu; Changyi Xiao; Shibo Hong; David Lo; Lin Qiu; Xuezhi Cao; Jiyuan He; Yixin Cao


The work turns loop quality into a measurable task or score. 580-scenario safety benchmark spanning six domains and five adversarial attack modes, run on LangChain, LlamaIndex, and Vectara agents; even the strongest configuration reaches only 74.8% accuracy, and a guardrail defense recovers 19.9% of failures at a 0.5% false-positive rate, mapping distinct failure regimes for guarding tool-using autonomous agents.

Use GuardianAgentBench: Where Agents Fail and How to Guard Them to measure progress and gate completion with repeatable evidence.

[2607.20982] GuardianAgentBench: Where Agents Fail and How to Guard Them

As large language model agents increasingly operate autonomously with access to tools and external environments, ensuring their safe and reliable behavior becomes critical. We present GuardianAgentBench (GABench), a benchmark of 580 scenarios across six domains evaluated on three production-ready frameworks: LangChain, LlamaIndex, and Vectara. The benchmark incorporates rigorous multi-stage validation and five adversarial attack modes. Experiments with six state-of-the-art models reveal that even the strongest configuration achieves only 74.8% overall accuracy and expose two distinct failure regimes: stronger models under-call required tools, while weaker models mis-select and over-call tools. Performance degrades monotonically with both tool-set size and sequential turn depth, with long-horizon planning proving the steeper bottleneck. Our guardrail implementation consistently outperforms system-prompt-based defenses across all models, recovering 19.9% of failures at a false positive rate of just 0.5%. These results demonstrate that execution-time structural intervention improves safety without disrupting correct agent behavior.

Vishal Ishwar Naik; Chenyu Xu; Donna Dong; Hussein Hassan; Abhishek Pradhan; Ofer Mendelevitch; Tallat Shafat; Humayun Irshad


LLMs Get Lost in Evolving User Intent

The work targets tasks that exceed a single context window or prompt session. Microsoft framework converting static single-turn tasks into dynamic multi-turn conversations where users disclose, revise, and reshape intent across the loop rather than specify it upfront, showing substantial performance drops across model families, as the interaction-loop counterpart to long-horizon execution and a follow-on to the influential 'LLMs Get Lost in Multi-Turn Conversation'.

Use LLMs Get Lost in Evolving User Intent to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.20734; inspect its method and evaluation before treating results as production evidence.

[2607.20734] LLMs Get Lost in Evolving User Intent

As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user-delegated tasks through iterative interaction. Yet genuine interaction is inherently dynamic: users rarely specify their intent upfront, instead disclosing, revising, and reshaping it as the conversation unfolds. Despite this, LLMs are still predominantly evaluated or trained in single-turn, fully-specified settings, leaving open a fundamental question: how well do LLMs track and act on user intent as it evolves over the course of a conversation? To study this, we introduce a framework that transforms static, single-turn tasks into dynamic multi-turn conversations in which the user's intent evolves across turns--incrementally revealed, revised, and at times redirected mid-conversation--while preserving each task's original evaluation protocol, enabling existing benchmarks to be reused as controlled testbeds without new annotation. Across multiple tasks, we surface a consistent phenomenon: strong static-setting performance does not transfer to the evolving-intent setting, with substantial drops across model families. Our findings point to a fundamental gap: today's LLMs do not yet faithfully track and act on the user's evolving intent, a capability invisible to static evaluation yet critical for future collaborative agents.

Jihoon Tack; Philippe Laban; Jennifer Neville


Control flow is represented as an inspectable graph rather than an opaque prompt loop. Benchmark generator producing arbitrarily scalable task graphs, natural-language problems paired with executable Python solvers linked by typed intermediate values, with controllable length, dependencies, distractors, and exact automatic verification. Tests whether tool-using agents retain, update, compose, and discard context across extended workflows; a Qwen3.5-27B agent loses up to 33.3% accuracy on branching dependency chains versus isolated tasks, exposing context-management failures invisible to single-task evals.

Use ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management to measure progress and gate completion with repeatable evidence.

[2607.20764] ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management

We introduce ARBIGRAPH, a benchmark generator for evaluating whether tool-assisted language agents can retain, update, compose, and discard task-relevant context across extended reasoning workflows. ARBIGRAPH represents each task as a natural-language problem with an executable Python solver, and composes tasks through typed intermediate states, instantiated here as scalar and list values. This design enables controllable task graphs whose length, dependency structure, distractor count, and value type can be varied while preserving exact automatic verification. We instantiate ARBIGRAPH with math, GSM-style word-problems, and Python-tracing task categories, and evaluate a Qwen3.5-27B tool-assisted agent across four topologies. The results show high accuracy on isolated tasks but substantial degradation on more complex dependent tasks: accuracy drops by up to 33.3% on branching chains of dependent math tasks. This shows that ARBIGRAPH exposes failures that are not visible from single-task evaluation alone. Our code, generated datasets, and evaluation results are available at https://github.com/pavelgolikov/ArbiGraph.git

Pavel Golikov; Evgenii Opryshko; Gennady Pekhimenko; Mark C. Jeffrey


Verification is promoted from a final check to a loop-control signal. Red-teams the monitor layer of coding-agent loops: can a weaker LLM verifier catch hidden bugs in code that already passes public tests? Introduces CodeMonitorBench and a monitor-red-teaming protocol varying generator adversarial pressure, verifier scaffolding, and weak-to-strong capability gaps; of 43,677 public-test-passing samples, 23,081 hide bugs, and weak monitors miss most of them at a 5% false-positive rate, degrading further when generators overfit the public tests.

Use Code Monitor Red Teaming for Public-Test-Passing Code to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.20852; inspect its method and evaluation before treating results as production evidence.

[2607.20852] Code Monitor Red Teaming for Public-Test-Passing Code

Visible tests are a common gate for LLM-generated code, but passing them does not certify specification correctness. We study a deployment-like monitoring problem: after code has passed public tests, can a weaker LLM verifier identify the residual hidden bugs? We introduce Code Monitor Red Teaming, a monitor-red-teaming protocol that fixes a public-check information boundary while varying generator pressure, verifier scaffolding, and weak-to-strong capability. We instantiate it as CodeMonitorBench, spanning function-level, data-science, and workflow code. Across 71,000 generated candidates, 43,677 pass public tests and 23,081 of those fail hidden tests. Weak verifiers improve with scaffolding and model family, but still miss most hidden bugs at 5% false-positive rate. As a robustness stress test, adversarial public-test-overfit pressure lowers verifier AUROC and raises low-FPR miss rates in most cells. A GLM-5.1 verifier recovers part of the gap under the same evidence boundary; an inferability audit shows that remaining misses mix verifier failures with M1 evidence limits.