Evaluation data is used as the feedback signal for improving loop behavior. 103 expert-curated tasks across 10+ scientific disciplines evaluating long-horizon agent information-seeking loops, scientific database navigation, ambiguous literature retrieval, missing-reference completion, and cross-source knowledge synthesis. Evaluation of 10+ SOTA models shows accuracy collapsing as loop depth grows from entity-level reasoning to domain-level synthesis.

Use SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration to measure progress and gate completion with repeatable evidence.

[2607.20926] SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration

Scientific research involves complex information-seeking and reasoning workflows across heterogeneous sources. However, existing benchmarks primarily emphasize general-domain retrieval or static scientific question answering, and therefore fail to assess key capabilities required in realistic scientific research workflows. We introduce SciExplore, a benchmark designed to evaluate scientific information-seeking and reasoning capabilities of LLMs and agents. SciExplore comprises four task types covering 103 expert-curated tasks across more than ten scientific disciplines: scientific database navigation, ambiguous literature retrieval, missing reference completion, and cross-source structured knowledge synthesis, which probe progressively higher-level abilities from entity-level reasoning and document-level identification to evidence-level grounding and domain-level synthesis. We evaluate over ten state-of-the-art LLMs and autonomous agents on SciExplore, revealing substantial performance gaps with performance degrading sharply as task complexity increases and extremely low accuracy on the most challenging structured synthesis tasks. These results highlight significant limitations of current models and agents in realistic scientific information-seeking scenarios.

Yinhao Tang; Youqing Fang; Yanan Sun; Wenran Liu; Weiming Zhang; Bin Liu; Kuikun Liu; Wenwei Zhang; Kai Chen

Published in Findings of the Association for Computational Linguistics: ACL 2026; the linked arXiv record remains available for open access.


RUMBA: Russian User Memory Benchmark

Evaluation data is used as the feedback signal for improving loop behavior. Long-term conversational memory benchmark built from timestamped multi-session user-assistant dialogues, with a fine-grained taxonomy of memory-centric questions spanning semantic type, session scope, temporal reasoning, and explicitness of temporal expressions. Russian-first with an aligned English subset; evaluates contemporary memory systems against long-context baselines to surface failure modes of different memory mechanisms, first major non-English entry in the agent-memory eval space.

Use RUMBA: Russian User Memory Benchmark to measure progress and gate completion with repeatable evidence.

Benchmark or leaderboard source with repeatable tasks or scores; compare systems only after checking setup and scope.

Gate progress with tests, evals, and evidence.

[2607.21447] RUMBA: Russian User Memory Benchmark

The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning. To address this, we introduce RUMBA (Russian User Memory BenchmArk) - a new benchmark for long-term conversational memory that provides a fine-grained taxonomy of memory-centric question types and a unified methodology accounting for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. RUMBA consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. While designed for Russian, we also provide an aligned English subset under the same methodology. We evaluate contemporary memory systems and long-context models, and show how RUMBA serves as a diagnostic tool to analyze model behavior across benchmark slices and identify strengths and failure modes of different memory mechanisms.

Elizaveta Shevtsova; Inna Glebkina; Mark Baushenko; Pavel Gulyaev; Alena Fenogenova


Evaluation data is used as the feedback signal for improving loop behavior. Formulates protocol validity -- a benchmark score supports a capability claim only if the evaluation protocol keeps the intended capability necessary for success -- and introduces HackDetect, a post-hoc audit that identifies an exposure, determines how the agent used it, and judges whether the score is misleading. Quantifies inflation with the Mislead gap (exploit score minus intended score). Directly relevant to anyone building eval loops where agents can read evaluation artifacts, recover public solutions, or manipulate feedback.

Use Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.22368; inspect its method and evaluation before treating results as production evidence.

[2607.22368] Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI

Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction. Their scores support capability claims only when the evaluation protocol keeps the intended capability necessary for success. Recent reward-hacking benchmarks and system reports show that agents can instead recover public solutions, read evaluation artifacts, infer generator structure, manipulate feedback, or benefit from invalid scoring paths; existing responses do not provide a common procedure for attributing these shortcuts and quantifying their effect across benchmarks. We formulate protocol validity and introduce HackDetect, a post-hoc audit that identifies an exposure, determines how the agent used it, and assesses whether the resulting score is misleading. We quantify score inflation with the Mislead gap, defined as the exploit score minus the intended score. We audit 2,385 traces across 15 agent benchmarks and find evidence of exposures and reward hacking in 67.0% of Frontier Science traces and 66.7% of AutoLab tasks. Across paired comparisons, we measure score inflation of 0.45-1.00, showing that benchmark reports should provide evidence that scores reflect the intended capability.

Jiaqi Shao; Hanck Chen; Wei Zhang; Maxm Pan; Bing Luo


Verification is promoted from a final check to a loop-control signal. Once an agent can change its own information state during evaluation, correctness stops distinguishing intended reasoning from answer acquisition. Names the missing evaluation object 'success provenance' and audits it with AcquaBench via matched CLEAN/GOLD/SHAM value substitution: GOLD-minus-CLEAN measures response to correct-target availability, GOLD-minus-SHAM tests whether that response tracks target correctness beyond matched source exposure. A sharper instrument than exposure detection for verified agent loops.

Use Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.24054; inspect its method and evaluation before treating results as production evidence.

[2607.24054] Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation

A correct answer can conceal why an agent succeeded. Once agents change their information state during evaluation, correctness no longer distinguishes intended reasoning from answer acquisition. Outcome evidence and exposure detection do not establish whether success depended on an acquired target; we call this missing evaluation object success provenance. AcquaBench audits it through matched CLEAN, GOLD, and SHAM value substitution on four standardized surfaces with joint qid-clustered analysis. CLEAN retains benchmark-authorized information. GOLD makes the correct target available. SHAM preserves source structure and exposure opportunity but substitutes a matched incorrect value. GOLD minus CLEAN measures the total score response to correct-target availability; GOLD minus SHAM tests whether that response tracks target correctness beyond matched source exposure. In D0, GOLD exceeds SHAM by 19.1 to 25.9 percentage points, showing that success follows the correct value. In D2, GOLD still exceeds SHAM under distributed sufficiency while coloc no longer transfers as a high-score marker, with AUROC 0.376 and 0.142. Behavioral dependence can thus persist beyond this probe's intended observation unit. In model comparison, a supported 5.0-point CLEAN score gap compresses to a raw GOLD difference of -0.6 points without establishing rank inversion. Agent benchmarks should report success together with whether the evaluated information state supported it.

17 pages, 3 figures, including supplementary material. Code: https://github.com/luojingkun22/acquabench


Context is managed as durable loop state rather than a single prompt payload. Introduces Skill Test Coverage: how completely a Skill's test suite covers its operational test obligations, as opposed to the usual question of whether the Skill improves task success. Passing available testcases never reveals which specified behaviors -- resource operations, user choices, validation steps, recovery paths -- have never been exercised. Because those obligations are implicit in natural-language Skill packages, they build a pipeline that recovers them and organizes their workflow context. Testing discipline for the harness layer, where almost none currently exists.

Use Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.22015; inspect its method and evaluation before treating results as production evidence.

[2607.22015] Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice

Cloud platforms increasingly deliver reusable Cloud Skills that guide AI agents in creating, configuring, monitoring, and troubleshooting cloud resources. Workflow-oriented Skills specify multiple resource operations, user choices, validation steps, and recovery behavior. Existing Skill evaluation primarily measures whether a Skill improves task success. Passing the available testcases, however, does not reveal which behaviors specified by the Skill have never been tested. This paper introduces Skill Test Coverage, which measures how completely a Skill's test suite covers its operational test obligations. We define the coverage units, testcase coverage relation, calculation procedure, and claim boundary. Because these units and links are not explicit in natural-language Skill packages, we develop a measurement pipeline that recovers operational obligations, organizes their workflow context, maps testcases using both user prompts and initial resource states, and produces an auditable coverage report. Model-assisted candidate generation is combined with expert review to ensure that the reported coverage remains grounded in the original Skill and testcase evidence. A post-review module then translates confirmed coverage gaps into source-grounded test-improvement recommendations.

Haotian Si; Junyi Chen; Shuyang Yu; Ruifeng Nie; Jiate Li; Jianqiang Zhao; Meng Li; Dengcheng He


Links loop design to measurable tasks where progress and failure can be compared. Frames tool creation as an investment decision -- pay a fixed cost now for possible future reuse -- and tests whether agents allocate a fixed budget toward a few highly reusable tools rather than many one-offs. The result is a clean transfer failure: every frontier model tested (Claude Haiku, Claude Opus, GPT-5.4-mini, GPT-5.6 Sol) acts near-optimally in the abstract text framing but collapses in the code-construction version, with three of four failing even when the scripts are never evaluated. Directly relevant to self-extending agents that build their own tooling.

Use AlloBench: Measuring Online Tool Allocation Capability in LLM Agents to measure progress and gate completion with repeatable evidence.

[2607.23332] AllocBench: Measuring Online Tool Allocation Capability in LLM Agents

Creating a reusable tool is an investment: an agent pays a fixed cost now in exchange for the potential of future reuse. Therefore, a user should prefer an agent that creates a small number of highly reusable tools, rather than many one-offs. We introduce a paired benchmark that tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in two contexts: an abstract text-based formulation and a code-construction task. We find that every frontier model we test---Claude Haiku, Claude Opus, GPT-5.4-mini, and GPT-5.6 Sol---acts near-optimally in the abstract framing but fails to transfer this ability to script-writing. Through further experiments, we identify the particular failure modes for each model. Notably, the first three models fail even when the scripts are not evaluated, while GPT-5.6 Sol stays selective under that weaker manipulation and collapses only at full construction. Furthermore, an open-source Qwen model policy-trained for abstract allocation generalizes this ability across held-out lexical variations, but sees no improvement at script allocation. Together, these results establish online tool allocation as a significant capability boundary, even for modern frontier models.


Evaluation data is used as the feedback signal for improving loop behavior. Makes the unit of evaluation a verifiable deliverable produced inside a constrained workflow rather than a knowledge, reasoning, or tool-use score.

Use SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented Workflows to measure progress and gate completion with repeatable evidence.

[2607.23123] SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented Workflows

Existing evaluations of large language models cover knowledge, reasoning, coding, and tool use, but they rarely treat a verifiable deliverable produced within a constrained workflow as the unit of evaluation. We introduce SQBench, a benchmark for evaluating production-oriented task delivery by language-model agents. SQBench v1.0 contains 220 standardized tasks organized into L1 atomic capabilities, L2 composite skills, and L3 business scenarios. Each task requires an agent to process input assets, use available tools, and produce an explicitly specified deliverable. The evaluation first computes functional Completion and then derives Risk Penalty and Performance from independently evidenced triggers in a 10D Risk Matrix. A Strict Pass requires Completion = 1 and Risk Penalty = 0. We evaluate 27 model configurations under a common protocol, with one run per configuration-task pair. The highest prespecified Weighted Pass@1 is 60.5%. Mean Strict Pass@1 on L3 is 18.5%, and every configuration performs worse on L3 than on both L1 and L2, indicating that delivery under domain constraints is a shared weakness within the current task set. Of 2,348 results with Completion = 1, 113 (4.8%) fail the Strict Pass criterion because of risks such as unverifiable citations, inappropriate resource use, or format violations. These results show that functional completion alone does not fully characterize delivery quality and that risk determinations should be reported separately.

17 pages, 8 figures. Code and aggregate results: https://github.com/shaqiu-ai/SQBench


Control flow is represented as an inspectable graph rather than an opaque prompt loop. Targets the gap between isolated-API-call benchmarks and agents that actually gather hidden information, compose tool calls, and commit state changes in stateful environments. 323 state-changing tasks across three real product domains (Honor of Kings, QQ Music, Tencent Meeting), fully synthetic but built by decoupling environment synthesis from task synthesis -- graph-guided database filling produces reusable orphan-free environments, and generator-solver asymmetry manufactures tasks with both an information gap and a tool gap.

Use E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios to measure progress and gate completion with repeatable evidence.

[2607.23722] E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, and committing state changes. We refer to this capability as multi-step tool use. Existing benchmarks have advanced tool-use agent evaluation, but often focus on isolated API calls, short trajectories, or settings that are difficult to scale or control. We introduce E-Bench, a fully synthetic benchmark with 323 state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. E-Bench decouples environment synthesis from task synthesis: graph-guided database filling builds reusable, orphan-free product environments, while generator-solver asymmetry creates tasks with both an information gap and a tool gap, requiring agents to discover hidden data and compose multiple tool calls before changing state. Outcomes are graded deterministically by database-state diffs. Since both environments and tasks are synthetic, E-Bench is controllable at the environment level and scalable at the task level. Benchmarking 11 cutting-edge LLMs shows that multi-step tool use remains challenging: Pass^3 stays below 60% for the strongest models, and even with code execution in the E-Bench-Code extension, reliability (Pass^3) remains below 70%.

Weihuang Zheng; Tianyuan Zou; Eileen Ye; Alphet Liu; Youyong Kong; Ya-Qin Zhang; Duran Zheng; Maxm Pan


Evaluation data is used as the feedback signal for improving loop behavior. Builds an evaluation environment with the properties real agent loops face and most benchmarks lack: live multi-turn read-write interaction with a running instrumented PostgreSQL under active workload, persistent state, observation spaces spanning thousands of time series plus business logs and concurrent activity, open solution spaces with different operational trade-offs, and faults cascading across internal mechanisms. Success is defined by measurable recovery rather than trajectory matching. A strong template for long-horizon ops-agent evaluation generally.

Use DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents to measure progress and gate completion with repeatable evidence.

[2607.22165] DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents

LLM-based database agents show promise, but differing task scopes, testbeds, and metrics hinder comparison. We identify four gaps between evaluation and production operations: live-environment fidelity (multi-turn read-write interaction with a running database); observation-space scale and complexity (causal diagnosis across thousands of time series, business logs, and concurrent activity); solution-space openness (multiple remediations with different operational trade-offs); and scenario complexity and coverage (faults cascading across internal mechanisms and operational domains). We present DBA-Bench, a benchmark addressing these gaps through production fidelity, outcome-first evaluation, and controlled scenario reproducibility. It uses instrumented PostgreSQL environments with active workloads, persistent state, and multi-source observations; defines success by measurable recovery or fault elimination under safety constraints; and restores snapshots with scenario-specific checks before each run. The benchmark contains 106 scenarios across seven task domains, with two public difficulty labels based on reference-path diagnostic depth and environmental complexity. We evaluate nine baseline groups, including six foundation-model systems, two GPT-5.5-backed database agents, and a Human DBA reference. Across 848 automated runs, Diagnosis, Outcome, and Safe Pass rates are 32.7%, 19.6%, and 12.4%; the best automated baseline reaches 17.9% Safe Pass versus 93.4% for the Human DBA reference. Automated Safe Pass falls from 19.6% on Easy scenarios to 7.6% on Hard scenarios, underscoring the difficulty of safe end-to-end remediation.

Junming Chen; Junyang Jiang; Xu Chen; Zibo Liang; Kai Zheng


Control flow is represented as an inspectable graph rather than an opaque prompt loop. Scores orchestration strategies on their own by building task dependency graphs and running a deterministic simulator, correlating strongly with real Claude Code performance at a fraction of the tokens. Lets teams iterate on delegation topology without paying for full agent runs.

Use OrchBench: Evaluating Multi-Agent Orchestration Plans in Isolation via Deterministic Simulation to measure progress and gate completion with repeatable evidence.

[2607.25656] OrchBench: Evaluating Multi-Agent Orchestration Plans in Isolation via Deterministic Simulation

Complex tasks often decompose into parallelizable yet interdependent subtasks, making orchestration critical to the performance of multi-agent systems (MAS). Existing evaluations typically rely on end-to-end execution, which conflates orchestration-plan quality with worker capabilities, tool reliability, and environmental noise. Moreover, the time and token costs of real execution grow rapidly with workflow scale, making systematic evaluation expensive. We present OrchBench, a simulation-based benchmark for evaluating multi-agent orchestration plans in isolation. Starting from real-world tasks, OrchBench constructs directed acyclic graphs (DAGs) that encode task dependencies, with controlled sizes and degrees of parallelism. Given a DAG, a per-agent context limit, and an agent budget, the evaluated planner assigns subtasks to agents and specifies cross-agent information transfers and their retention ratios. A deterministic simulator evaluates the resulting plan without invoking worker agents and returns interpretable measures of result quality, makespan, and token cost. The simulated scores produced by OrchBench correlate strongly with quality scores from Claude Code executions, achieving a Pearson correlation of \(r=0.816\), while requiring only \(1.3\%\) of the tokens and \(10.3\%\) of the wall-clock time. Across diverse planners and workflow scales, we find that preserving task-critical information is more important than simply increasing the number of agents, and the benefits of parallelism diminish as coordination failures accumulate. These results establish OrchBench as an efficient and interpretable benchmark for comparing and diagnosing multi-agent orchestration plans.

Zhenzhen Ren; Jiyan He; Xinpeng Zhang; Zhenxing Qian; Ke Han; Shuxin Zheng; GuoBiao Li; Xiaoqing Zhang


The work turns loop quality into a measurable task or score. 65 tasks requiring agents to obey 20-124 page policy documents in simulated professional settings, where the best configuration reaches only 36.2% under strict grading and agents override policy for in-environment requests or lose rules over long runs. Measures whether the standing instructions a loop depends on actually hold.

Use HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following to measure progress and gate completion with repeatable evidence.

[2607.25398] HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let that document govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document constrains its behavior over an extended tool-use horizon. We present HANDBOOK_md, a benchmark of 65 agentic tasks modeled on how employees follow company handbooks. Each task places an agent in a self-contained company environment (a file workspace with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol) and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20-124 pages. Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and 10 fictional companies. To resist memorization, every task modifies one of 10 base handbooks, altering the specific rules and thresholds on which grading depends, so no two tasks share the same set of policies. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not. Under strict grading, where a trial passes only if every criterion is satisfied, the strongest evaluated model passes 36.2% of trials, and most frontier models remain below 25%. Failures follow consistent patterns: agents let a plausible but unauthorized in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve. We release the tasks, environments, and evaluation harness.

Liudas Panavas; Sebastian Minus; Bradley Monton; Derek Ray; Suhaas Garre; Sushant Mehta; Edwin Chen

16 pages, 3 figures, 5 tables. Accepted to the Workshop on Agent Behavior (WAB) at COLM 2026. Benchmark, environments, and evaluation harness: https://github.com/surge-ai/handbook


The work targets tasks that exceed a single context window or prompt session. 100 long-horizon office-suite tasks paired with economic signals so agent output can be priced against human labor; models come out faster and cheaper but not yet at human deliverable quality. The cost-grounded framing is more decision-useful than raw success rates.

Use OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding to measure progress and gate completion with repeatable evidence.

[2607.27155] OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.

Jingbo Zhou; Yusai Zhao; Qi Bao; Jingjia Cao; Zhenghai Chen; Chang Gao; Kaiqi Guo; Muxin Guo; Mingxuan Li; Xinjiang Lu; Yanru Ma; Yixiong Xiao; Zenghui Zhang; Le Zhang; Hua Wu


The work turns loop quality into a measurable task or score. Controlled benchmark for agents iteratively refining their own training-data strategy: agents beat their first attempt in 58.33% of runs, but 78.26% of searches that ran past peak ended below it. Quantifies the knowing-when-to-stop failure that undercuts recursive self-improvement claims.

Use RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement to measure progress and gate completion with repeatable evidence.

[2607.25886] RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement

Recursive self-improvement requires turning evidence of model failures into better models. Data-centric post-training research entails diagnosing capability gaps, designing and validating training-data strategies, and learning from checkpoint feedback. Can LLM agents automate this loop? Existing benchmarks entangle research decisions with optimization, serving, evaluation, and systems implementation, obscuring agents' research capability. We introduce RSIBench-Data, a controlled benchmark of LLM agents as data-centric researchers with a fixed post-training stack. Agents iteratively revise training-data strategies for a fixed target model; training and serving use Tinker-backed services, official evaluation runs through Harbor and E2B sandboxes, and budgets are fixed across agents. We evaluate four frontier agents on six benchmarks across software engineering, terminal use, scientific question answering, and mathematics. Agents demonstrate core data-centric research capabilities: in 58.33\% of settings, they improve upon the first valid attempt by refining strategies from feedback. However, improvement is inconsistent. Among searches continuing after the best observed score, 78.26\% end with a lower-scoring final attempt, while the rest only recover the same peak. A strong candidate may therefore appear early or midway through a run even as later revisions fail. Trajectory analysis identifies four patterns in stronger runs: accurate hypotheses, validation-grounded supervision, behavior-aligned data, and preservation of strong checkpoints. These findings suggest that current agents can make useful data-centric discoveries but cannot yet translate feedback into consistent improvements. RSIBench-Data provides a measurable, auditable testbed for the research capabilities required for recursive self-improvement. We open-source our code at https://github.com/evolvent-ai/RSIBench-Data.

Fanqing Meng; Lingxiao Du; Qiguang Chen; Ziqi Zhao; Haocheng Lu; Mengkang Hu; Michael Qizhe Shieh


Control flow is represented as an inspectable graph rather than an opaque prompt loop. 1,151 auditable tasks testing whether enterprise agents route to the right surface among documents, tables, and dependency graphs; agents route well but answer poorly, showing source selection is necessary and far from sufficient. Isolates a routing stage most agent evaluations collapse into end-to-end accuracy.

Use WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing to measure progress and gate completion with repeatable evidence.

[2607.25765] WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing

Enterprise agents often need to integrate heterogeneous knowledge sources: documents for narrative facts, tables for computation, and dependency graphs for file relationships. Existing benchmarks typically evaluate retrieval or tool use without distinguishing whether an agent first selects the appropriate knowledge sources. We introduce WorkSurface-Bench, a benchmark for evaluating this capability as surface routing. It contains 1,151 atomic tasks derived from persona-scoped Workspace-Bench-Lite workspaces, spanning document, table, graph, and cross-surface questions. Its reference answers are auditable: table answers are reproduced through executed DuckDB queries, document answers are grounded in verified text spans, and graph answers are traced to source dependency annotations. We evaluate four model backbones across six controlled agent settings, yielding 27,624 protocol-error-free trajectories. Under gold-constrained tool access, agents achieve 98.7-99.8 Route F1, while Answer remains only 56.1-75.3 percent, showing that correct surface selection is necessary but insufficient for task completion. Matched interventions further show that surface hints improve Answer for three of four models, whereas removing irrelevant tools primarily improves routing and efficiency. In an independent three-annotator audit, all 200 sampled tasks pass all six quality criteria by majority vote, with 192 receiving unanimous judgments on every criterion. We release the dataset, construction pipeline, scoring code, and agent harness at https://github.com/haolpku/WorkSurface-Bench.

Hao Liang; Meiyi Qiang; Sizhe Qiu; Linzhuang Sun; Wentao Zhang


The agent workflow includes explicit self-checking or gated completion. 2,013 human-verified offline instances testing whether computer-use agents can reconstruct the causal effect of their own actions, verifying state change, attributing which action caused it, and handling context-aware control across apps. Targets the self-verification step an unattended GUI loop needs before it can act again.

Use Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions? to measure progress and gate completion with repeatable evidence.

[2607.26041] Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?

Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks. Current benchmarks primarily measure end-task success or single-frame grounding. Neither isolates whether a model can reconstruct the causal, task-relevant transition produced by an action- crucial for rejecting stale observations, verifying progress, and recovering from failure. This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous: the next observation may be delayed, occluded, transient, or unrelated, then misread as progress and carried into subsequent planning. We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories across ~15 applications and 50 task domains. DDB trajectories targets 3 failure dimensions- state verification, source tracking, and context-aware control- through 2 complementary tasks: 463 3-frame temporal-ordering instances, including 105 with a cross-trajectory decoy, and 1,550 before-after pairs labeled from 5 actions + its payload. We evaluate 8 closed and open-source model families across 32 ordering and 16 single-action settings, observing consistent gaps. Ordering remains unsaturated: best non-decoy and decoy exact-match rates are 65.1% and 65.7%. Task context improves decoy identification by 6.9 percentage points but reduces non-decoy exact match by 2.2 points; error analysis reveals systematic copying of the presented A-B-C order. Single-action results show that inferring the action family is harder than locating it: click F1 is 0.96 vs, 0.76 for drag, while recognized drags are generally localized well. DDB, thus, complements end-to-end benchmarks by filling the missing diagnostic layer between GUI grounding and final task success, enabling targeted improvements to desktop CUA verification, reliability, and recovery.

Abhishek Pillai; Samir Kumar Nayak; Yuan Chen


Evaluation data is used as the feedback signal for improving loop behavior. Unified corpus of nearly a million evaluation records spanning 30 benchmarks and over 700 agents, standardizing reporting and showing progress varies sharply by task type while some scoring methods distort capability estimates. Infrastructure for meta-analysis rather than another leaderboard.

Use Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation to measure progress and gate completion with repeatable evidence.

[2607.25891] Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules. Existing efforts focus on narrow settings, remain limited in scale, or require costly reruns, leaving much of the empirical record incomparable. We introduce Messier, a unified corpus of 957,253 records that span 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. Messier consolidates public benchmark scores and supplements them with five-agent runs across six underrepresented professional and scientific domains, including a recent legal benchmark. Each record is standardized by model, scaffold, environment, task, verifier, and aggregation rule, with SOC/NAICS classifications for occupational and industry analysis. Using this corpus, we show frontier progress is uneven across benchmark types, with "function calling" saturated, "programming" improving the fastest, and "enterprise workflows" remaining the most challenging. Furthermore, counterfactual rescoring shows that strict all-pass aggregation in multi-verifier tasks can obscure progress and artificially alter agent rankings. From these standardized records, we derive capability scales that align with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.81 and can be specialized by domain, occupation, action space, or verifier type. Messier provides a foundational, reusable infrastructure for agent capability scaling, benchmark auditing, and fine-grained analysis of evaluation failures.

Stefan Krsteski; Charlotte Meyer; Guillaume Allegre; Tony O'Halloran; Alexandre Sallinen


smevals

Evaluation data is used as the feedback signal for improving loop behavior. Simon Willison's writeup of smevals, an eval framework he built at Jesse Vincent's Prime Radiant lab that treats the agent harness as a first-class variable under test rather than a fixed backdrop.

Use smevals to measure progress and gate completion with repeatable evidence.

Working implementation, framework, runtime, or repository; signal comes from usable code and ecosystem adoption.

smevals - a small eval suite for evaluating models, prompts, and harnesses | Prime Radiant


Links loop design to measurable tasks where progress and failure can be compared. Puts coding agents into a production-fidelity oncall setting, with a live instrumented microservice system and root-cause tasks whose ground truth is signed off by expert SREs. Best root-cause accuracy is 25.3 percent on realistic medium-difficulty inputs, the weakest model invents a root cause in 40 percent of reports, and the authors frame their numbers as a lower bound.

Use ORCA-bench: How Ready Are Language Model Agents for Oncall? to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.28545; inspect its method and evaluation before treating results as production evidence.

[2607.28545] ORCA-bench: How Ready Are Language Model Agents for Oncall?

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $\kappa_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/orca-bench.

Albert Gong; Kyuseong Choi; Abhineet Agarwal; Jason Schechner; Ryan Huang; Raj Agrawal; Anish Agarwal; Raaz Dwivedi


The work turns loop quality into a measurable task or score. Treats a benchmark score as the output of a pipeline that can fail at four stages, tasks go stale, trajectories omit decisive visual evidence, evaluators reject valid alternatives, and aggregate reports hide the cause of failure, then audits 150 public failure-scored trajectories across five web, enterprise-workflow, and desktop-control benchmarks.

Use How Benchmarks Mis-Score Computer-Use Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.28367; inspect its method and evaluation before treating results as production evidence.

[2607.28367] How Benchmarks Mis-Score Computer-Use Agents

Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3\% of FAIL verdicts are wrong: 10.7\% are evaluator false negatives and 4.7\% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.

Zihan Dong; Zhiyuan Ma; Zekun Wang; Yunqing Li; Zirou Liu; Ruixuan Deng; Qishi Zhan; Rui Qian


Links loop design to measurable tasks where progress and failure can be compared. Puts agents on forensic disk snapshots and host alerts across ten cyber ranges spanning multiple operating systems and attack techniques, testing whether they can both resolve the flagged issue and hunt for what the alert missed. No model achieved complete detection and remediation on any single range.

Use SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.26791; inspect its method and evaluation before treating results as production evidence.

[2607.26791] SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.

Lehan Wang; Boli Chen; Ruixue Ding; Pengjun Xie; Jinwei Huang; Zhendong Liu; Shuo Wang; Tao Lei; Xin Ouyang; Xiaomeng Li


Verification is promoted from a final check to a loop-control signal. Scores the trace, not just the outcome: a Process Grader rates goal alignment, efficiency, information utilization, and result verification against 12,541 rubric items over 320 tasks in 8 domains, explicitly to separate reliable reasoning from lucky success.

Use ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.28037; inspect its method and evaluation before treating results as production evidence.

[2607.28037] ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents

As LLM-based agents are deployed in complex, multi-step workflows, a critical evaluation gap has emerged: most existing benchmarks judge only final outcomes, unable to distinguish reliable reasoning from lucky success or attribute failures to specific process deficiencies, hindering attribution in long-horizon tasks. In this work, we present ClawTrack, a dual-assessment benchmark that simultaneously measures what an agent achieves (Task Score) and how it achieves it (Process Score). ClawTrack comprises 320 tasks across 8 domains with 25+ deterministic mock services. A Process Grader scores each reasoning turn along four dimensions (goal alignment, efficiency, information utilization, and result verification), anchored by 12,541 task-specific rubric items. Evaluating 21 models over 16,000+ trials, we find that: (1) process scores effectively attribute success and failure to specific reasoning dimensions, filtering lucky passes invisible to outcome-only evaluation; (2) the four dimensions are complementary, with result verification as the systematic bottleneck; (3) the framework is robust to evaluator choice across different judge LLMs; and (4) process-based trajectory filtering yields consistent post-training improvements across model scales.

Xingjian Wu; Xuhang Zhu; Xingchen Liu; Junlin Liu; Jianing Wang; Linsen Guo; Xiaoyu Li; Xuezhi Cao; Xunliang Cai


The work turns loop quality into a measurable task or score. 100 end-to-end data-engineering tasks across five execution engines (PySpark, MySQL, HiveSQL, PrestoSQL/Trino, FlinkSQL) built from production enterprise code, graded by deterministic rules rather than an LLM judge. Strongest model scores 74.9 overall, and the deterministic grading makes it usable as a real loop exit gate.

Use DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.28033; inspect its method and evaluation before treating results as production evidence.

[2607.28033] DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness

Large language models (LLMs) and LLM-based agents are increasingly being deployed to automate complex workflows, promising to revolutionize data management and processing. However, existing benchmarks predominantly focus on simplified Text-to-SQL translation or data analysis, leaving the critical and complex domain of end-to-end data engineering largely unexplored. To bridge this gap, we introduce DataClawEval, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios. Built upon production-grade code authored by professional enterprise data engineers, it comprises 100 rigorous, end-to-end tasks spanning five execution engines: PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL. Rather than non-deterministic LLM-as-a-judge scoring, each task is executed within a case-specific, isolated sandbox and graded by deterministic, rule-based scripts. Evaluating 16 frontier agents exposes critical limitations: The strongest model attains only 74.9 overall, and no single model dominates, as each excels on a different engine, revealing strict domain specialization rather than omnipotent proficiency. Thus, autonomous data engineering remains a formidable, unresolved challenge. We release our dataset, containerized environments, and deterministic evaluation scripts at https://github.com/Dicemy/DataClawEval/tree/master

Debin Meng; Jiaming Yang; Zefang Zong; Tengyue Xu; Haining Xie; Yang Li; Peng Chen


Verification is promoted from a final check to a loop-control signal. Finds 13.6% of SWE-bench Verified instances have a pull request that does not actually match its linked issue, across five misalignment patterns, then builds a multi-agent checker combining pattern identification, cross-agent synthesis, and code-level verification.

Use PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.28587; inspect its method and evaluation before treating results as production evidence.

[2607.28587] PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.

Accepted at the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026). Github Repo: https://github.com/manyiResearch/PAIChecker


Links loop design to measurable tasks where progress and failure can be compared. 188 tasks from real Python pull requests where the goal is quality improvement without behavior change, graded by 92 executable rules.

Use SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.27409; inspect its method and evaluation before treating results as production evidence.

[2607.27409] SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements

Although coding agents have achieved impressive performance on correctness-oriented benchmarks, their ability to make behavior-preserving non-functional improvements (NFIs) remains underexplored. In real-world software development, developers continuously improve software quality without changing observable behavior, yet existing benchmarks primarily evaluate functional correctness and provide limited support for assessing these non-functional improvements. In this paper, we present SWE-NFI, a benchmark for evaluating coding agents on NFIs beyond functional correctness. Our benchmark contains 188 tasks constructed from real merged pull requests in open-source Python projects. We operationalize developer-oriented NFIs into 92 executable rules and develop a comprehensive evaluation suite that combines functional correctness testing with rule-based NFI evaluation. We evaluate state-of-the-art commercial and open-source coding agents. Although the best-performing agent achieves a 70.0\% functional correctness rate, all evaluated agents generally fall short of human developers in overall NFI capability. The gap is particularly evident for structural code improvements, where agents' NFI scores range from 0.0 to 1.3, compared with 1.5 for the human reference. Our benchmark and findings provide a reproducible foundation for evaluating and advancing coding agents beyond functional correctness.

Pengyu Xue; He Yang Yuan; Xin Wang; Junkai Chen; Haonan Zhang; Boyuan Chen; Zishuo Ding; Zhenhao Li; Weiyi Shang


Verification is promoted from a final check to a loop-control signal. Turns repository history into verified, executable agent tasks with reconstructed environments via patch reversal and code mapping, validating correctness at each step. Covers bug fix, feature addition, test generation, API migration, and security repair, the pipeline you need to keep a coding loop's eval set fresh instead of frozen.

Use Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.28591; inspect its method and evaluation before treating results as production evidence.

[2607.28591] Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments

Scaling coding agents requires a continuing supply of executable data for training, benchmarking, and continuous evaluation. Each task must couple a realistic software state with a specification, development tools, and reliable verification. To expand this supply, we present Change2Task, a system grounded in repository history that converts merged pull requests into verified tasks on healthy modern revisions of the same repository. It aligns historical evidence with evolved code, reconstructs task states through Patch Reversal, Code Mapping, or Agent Reconstruction, and validates the lifecycle from a healthy base to a task state and a restored state. By deriving multiple tasks grounded in developer evidence from maintained environments, Change2Task provides executable data for coding agent training and evaluation while reducing repeated environment setup, storage, and task construction effort. We evaluate the system through five common and widely adopted coding agent task families: Bug Fix, Feature Addition, Test Generation, Application Programming Interface Migration, and Security Repair. Starting from 1,130 source changes eligible for construction, Change2Task achieves 79.6% verified task construction success across these task families. On a matched candidate set, it recovers 29.2% more verified tasks than a construction baseline based on pull requests. Historical and reconstructed cases achieve up to 98.0% matched outcome agreement under agent evaluation, while reuse of modern bases reduces measured expenditure across the complete pipeline by 10.8%.

Haomin Qi; Xingliang Wang; Xuanqi Gao; Baihui Sang; Xin Zhang; Minghua Ma; Pengfei Gao; Yu Kang; Qingwei Lin; Saravan Rajmohan; Dongmei Zhang; Qi Zhang

15 pages, 7 figures, and 15 tables, including appendices


Evaluation data is used as the feedback signal for improving loop behavior. Compiles specifications into stateful applications and runs a co-evolution loop that improves the environment and the model together from graded rollouts, isolating behavioral depth, interaction coverage, and continuous environment improvement as the properties that matter. A 9B model goes from 36.5% to 67.1% across evaluation splits.

Use Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.28074; inspect its method and evaluation before treating results as production evidence.

[2607.28074] Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale

Computer-use agents learn from what their actions change, so training one needs applications it can act on, break and reset. The applications that matter most are login-gated and stateful, so synthetic environments stand in for them. Recent pipelines generate such environments in bulk, which moves the bottleneck from how many exist to what is inside each one. The returns, we find, come from three properties: how much behavioural depth an environment carries, whether it targets the interaction an agent actually fails, and whether it improves alongside the model. We present Echoverse, which compiles specifications into stateful applications whose tasks are graded against the application's own database, and a co-evolution loop that reads every graded rollout twice: as repairs to the environment, its tasks and its verifier, and as training signal for the model. Trained on twelve such environments, a 9B model improves from $36.5\%$ to $67.1\%$ across fourteen evaluation splits, within fourteen points of the much larger frontier model that taught it. We examine each property in turn. On the same domains, shallow environments push live-site accuracy below the base model ($80.0 \to 75.0$) while deep ones raise it ($80.0 \to 85.0$ and $48.0 \to 65.0$); drilling one interface control across many renderings transfers to held-out widget families and to the open web; and repairing a single environment lifts the model trained on it from $16.2\%$ to $38.5\%$. The same worlds serve as reinforcement-learning environments, where a reward combining the grounded verifier with a dense per-step judge raises held-out score from $58.8\%$ to $68.0\%$. We release four environments as a benchmark, with their applications, seed data and grounded graders. Code: https://aka.ms/echoverse

Yash Pandya; Sahil Gupta; Sarthak Harne; Archana Yadav; Kavyansh Chourasia; Hussein Mozannar; Vibhav Vineet; Sara Abdali; Corby Rosset; Yash Lara; Ahmed Awadallah; Ece Kamar; Akshay Nambi


Links loop design to measurable tasks where progress and failure can be compared. Self-referential loop design: task synthesis is itself posed as a Terminal-Bench-format task, so agents generate, execute, and verify candidate tasks inside real containers and only executable, internally consistent tasks survive. Fine-tuning on just 3,221 synthesized trajectories reaches competitive Terminal-Bench 2.0 results.

Use Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.27929; inspect its method and evaluation before treating results as production evidence.

[2607.27929] Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training

Training terminal agents at scale requires diverse, verifiable terminal tasks and high-quality interaction trajectories, yet acquiring such data remains a significant challenge. Existing synthesis methods face two key limitations: (1) weak reliability caused by the disconnect between task generation and real execution, and (2) limited diversity and scalability due to dependence on existing repositories. We propose Meta-Task, a framework that redefines terminal task synthesis as a Terminal-Bench-format task itself: an agent operates within a real container environment to iteratively generate, execute, and verify tasks, so that synthesized components are checked for internal consistency and executability within the generation loop itself. Building upon this, we decouple the target task requirements along multiple dimensions, introduce a multi-phase mechanism that dynamically designs novel task specifications before producing the actual tasks, and incorporate optional external material support to enhance diversity and realism. We additionally apply LLM-as-Judge filtering to ensure the quality of the final training data. Experiments on Terminal-Bench 2.0 show that fine-tuning on only 3,221 Meta-Task synthesized trajectories achieves 22.5% and 31.8% Avg Pass@1 for Qwen3-14B and Qwen3-32B respectively, outperforming concurrent approaches with significantly less training data.

Zhihong Pan; Jiyuan He; Kai Zhang; Yupeng Han; Ze Liu; Yuze Zhao; Yongcong Ye; Zhaohua Yang


Agentic Engineering: The Agent Loop

Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Minimal mental model for the loop underlying agent operation.

Use Agentic Engineering: The Agent Loop to bound risk before recurring or unattended execution.

Contextual source from junpingyi.com; useful for practice signals or boundary conditions, not independent validation.

Bound permissions, cost, failure, and escalation.

Chapter 1: The Agent Loop — Agentic Engineering: How to Build AI Agents Like Claude Code

Chapter 1: The Agent Loop from Agentic Engineering: How to Build AI Agents Like Claude Code


Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Practical walkthrough of the base loop and common variants.

Use The agent loop: ReAct, plan-and-execute, reflection to bound risk before recurring or unattended execution.

Contextual source from www.kunwar.page; useful for practice signals or boundary conditions, not independent validation.

Chapter 67: The agent loop: ReAct, plan-and-execute, reflection — The Holy Grail Basic agent loop: generate, check for tool calls, execute tools and loop back, or return final answer on no tool call. ReAct interleaves Thought, Action, and Observation triplets; each Thought improves the next Action choice by externalizing reasoning. Agent cost vs single-shot: one LLM call versus 5-12 interleaved LLM and tool calls, showing the 10x cost and latency multiplier.

An agent is a loop of `model.generate()` calls with tool calls in between. The loop is the entire pattern


How to Build an Agent

Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Thorsten Ball's demystification of the inner agent loop: a model, a loop, and enough tokens.

Use How to Build an Agent to bound risk before recurring or unattended execution.

Contextual source from ampcode.com; useful for practice signals or boundary conditions, not independent validation.

Building a fully functional, code-editing agent in less than 400 lines.


Agentic Coding Recommendations

Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Armin Ronacher's field notes on which practices hold up when agents do most of the work.

Use Agentic Coding Recommendations to bound risk before recurring or unattended execution.

Contextual source from lucumr.pocoo.org; useful for practice signals or boundary conditions, not independent validation.

Agentic Coding Recommendations | Armin Ronacher's Thoughts and Writings

Current recommendations of agentic coding.


Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Practical delegation guidance from the Devin team on scoping tasks agents can actually finish.

Use Coding Agents 101: The Art of Actually Getting Things Done to bound risk before recurring or unattended execution.

Contextual source from devin.ai; useful for practice signals or boundary conditions, not independent validation.


How Anthropic teams use Claude Code

Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Cross-team field report of real recurring agent workflows in engineering, security, and data science.

Use How Anthropic teams use Claude Code to bound risk before recurring or unattended execution.

Contextual source from claude.com; useful for practice signals or boundary conditions, not independent validation.

How Anthropic teams use Claude Code | Claude by Anthropic

Teams across Anthropic use Claude Code for everything from debugging production issues and navigating unfamiliar codebases to building custom automation tools. Here's how.


How Boris Uses Claude Code

Workspace isolation is part of the loop design, not an afterthought. Unofficial but concrete compilation of Boris Cherny's autonomous setups: parallel worktrees, auto mode, `/loop`, `/schedule`, dynamic workflows, and `/goal` completion conditions.

Use How Boris Uses Claude Code to bound risk before recurring or unattended execution.

Contextual source from howborisusesclaudecode.com; useful for practice signals or boundary conditions, not independent validation.

Boris Cherny's Claude Code Tips — How He Actually Uses It (127+ Tips)

127+ tips from Boris Cherny, creator of Claude Code, on his daily workflow: CLAUDE.md, worktrees, plan mode, hooks, subagents, and more.


Primary-source operational guidance rather than commentary. Official walkthrough of a daily scheduled agentic workflow that ingests PR data, analyzes it, and publishes findings to a Discussion, a concrete recurring loop with trigger, intake, analysis, and output.

Use Agent of the Day: Copilot Agent PR Analysis to bound risk before recurring or unattended execution.

Contextual source from github.github.com; useful for practice signals or boundary conditions, not independent validation.

Agent of the Day – May 26, 2026 | GitHub Agentic Workflows

Copilot Agent PR Analysis: a daily workflow that monitors GitHub Copilot coding agent performance across pull requests


Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Production lifecycle in which repeatedly validated agent-loop behaviors are promoted into deterministic workflows and demoted on regression, cutting per-incident agent cost by over 70% across eight months of a cloud AIOps system.

Use Progressive Crystallization: Turning Agent Exploration into Deterministic, Lower-Cost Workflows to bound risk before recurring or unattended execution.

Research source arXiv:2607.07052; inspect its method and evaluation before treating results as production evidence.

[2607.07052] Progressive Crystallization: Turning Agent Exploration into Deterministic, Lower-Cost Workflows in Production

AI agents deployed for IT operations are typically permanent cost centers because every execution requires full LLM inference, even for previously solved problems. This paper introduces progressive crystallization, a lifecycle that treats agent exploration as a discovery mechanism rather than a permanent execution model. It defines a three-stage execution taxonomy, from fully agent-orchestrated to hybrid to fully deterministic workflows, together with an evidence-based promotion mechanism that converts repeatedly validated agent behaviors into cheaper and more reproducible deterministic workflows, while automatically demoting workflows that regress. Evaluated on a production cloud networking AIOps system processing tens of thousands of incidents per month, the approach increased deterministic execution from 0% to 45% over eight months, reduced per-incident agent costs by more than 70% despite doubling incident volume, and improved safety through greater reproducibility and auditability. The paper also presents the execution taxonomy, promotion and demotion criteria, trace extraction methodology, economic model, safety considerations, and discusses limitations and threats to validity.

Conference-style paper; 10 pages (estimated from manuscript formatting if applicable); focuses on agentic AI, AIOps, workflow automation, deterministic execution, and LLM cost optimization


Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Production tool-making pipeline that mines live execution traces to compile recurring SOP steps into validated, versioned tools agents call instead of regenerating code, cutting median latency 42% and errors up to 53% in a fulfillment-center alarm-triage deployment.

Use Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems to bound risk before recurring or unattended execution.

Research source arXiv:2607.08010; inspect its method and evaluation before treating results as production evidence.

[2607.08010] Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request. We replace this inference-time coding loop with an agentic tool-making pipeline that compiles repeated SOP steps into validated, versioned tools before deployment. The tool-maker grounds synthesis in the live environment as it collects execution traces, observes backend schemas and values, generates candidate tools, and repairs them against labeled cases. At runtime, the production agent calls these tools directly and falls back to code generation only when needed. We deploy the approach in a Fulfillment Center alarm-triage system, where an agent diagnoses alarms against a 44-node SOP over heterogeneous metric backends. In production, tool calls reduce p50 latency by 42%. On 1,500 historical alarms, they reduce end-to-end error rate by up to 53% by suppressing run-to-run variance in repeated steps. Because tools return compact structured verdicts, they also enable a simpler direct-call architecture, reducing p50 latency by a further 62% in a controlled ablation. Versioned tools also improve auditability and expose specification gaps and upstream data drift. Our results show that self-evolving agents can make industrial LLM systems faster, more reliable, and easier to operate.

Kalle Kujanpää; Ning Liu; Shahnawaz Alam; Yeshwanth Reddy Sura; Tianyu Yang; Kristina Klinkner; Shervin Malmasi


Verification is promoted from a final check to a loop-control signal. Sabrina Ramonov's practitioner walkthrough of building verified loops with Claude Code `/goal` and cloud routines: a six-part loop-engineering framework, five worked examples each with a verifiable end state, and a production daily support-ticket cleanup routine whose independent checker agents illustrate why the checker is the hard part.

Use AI Loop Engineering: Build Autonomous Agents with Claude Code /goal and Routines to bound risk before recurring or unattended execution.

Contextual source from www.sabrina.dev; useful for practice signals or boundary conditions, not independent validation.

AI Loop Engineering: Build Autonomous Agents with Claude Code /goal + Routines

What loop engineering means in 2026, how to use the Claude Code /goal command, and how to build your first autonomous AI agent with a routine.


The work separates roles across agents, verifiers, or orchestration layers. Proactive health-trajectory prediction for long-horizon multi-agent systems, aggregating 20 heterogeneous signals across five layers into a trust-margin metric with 8-hour forecasts at 76.8% direction accuracy, detecting degradation concealed by normal surface metrics across 15 days of production traffic.

Use Agent Delivery Engineering Predictive Reliability Framework to bound risk before recurring or unattended execution.

Research source arXiv:2607.07689; inspect its method and evaluation before treating results as production evidence.

[2607.07689] Agent Delivery Engineering Predictive Reliability Framework

Long-horizon LLM multi-agent systems face reliability risks invisible to infrastructure monitoring. We propose the ADE Predictive Reliability Framework (ADE-PRF), enabling proactive health trajectory prediction from passive degradation detection. ADE-PRF aggregates 20 heterogeneous signals across five layers into a Trust Margin (TM) metric (39.2-point dynamic range). Triple-method parallel prediction enables 8-hour forecasts: the Exponential method achieves MAE=1.228, Direction Accuracy=76.8%, with 99.65% within +/-10-point tolerance. Production validation spans 380,227 predictions and 280,579 validations across six agent profiles over 15 continuous days, plus seven sandbox-controlled experiments. Key findings include detection of "false prosperity" -- degradation concealed by normal surface metrics -- and immediate TM coupling with ground-truth states upon ADE plugin integration, with 16/20 factors relying on ADE-collected data. Exponential consistently outperforms Kalman. ADE-PRF provides among the earliest reliability quantification with forward-looking warnings for production LLM agents.


The trigger or cadence is explicit, making the workflow recurring rather than one-off. Makes cost part of the loop contract: an Oracle executes queries, a Driver chooses them, a shared vector budget covers wall time, CPU, money, and LLM tokens, and an append-only ledger records declared versus settled cost for every transaction. The Ising-model case study shows how mixed-fidelity scheduling can outperform high fidelity throughout at the tested budget.

Use CADAQUES: A Cost-Aware Dual Architecture for Query-Efficient Autonomous Discovery to bound risk before recurring or unattended execution.

Research source arXiv:2607.16127; inspect its method and evaluation before treating results as production evidence.

[2607.16127] CADAQUES: A Cost-Aware Dual Architecture for Query-Efficient Autonomous Discovery

Autonomous discovery systems couple a resource that answers queries (a simulator, instrument, or analytic model) to an algorithm that selects what to query next. Most software frameworks for this loop inherit the control structure of numerical optimization: campaigns run for a fixed number of iterations, query costs are absent from the programming interface, and decision-making is treated as free. In practice, queries may differ in cost by orders of magnitude, and planners built on large language models or expensive surrogates consume resources of their own. Here we present CADAQUES, an open-source Python framework built on one architectural principle: cost is a first-class primitive of the discovery loop. CADAQUES separates the loop into two structural protocols, an Oracle that answers queries and a Driver that proposes them, and charges both evaluations and decisions against a common vector-valued budget spanning wall time, CPU hours, monetary cost, and language model tokens. An append-only ledger records, for each transaction, the cost declared before execution and the cost settled afterwards, making their discrepancy an observable property of the campaign. We evaluate the architecture by locating the critical temperature of the two-dimensional Ising model from noisy finite-size estimates against the exact thermodynamic-limit reference. In this noisy setting, strategies that concentrate around the best observed result can be misled by noise-induced peaks, whereas a schedule that explores with cheap low-fidelity queries and refines with higher-fidelity ones yields lower and less variable errors than high fidelity throughout, at the studied budget scale. Metering adds tens of microseconds per iteration, three orders of magnitude below the cheapest oracle query. The framework is MIT-licensed and archived at Zenodo (doi:10.5281/zenodo.21293589).


rocketplaneIO

Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Self-hosted AI SRE for Kubernetes whose copilot investigates autonomously via eBPF traces, logs, and live service maps, but can only act through named, reversible, risk-graded operations.

Use rocketplaneIO to bound risk before recurring or unattended execution.

Inspectable GitHub source (159 stars; 4 forks; Apache-2.0 license; updated 2026-08-05); popularity is context, not proof of reliability.

GitHub - olemeyer/rocketplaneIO: Self-hosted AI SRE for Kubernetes — zero-instrumentation eBPF observability plus a copilot that fixes issues through guardrailed, self-verifying actions. BYO-LLM, air-gapped capable. · GitHub

Self-hosted AI SRE for Kubernetes — zero-instrumentation eBPF observability plus a copilot that fixes issues through guardrailed, self-verifying actions. BYO-LLM, air-gapped capable. - olemeyer/rocketplaneIO


Evaluation data is used as the feedback signal for improving loop behavior. Lorenzo Gentile's July 2026 case study of swapping the model inside Ploy's production website-building agent (plans pages, writes components, screenshots its own work, decides when done): roughly a third of initial cross-model eval failures traced to Opus-specific harness assumptions rather than the new model, GPT-5.6 invented placeholder values for optional tool parameters (fixed via required-but-nullable schemas, after 52-64% of file reads silently returned empty), cache keys needed workspace scoping (0% to 83.7% first-call hits), and reasoning state moved to self-contained encrypted blobs, concrete evidence that eval harnesses and regression gates are the load-bearing layer when changing the model inside a production loop.

Use Migrating a Production AI Agent to GPT-5.6 to bound risk before recurring or unattended execution.

Contextual source from ploy.ai; useful for practice signals or boundary conditions, not independent validation.

Migrating a production AI agent to GPT-5.6 | Ploy

For four months, no frontier model beat Claude Opus in our production evals. GPT-5.6 did. This is what we learned while migrating.


Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Runs coding agents independently 12 times across four scientific machine-learning papers; all workspaces satisfy the study's completion criteria, and 158 implementation targets are linked to report evidence, offering a concrete playbook for evidence-backed research replication.

Use Coding-agents can replicate scientific machine learning papers to bound risk before recurring or unattended execution.

Research source arXiv:2607.02134; inspect its method and evaluation before treating results as production evidence.

[2607.02134] Coding-agents can replicate scientific machine learning papers

Scientific machine learning papers typically make computational claims, e.g., that the relative mean square error is less than 5% or that the 95% predictive credible interval covers the test data. A coding agent can be prompted to replicate those claims from paper materials alone, but the prompt does not by itself reliably preserve progress or check whether generated evidence supports the paper's claims. We introduce Paper-replication, a workflow that makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill. The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims, record where matched evidence appears in the replication report, and pass validation checks before completion. We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers. All twelve workspaces pass the completion gate, and all 158 recorded targets are matched with report coverage. Even in this completed workspace state, repeated runs differ in how papers are divided into targets, in numerical fidelity to the source papers, in elapsed replication time, in the number of intermediate executions replaced before final evidence is accepted, and in the rules used to accept evidence. Paper-replication makes completion depend on workspace evidence and validation checks rather than on the agent's final message.


Control flow is represented as an inspectable graph rather than an opaque prompt loop. Reference architecture that decomposes ERP operations into role-aligned LLM agents under a graph-based planner-executor-reflector-responder orchestration with a risk-tiered human-in-the-loop harness, evaluated across six orchestration paradigms and a 365-day agent-in-the-loop simulation against rule-based RPA, recurring enterprise work run as a governed agent loop.

Use Agentic ERP: Multi-Agent Large Language Model Architecture for Autonomous Enterprise Resource Planning to bound risk before recurring or unattended execution.

Research source arXiv:2607.17331; inspect its method and evaluation before treating results as production evidence.

[2607.17331] Agentic ERP: Multi-Agent Large Language Model Architecture for Autonomous Enterprise Resource Planning

Enterprise Resource Planning (ERP) systems record transactions reliably but still delegate almost all operational decision-making to human specialists, because classical rule-based automation cannot reason about exceptions and monolithic AI assistants degrade when asked to coordinate across functional boundaries. This paper presents Agentic ERP, an expert-system architecture that combines role-aligned large-language-model (LLM) agents with a risk-tiered human-in-the-loop harness and a graph-based orchestrator to execute end-to-end business workflows on a production ERP backend. First, autonomous ERP operation is formulated as a constrained sequential-decision problem over a structured enterprise state, with a decomposition argument linking role-aligned agents to a measurable reduction in per-step tool-selection complexity. Second, a graph-based Planner--Executor--Reflector--Responder orchestration decouples generation from evaluation through externalised grading criteria and sprint contracts, packaging recent harness-engineering principles as inspectable expert-system artefacts. Third, the system is evaluated at three levels: a scenario-based task suite, a comprehensive comparison of six orchestration paradigms on cross-functional crisis tasks, and a 365-day agent-in-the-loop simulation against rule-based RPA and no-intervention baselines. Across these levels the proposed multi-agent method is significantly better than the baseline, and the system sustains a simulated year of operation with zero stockouts while the rule-based baseline accumulates hundreds under the same demand stream. The work shows that role-aligned LLM agents under human oversight can move an ERP system from passively recording transactions to actively executing operational decisions, and it provides a reference architecture and an evaluation protocol for autonomous enterprise resource planning.

Zhihao Liu; Tianyu Wang; Xi Vincent Wang; Lihui Wang


Orchestration and control flow are made explicit and inspectable. Controlled comparison across six foundation models showing the harness, not model choice, dominates agent economics: a redesigned Writer Agent Harness cuts blended cost per task 41%, tokens 38%, and wall-clock 44%, arguing the harness is the one component whose efficiency multiplies across every model an org runs. Older than the window (Jul 8) but squarely core harness engineering and confirmed absent.

Use The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agents to bound risk before recurring or unattended execution.

Research source arXiv:2607.06906; inspect its method and evaluation before treating results as production evidence.

[2607.06906] The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI

Agentic AI development today runs on token maxing: buying capability with tokens -- longer reasoning traces, more turns, wider tool payloads, bigger replayed contexts -- so tokens per task grow faster than task value. Falling per-token prices mask the pattern; total spend rises anyway. We argue the decisive lever against token maxing is the harness: the orchestration layer that assembles context, exposes tools, sequences turns, delegates work, and carries enterprise observability and governance. We isolate it with a controlled swap: 22 locked evaluation tasks, six foundation models (Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, Palmyra X6), changing only the orchestration layer -- a frozen conventional production loop versus the Writer Agent Harness. Holding models constant, the harness cuts blended cost per task 41% ($0.21->$0.12), median wall-clock 44% (48s->27s), and tokens per task 38% (14.2k->8.8k), with task-completion quality at parity (0.78->0.81, directional at this sample size). Efficiency is model-invariant -- every model gets cheaper (33-61%) -- while quality gains are capability-dependent: a model's gain correlates almost perfectly with its baseline strength (r=0.99, n=6), a phenomenon we term harness leverage. Quality per dollar rises 82%; task-completions per million tokens rise from 54.9 to 92.0. On this workload the orchestration layer moved cost per task more than the full spread of the model menu did. We formalize token economics at the orchestration layer (including effective input price under prompt caching), detail the six mechanism families behind the effect -- cache-shape discipline to failure-spend governance -- compare six widely used agent systems on the same axes, and argue the harness is the one component whose efficiency multiplies across every model an organization runs -- present and future.

Muayad Sayed Ali; Aliaksandra Novik; Anji Boddupally; Artem Yavorskyi; Chris Nickerson; Daniel Rica; Emily DuGranrut; Felix Leung; Garrett Prince; Grace Barnett; Heath Robinson; Hosain Al Ahmad; Jesse Resnick; Juan Carlos Farah; Jyothi Swaroop Meruga; Leonid Kuznetsov; Luke Gorham; Marie Schmoll; Michael Paciullo; Saumya Das; Sharath Sheripally; Tommy Griscom; Mykyta Osadchyi; Neha Mantri; Nick Westrum; Olivia Benowitz; Parikshith Kulkarni; Radik Chernyshov; Rakshith Vasudev; Rohith Nadimpally; Vikas Gangadevi; Waseem AlShikh


claude-thermos

The work separates roles across agents, verifiers, or orchestration layers. Keeps a Claude Code session's prompt cache warm while the main agent waits on long-running subagents, so an orchestrator that blocks for more than a few minutes does not silently pay to rebuild its cache on every resume.

Use claude-thermos to bound risk before recurring or unattended execution.

Inspectable GitHub source (202 stars; 9 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - izeigerman/claude-thermos: Keeps your Claude session warm for you · GitHub

Keeps your Claude session warm for you. Contribute to izeigerman/claude-thermos development by creating an account on GitHub.


Worktrunk

Workspace isolation is part of the loop design, not an afterthought. CLI for Git worktree management built for running several coding agents in parallel, giving each agent an isolated checkout so concurrent work does not collide in a shared working directory.

Use Worktrunk to bound risk before recurring or unattended execution.

Inspectable GitHub source (6,329 stars; 230 forks; NOASSERTION license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - max-sixty/worktrunk: Worktrunk is a CLI for Git worktree management, designed for parallel AI agent workflows · GitHub

Worktrunk is a CLI for Git worktree management, designed for parallel AI agent workflows - max-sixty/worktrunk


Evaluation data is used as the feedback signal for improving loop behavior. Treats agent skills as software artifacts subject to single responsibility and low coupling, with concrete guidance on skill structure, loading, selection, and evaluation, plus comparison against other behavior-shaping mechanisms. The first disciplined treatment of a format most teams are currently authoring by feel.

Use Authoring Agent Skills: A Software-Engineering Approach to bound risk before recurring or unattended execution.

Research source arXiv:2607.25032; inspect its method and evaluation before treating results as production evidence.

[2607.25032] Authoring Agent Skills: A Software-Engineering Approach

Agent Skills are an emerging way to extend large language model agents with reusable procedural knowledge that the agent loads on demand. Anthropic introduced Agent Skills and published the format as an open specification supported across several agent tools. This note argues that a skill is a software artefact and that its construction should follow software-engineering principles, with qualifications: single responsibility, separation of interface from implementation, low coupling, and economy in a shared token budget, together with behavioural evaluation in place of deterministic testing. Using Claude Code as the reference implementation, it describes how a skill is structured, how its contents are loaded in stages, and how to write the description on which selection depends. It places skills against the other mechanisms a developer can use to shape agent behaviour, like project memory files, slash commands, subagents, external tool connections, and hooks, and gives a rule for choosing between them based on who decides that a mechanism runs and what guarantee it provides. It then sets out an evaluation-driven authoring process, a set of patterns and faults commonly encountered in authoring, and the trust question raised by using skills from third parties. We illustrate the comparison drawn in UML class style, the loading model, the anatomy of a skill, the relative position of each mechanism, and the points at which skills and hooks act during a session.


Verification is promoted from a final check to a loop-control signal. Benchmark of 106 issues across 49 repositories showing agents rarely retrieve contribution rules on their own, will disclose assistance and run verification steps when prompted, but consistently fail to refuse work in AI-banned projects. Concrete governance gap for anyone pointing unattended loops at public repos.

Use A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities to bound risk before recurring or unattended execution.

Research source arXiv:2607.26819; inspect its method and evaluation before treating results as production evidence.

[2607.26819] A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities

Open source communities have been flooded with AI-generated contributions. In defense, they have written contribution rules to regulate coding agents' behavior, spanning from a total ban, mandatory disclosure, to verification gates and human sign-offs. Yet, whether coding agents read and follow those rules, and behave in open source repositories, remains unknown. To estimate real-world rule compliance of coding agents, we curate 106 issues from 49 repositories containing AI contribution rules into RepoComplianceBench. We judge the trajectory of each run against the repository's rules, measuring whether the agent refuses to contribute, discloses its assistance truthfully, clears the required verification gates, or escalates critical steps to a human. We also test if extra prompts, rule disclosure, or feedback from the compliance verifier help with the situation. Our experiments on four frontier models show that today's agents almost never proactively retrieve the contribution rules. Agents pick up disclosure and verification with reminder prompts, rule quotes, and verifier feedback; however, they never refuse to contribute in AI-banned repositories under any condition we tested. The status reveals that verification and disclosure issues are solvable with existing mechanisms, yet enforcing bans and human escalations remains an open problem.


Context is managed as durable loop state rather than a single prompt payload. Documents how scientists adopting coding agents invent personal landmarking strategies to separate human-readable artifacts from agent context, and warns this de-standardization will fragment teams without explicit conventions. Practical prompt for codifying readability norms before agent-written code accumulates.

Use Who is scientific code for? Maintaining human-readable landmarks in agent-written code to bound risk before recurring or unattended execution.

Research source arXiv:2607.25975; inspect its method and evaluation before treating results as production evidence.

[2607.25975] Who is scientific code for? Maintaining human-readable landmarks in agent-written code

Scientific research involving code has long rested on the assumption that at least one person understands why the code exists. As scientists adopt coding agents, this assumption is breaking down. Drawing on an ongoing contextual inquiry of scientific programmers working with agentic tools (four cases to date), a survey of over 800 scientific programmers, and my own analysis workflows, this position piece describes how scientists are inventing personal conventions, "landmarking strategies", for marking which artifacts in a codebase are meant for human understanding and which are context for agents. They repurpose shared infrastructure (version control especially) in idiosyncratic ways, and I argue that this quiet de-standardization could complicate collaboration in teams with heterogeneous software practices. Alternatively, teams that explicitly delineate what is human-readable versus agent context will be better able to develop, document, and maintain scientific code.

Position piece submitted to Infrastructure @ CSCW 26 workshop


Translates agent-loop ideas into operator-facing workflows for repeated delegated work. Mike Wilkes turns the Hugging Face forensic timeline into an incident-response playbook organized around four decision points a team actually faces mid-incident, which is the operational complement to the technique-level writeups already in the list.

Use Four Incident-Response Lessons from the Hugging Face Breach to bound risk before recurring or unattended execution.

Contextual source from www.aikido.dev; useful for practice signals or boundary conditions, not independent validation.

Four incident-response lessons from the Hugging Face breach

Recon, stolen credentials, hidden C2, and rebuild-or-patch. Four Hugging Face breach decisions that show whether you can catch an attack in progress.


Orchestration and control flow are made explicit and inspectable. Sajjad Khan catalogs 63 confirmed production budget-overrun incidents drawn from 21 orchestration frameworks across 2023-2026, each with quoted maintainer or user evidence and documented dollar losses, organized into an eight-cluster failure taxonomy, the empirical base that the widely-circulated anecdotes (the $47K two-agent conversation loop, the $4,200/63-hour burn) individually lack.

Use Token Budgets: An Empirical Catalog of 63 LLM-Agent Budget-Overrun Incidents, with an Affine-Typed Rust Mitigation as a Case Study to bound risk before recurring or unattended execution.

Research source arXiv:2606.04056; inspect its method and evaluation before treating results as production evidence.

[2606.04056] Token Budgets: An Empirical Catalog of 63 LLM-Agent Budget-Overrun Incidents, with an Affine-Typed Rust Mitigation as a Case Study

LLM-agent budget overruns are a documented production failure class: a single retry loop can spend thousands of dollars before an operator notices, and the in-process integrity properties that would prevent it (no aliasing, no double-spend, no use-after-delegation of a cost-bearing value) are enforced, if at all, by ad-hoc wrappers rather than by the type system. Our central contribution is empirical: a catalog of 63 confirmed production incidents from 21 orchestration frameworks (2023-2026), each backed by a quoted GitHub issue and, where reported, a dollar loss, organized into an eight-cluster failure taxonomy (inter-rater Cohen's kappa = 0.837, N = 113), plus 47 supplementary structural entries. As one mitigation evaluated against this taxonomy, we build token-budgets, an 1,180-line Rust crate (no unsafe) that operationalizes affine ownership so that cloning, double-spending, or using a budget after delegating it are compile errors rather than runtime hazards an operator must remember to avoid. The dollar cap is runtime arithmetic under an estimator assumption; the affine layer makes that arithmetic non-bypassable. On single-agent workloads a 4-line Python counter matches the crate at 0/30 overshoot, so the distinguishing value is non-bypassability under operator error in multi-agent delegation: the delegation-fanout race documented in 11 incidents is rejected by the borrow checker at compile time, while the same pattern under asyncio overshoots 30/30 and three disciplined alternatives overshoot 0/30. Across five runtimes, three providers, and a temperature-stratified live-API test (N = 160), the approach reports zero cap violations and zero false refusals, at operational parity with concurrent work. Static over-reservation is 4-6x (2.11x adaptive). Binary-level cap-soundness on the running binary is left open.

26 pages. Artifact (catalog CSV, Rust crate, formal proofs): https://github.com/sajjadanwar0/token-budgets