Durable execution and replay are treated as first-class loop infrastructure. Explicitly diagnoses four Claude Code Agent Teams failure modes and builds a persistent workspace to fix them: irrecoverable agent teams (state lost when the terminal closes), compaction eroding working detail, agentic 'technical debt' trapped in compacted old chats, and heavy handoff prompt writing. This is loop engineering at the harness layer -- durable working state that survives process death and compaction, the exact gap between a session and a long-lived agent team.

Use Agent Team Work Zone: An Automated, Persistent Workspace for Long-Lived Coding Agent Teams to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.22917; inspect its method and evaluation before treating results as production evidence.

[2607.22917] Agent Team Work Zone: An Automated, Persistent Workspace for Long-Lived Claude Code Agent Teams

Large Language Model (LLM) agents have significantly improved coding and programming workflows. Claude Code, in particular, is one of the most powerful LLM coding agents and is capable of conducting complex coding tasks. However, several drawbacks can undermine long-term agentic workflows. (1) Irrecoverable agent teams: The Agent Teams feature is powerful, but the working state accumulated by each teammate is lost and cannot be resumed once the process stops, for example, when a terminal is closed. (2) Compaction erodes working detail: Compaction condenses the conversation into a summary, causing an agent's working details to become vague. (3) Agentic "technical debt": Over time, a user's decisions and the agents' operations become trapped in compacted old chats, making the project increasingly difficult to maintain and review. (4) Heavy prompt writing: Assigning or handing off tasks requires users to repeatedly write long prompts to achieve the expected agentic performance. We propose ATWZ (Agent Team Work Zone), a filesystem-based operations layer built around Claude Code's native Agent Teams that addresses these problems. Its central design principle is to treat each agent and teammate as a human employee and preserve their important working state in files stored in a dedicated directory called a "workstation," together with the skills, hooks, and scripts that use and maintain these files. With ATWZ, an agent team can periodically back up its working state, allowing an agent's knowledge to be recovered after compaction. After a process ends, the team can be restored with a single command. These features also substantially mitigate the agentic "technical debt" described above. Moreover, within ATWZ, agent "employees" can send documents to one another, greatly reducing the effort required to write prompts.


Uses real automated software-engineering systems as evidence for practical loop architectures. Reframes parallel coding agents as a pre-write admission problem instead of a merge-time repair problem. Each worker declares a versioned ChangeIntent (exact base commit, typed resources, dependencies, operations marked committed or contingent) and a deterministic control plane atomically admits compatible intents, constrains same-file parallelism to declared regions, serializes unresolved overlap, tracks dependency invalidation, and fails closed on ambiguous authority. The cleanest formalization yet of coordinating multiple concurrent coding agents on one repo.

Use Claim Plane: Enforceable Change Intents and Dynamic Scope for Parallel Coding Agents to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.21909; inspect its method and evaluation before treating results as production evidence.

Choose runtimes, tools, and delegation surfaces.

[2607.21909] Claim Plane: Enforceable Change Intents and Dynamic Scope for Parallel Coding Agents

Parallel coding agents can independently produce locally valid changes while still interfering at integration time, expanding beyond planned scope, or relying on premises invalidated by concurrent work. Existing responses emphasize communication, isolated workspaces, late merge-time repair, continuous supervision, or post-hoc runtime recovery. This paper presents Claim Plane, a model-agnostic coordination architecture that treats concurrent software change as a pre-write admission problem. Before implementation, each worker declares a versioned ChangeIntent containing an exact base commit, typed resources, dependencies, and operations marked as committed or contingent. A deterministic control plane atomically admits compatible intents, constrains same-file parallelism to declared regions, serializes unresolved overlap, tracks dependency invalidation, and fails closed on ambiguous authority. During execution, a contingent mutation does not reserve write ownership initially; the first attempted mutation triggers atomic scope promotion and re-admission against the current active set. Brokered execution binds capabilities to intent versions, leases, OS-level worktree locks, monotonic fencing tokens, and Git-tree provenance, while integration verifies immutable patches and evidence. A preliminary six-pair CooperBench mechanism check is reported only as feasibility evidence: static Claim Plane achieved 6/6 pair passes with full serialization, while dynamic scope retained parallel admission on half of the pairs, performed seven successful scope promotions, and failed closed on two undeclared mutations. The sample is intentionally too small for comparative claims. We argue that separating probabilistic planning from deterministic authority provides a foundation for a future learned semantic-dependency model and frontier-model escalation only on unresolved cases.


Uses real automated software-engineering systems as evidence for practical loop architectures. Plan Mode is now standard in agentic coding tools, but nobody has tested whether the transparency-and-control benefit survives outside developer contexts. A within-subjects study (N=24) with a spreadsheet Plan Mode prototype against a non-planning baseline finds task outcomes essentially unchanged, yet Plan Mode reduced refinement iterations and improved perceived creativity support and human-machine collaboration. Useful counterweight for teams assuming plan-then-execute always pays off in end-user environments where users work iteratively.

Use Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agents to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.23670; inspect its method and evaluation before treating results as production evidence.

[2607.23670] Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agents

Plan Modes have become standard features in agentic programming tools, allowing users to gain transparency and control by working with the agent to develop a plan before task execution. However, it remains unclear whether the benefits of this feature translate to end-user programming environments such as spreadsheets. Since spreadsheet programmers tend to work iteratively and care less about technical correctness, upfront planning may not fit into their workflows as easily. In this paper, we build a prototype of a Plan Mode for spreadsheet programming and evaluate it against a non-planning baseline through a within-subjects user study (N=24). We found that despite similar task outcomes with both tools, using Plan Mode led to a reduction in refinement and a better perception of the tool across dimensions of creativity support and human-machine collaboration. We discuss the implications of these results for the future design of Plan Modes, and for the broader role of human-AI planning in end-user programming.

Aayush Kumar; Avik Dutta; Sumit Gulwani; Gustavo Soares; Advait Sarkar; Emerson Murphy-Hill


Evaluation data is used as the feedback signal for improving loop behavior. Open harness for long-horizon multimodal production, built on the observation that real creative work is an evolving project state -- references, drafts, alternatives, edits, failed attempts, version relations, tool actions, evaluation signals, human feedback -- that prompt-based, chat-based, and node-based systems discard, linearize, or force users to wire manually. Commercial systems are moving this way but are closed, making it hard to study how agents represent and revise project state. Extends harness engineering into a domain where it is usually absent.

Use JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.23588; inspect its method and evaluation before treating results as production evidence.

[2607.23588] JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents

Creative AI is moving from single-step asset generation toward long-horizon multimodal production. Although recent generative models can synthesize high-quality images, videos, audio clips, UI elements, storyboards, slides, and other creative assets, real-world creative work requires more than isolated prompt-output interactions. It involves references, drafts, alternatives, edits, failed attempts, version relations, tool actions, evaluation signals, and human feedback, which together form an evolving project state. Existing prompt-based, chat-based, and node-based generation systems only partially support this state, as they often discard intermediate context, rely on linear conversations, or require manually specified workflows. Recent commercial systems indicate a shift toward agent-assisted creative production, but their closed architectures make it difficult to study how agents represent context, choose tools, revise artifacts, recover from failures, and maintain consistency over time. To address this gap, we introduce JarvisHub, a canvas-native creative agent harness for long-horizon multimodal creation. JarvisHub treats an editable canvas as the user workspace, the agent's external memory, action space, and shared project state, representing multimodal artifacts, dependencies, versions, and feedback as typed canvas nodes and links. Through a three-layer architecture of canvas state, protocol bridge, and agent runtime, JarvisHub enables agents to act within an inspectable and editable creative state. This design moves creative agents beyond isolated tool use toward sustained, human-steerable creative automation, where agents can progressively plan, generate, revise, and organize multimodal projects while users remain able to inspect, guide, and intervene throughout the process.

Yunlong Lin; Zixu Lin; Zhaohu Xing; Biqiang Li; Chenxin Li; Haonan Wang; Haitao Wu; Hengyu Liu; Jianghai Chen; Kaituo Feng; Kaixin Li; Shawn Chen; Shijue Huang; Sixiang Chen; Tsung-Yi Ho; Wenxuan Huang; Xiangyan Liu; Xiaomeng Hu; Xuanhua He; Yan Sun; Yunqing Zhao; Zhiqin Yang; Zehan Wang; Zhengyang Tang; Tianyu Pang; Xiangyu Yue

15 pages, 9 figures. Project page: https://www.jarvishub.site/ Code github: https://github.com/LYL1015/JarvisHub


The work turns loop quality into a measurable task or score. Coordinator-proposer architecture for bugs needing coordinated edits across multiple locations, combining tool-augmented analysis, fine-grained repair context construction, and iterative patch generation to reach state of the art on multi-hunk benchmarks. Targets the repair case single-agent loops most reliably fail.

Use MultiFixer: A Coordinator-Proposer Based Multi-Agent Framework For Fixing Multi-Hunk Bugs to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.26591; inspect its method and evaluation before treating results as production evidence.

[2607.26591] MultiFixer: A Coordinator-Proposer Based Multi-Agent Framework For Fixing Multi-Hunk Bugs

Automated Program Repair (APR) has benefited greatly from Large Language Models (LLMs), but existing LLM-based APR methods still struggle with multi-hunk bugs that require coordinated changes across multiple locations. These bugs demand repository-level context understanding, repair-order scheduling, and effective hunk-level patch generation and selection. To address these challenges, we propose MultiFixer, a novel Coordinator-Proposer based multi-agent framework for multi-hunk repair. MultiFixer performs tool-augmented bug analysis, constructs fine-grained repair context, iteratively generates patches through a Coordinator-Proposer architecture, and applies two-stage patch refinement for syntactic and semantic correctness. We evaluate MultiFixer on 835 bugs from Defects4J and three vulnerability benchmarks. On Defects4J, MultiFixer fixes 326 bugs, including 62 multi-method and 27 multi-file bugs, and outperforms prior APR baselines in the reported comparisons with the same base model. Moreover, MultiFixer also fixes 46 multi-hunk bugs among 95 unique fixes. When combined with Claude-3.5-Sonnet, MultiFixer repairs 420 bugs, establishing a new state of the art on Defects4J. On VUL4J, MultiFixer repairs 24 real-world vulnerabilities, including 5 multi-hunk cases. On the multi-hunk subsets of SEC-bench and PatchEval, MultiFixer fixes 11 and 19 vulnerabilities, respectively, outperforming all compared baselines under GPT-3.5. These results demonstrate the effectiveness of MultiFixer for multi-hunk repair.

Haichuan Hu; Chunrong Fang; Ye Shang; Jiawei Liu; Weifeng Sun; Guoqing Xie; Chenxing Zhong; Quanjun Zhang

Accepted to 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026)


ECC

Uses real automated software-engineering systems as evidence for practical loop architectures. Self-described agent harness operating system, packaging planning, execution, and optimization for coding agents behind one runtime with documentation in several languages. Very widely starred; treat adoption counts as a popularity signal rather than a quality judgement and read the source before relying on it.

Use ECC to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (238,453 stars; 36,209 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - affaan-m/ECC: The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond. · GitHub

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond. - affaan-m/ECC


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Argues for deterministic boundaries, contracts, and execution gates around probabilistic agent reasoning.

Use Why Agentic Systems Must Produce Deterministic Outputs to Scale to measure progress and gate completion with repeatable evidence.

Contextual source from streamzero.com; useful for practice signals or boundary conditions, not independent validation.

Gate progress with tests, evals, and evidence.

Agentic systems are gaining traction, but their inherent non-determinism poses a significant challenge for production environments. This document argues that deterministic outputs are essential for scaling agentic systems, enabling validation, security, and compliance in critical applications.


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Explains how to turn tests, linters, builds, traces, and other signals into feedback loops for coding agents.

Use Stop Babysitting Your Coding Agent. Give It Backpressure. to measure progress and gate completion with repeatable evidence.

Operational pattern or playbook; signal comes from reusable loop structure and practical transferability.

Backpressure is feedback that reaches the agent before the agent reaches the human.


The agent workflow includes explicit self-checking or gated completion. Uses hooks to enforce syntax, intent, and regression checks before an agent can finish.

Use How to Build a Self-Verification Loop in Claude Code to measure progress and gate completion with repeatable evidence.

How to Build a Self-Verification Loop in Claude Code (3 Layers, 20 Minutes) - DEV Community Navigation menu Search Search Close More... Copy link Enter fullscreen mode Exit fullscreen mode Enter fullscreen mode Exit fullscreen mode Enter fullscreen mode Exit fullscreen mode Enter fullscreen mode Exit fullscreen mode Enter fullscreen mode Exit fullscreen mode Enter fullscreen mode Exit fullscreen mode

Claude Code's Stop hook blocks the agent from finishing until verification passes. Combine it with... Tagged with ai, programming, productivity, claude.


Agentic Code Review

Verification is promoted from a final check to a loop-control signal. Addy Osmani argues that review, not code generation, is the bottleneck in agentic workflows, proposing risk-tiered verification depth, heterogeneous AI reviewers, and hard CI gates while warning against closed loops of models with correlated blind spots.

Use Agentic Code Review to measure progress and gate completion with repeatable evidence.

Contextual source from addyosmani.com; useful for practice signals or boundary conditions, not independent validation.

Coding agents are extraordinarily good now, and getting better fast. The interesting consequence is that the hard part of engineering moved from writing code...


Evaluation data is used as the feedback signal for improving loop behavior. Simon Willison wires a DSPy evaluation harness to a live Datasette instance with real tool calls and gold-standard metrics, then uses the eval traces to find and fix weaknesses in the agent's SQL system prompt.

Use Using DSPy to Evaluate and Improve Datasette Agent's SQL System Prompts to measure progress and gate completion with repeatable evidence.

Contextual source from simonwillison.net; useful for practice signals or boundary conditions, not independent validation.

Research: Using DSPy to evaluate and improve Datasette Agent's SQL system prompts

Leveraging the DSPy framework, this project evaluates and refines the core production system prompts used by Datasette Agent’s read-only SQL question answerer. The methodology involves a harness where DSPy agents …


Agentic coding notes

The work turns loop quality into a measurable task or score. Dan Luu's first-hand benchmarks and workflows arguing that systematic test infrastructure such as fuzzing and randomized testing, not human review, is what lets agent-generated code ship, and documenting why a self-contained agentic quality loop has so far eluded him.

Use Agentic coding notes to measure progress and gate completion with repeatable evidence.

Contextual source from danluu.com; useful for practice signals or boundary conditions, not independent validation.

Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island


Understanding Is the New Bottleneck

Verification is promoted from a final check to a loop-control signal. Geoffrey Litt argues that human understanding, not verification, is the real bottleneck in agent loops, warning that cognitive debt accrues when iterations outpace comprehension and proposing literate diffs, quizzes, and interactive micro-worlds as speed regulators.

Use Understanding Is the New Bottleneck to measure progress and gate completion with repeatable evidence.

Contextual source from www.geoffreylitt.com; useful for practice signals or boundary conditions, not independent validation.

Agents can write code faster than we can absorb it. Here's why it still matters for humans to understand what they build — and some techniques for doing that efficiently: explainer docs, quizzes, micro-worlds, and shared spaces.


Verifying Agentic Development at Scale

Verification is promoted from a final check to a loop-control signal. Cognition details the verification stack behind Devin sessions going majority-async: source-grounded test plans, deterministic reusable testing skills, and annotated video artifacts with pass/fail assertions so unattended runs return merge-ready results.

Use Verifying Agentic Development at Scale to measure progress and gate completion with repeatable evidence.

Contextual source from cognition.com; useful for practice signals or boundary conditions, not independent validation.

Verifying Agentic Development at Scale | Cognition

What we’ve learned building end-to-end testing capabilities in Devin’s virtual machine


Verification is promoted from a final check to a loop-control signal. Sonar formalizes a two-tier verification gate for agent loops, pairing a probabilistic LLM verifier sub-agent for intent with a deterministic analysis gate as the hard halt, arguing that LLM-only verification amounts to two optimists agreeing.

Use Loop Engineering Without Verification Is Just Automation to measure progress and gate completion with repeatable evidence.

Contextual source from www.sonarsource.com; useful for practice signals or boundary conditions, not independent validation.

Loop engineering without verification is just automation | Sonar

Explore how LLM reviewers and deterministic checks work together to keep coding agent loops from shipping unfinished code.


Verification is promoted from a final check to a loop-control signal. Datadog engineers' case for harness-first engineering once agents write code faster than humans can review, using deterministic simulation testing across millions of seeds as the verification gate.

Use Closing the Verification Loop: Observability-Driven Harnesses to measure progress and gate completion with repeatable evidence.

Contextual source from www.datadoghq.com; useful for practice signals or boundary conditions, not independent validation.

Closing the verification loop: Observability-driven harnesses for building with agents | Datadog security-platform rum ci dashboard host-map apm security-platform rum ci dashboard host-map apm security-platform rum ci dashboard Icon/world

Learn how Datadog verifies AI-generated systems at scale using deterministic testing, formal methods, and observability-driven feedback loops.

Alp Keles, Jai Menon, Sesh Nalla, Vyom Shah


Evaluation data is used as the feedback signal for improving loop behavior. Trace-evaluate-debug-refine loop for improving agent behavior from real runs.

Use How to build a better agent harness with traces and evals to measure progress and gate completion with repeatable evidence.

Contextual source from arize.com; useful for practice signals or boundary conditions, not independent validation.

How to build a better agent harness with traces and evals - Arize AI

Agents are easy to prototype and hard to improve. A repeatable loop of traces, evals, failed-span inspection, and targeted harness changes makes agent behavior easier to debug and improve.


Evaluation data is used as the feedback signal for improving loop behavior. LangChain's recipe for using evals as the learning signal for harness improvement.

Use Better Harness: A Recipe for Harness Hill-Climbing with Evals to measure progress and gate completion with repeatable evidence.

Contextual source from www.langchain.com; useful for practice signals or boundary conditions, not independent validation.

We can build better agents by building better harnesses. But to autonomously build a “better” harness, we need a strong learning signal to “hill-climb” on. We share how we use evals as that signal, plus design decisions that help our agent generalize instead of overfit. Better-Harness is a system for iteratively sourcing and improving your harness with evals.


The agent workflow includes explicit self-checking or gated completion. Practical discussion of self-verification, traces, middleware, and loop detection for coding agents.

Use Improving Deep Agents with harness engineering to measure progress and gate completion with repeatable evidence.

Harness engineering improved LangChain's coding agent from Top 30 to Top 5 on Terminal Bench using self-verification, tracing, and context optimization.


Verification is promoted from a final check to a loop-control signal. Closed loop that turns each harness edit into a falsifiable contract verified against trajectory outcomes, so the harness evolves from observability rather than trial and error.

Use Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses to measure progress and gate completion with repeatable evidence.

Research source arXiv:2604.25850; inspect its method and evaluation before treating results as production evidence.

[2604.25850] Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE), a closed loop that addresses these challenges through three matched observability pillars: (1) component observability gives every editable harness component a file-level representation so the action space is explicit and revertible; (2) experience observability distills millions of raw trajectory tokens into a layered, drill-down evidence corpus that an evolving agent can actually consume; and (3) decision observability pairs every edit with a self-declared prediction, later verified against the next round's task-level outcomes. Together, these pillars turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error. Empirically, ten AHE iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI (71.9%) and the self-evolving baselines ACE and TF-GRPO. The frozen harness transfers without re-evolution: on SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains across three alternate model families, indicating the evolved components encode general engineering experience rather than benchmark-specific tuning. Ablations localize the gain to tools, middleware, and long-term memory rather than the system prompt, suggesting factual harness structure transfers while prose-level strategy does not.

Jiahang Lin; Shichun Liu; Chengjun Pan; Lizhi Lin; Shihan Dou; Zhiheng Xi; Xuanjing Huang; Hang Yan; Zhenhua Han; Tao Gui; Yu-Gang Jiang


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Optimizes the surrounding harness (tools, prompts, control flow) end to end against task outcomes, turning harness tuning into a measurable improvement loop instead of manual trial and error.

Use Meta-Harness: End-to-End Optimization of Model Harnesses to measure progress and gate completion with repeatable evidence.

Research source arXiv:2603.28052; inspect its method and evaluation before treating results as production evidence.

[2603.28052] Meta-Harness: End-to-End Optimization of Model Harnesses

The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.

Yoonho Lee; Roshen Nair; Qizheng Zhang; Kangwook Lee; Omar Khattab; Chelsea Finn


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Analyzes production agent traces to find harness-level failure modes, hands its report to a coding agent to apply fixes, and repeats the collect-analyze-fix-redeploy cycle, reporting AppWorld gains from harness changes alone.

Use HALO (Hierarchical Agent Loop Optimizer) to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (1,134 stars; 87 forks; updated 2026-08-06); popularity is context, not proof of reliability.

GitHub - context-labs/HALO: Hierarchal Agent Loop Optimizer · GitHub

Hierarchal Agent Loop Optimizer. Contribute to context-labs/HALO development by creating an account on GitHub.


Evaluation data is used as the feedback signal for improving loop behavior. Agentic RL framework in which one model both solves tasks and edits its own harness, including repairing faulty evaluation code, co-evolving weights, harness, and solutions so a trained Qwen3-8B matches a much larger baseline.

Use Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.03935; inspect its method and evaluation before treating results as production evidence.

[2607.03935] Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions

Self-evolving frameworks usually optimize task solutions while treating the surrounding harness as fixed. We introduce Harness-Aware Self-Evolving (HASE), an agentic reinforcement-learning framework in which a single model can generate task solutions or edit selected harness components in a multi-turn action space. HASE enables a single Qwen3-8B model to match the text-classification performance of a GPT-OSS-120B model that uses Claude Code as the harness proposer. In alpha factor mining, HASE outperforms the reported GPT-OSS-120B baseline. HASE also repairs imperfect evaluation components and converges to state-of-the-art performance in circle-packing algorithm discovery. These results show that HASE improves the harness and the solution through one unified agentic process.

Haochen Luo; Yi Huang; Sichun Luo; Fengyuan Liu; Lei Li; Zefa Hu; Junlan Feng; Qi Liu


auto-harness

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Bring-your-own-agent framework for self-improving agentic systems that mines failures from runs, optimizes the harness in response, and gates every change behind regression checks.

Use auto-harness to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (529 stars; 60 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - neosigmaai/auto-harness: Bring your own agent and build a self-improving agentic system. Automatically mine failures, optimize the agent harness, and gate against regressions. · GitHub

Bring your own agent and build a self-improving agentic system. Automatically mine failures, optimize the agent harness, and gate against regressions. - neosigmaai/auto-harness


OpenAI agent evals

Evaluation data is used as the feedback signal for improving loop behavior. Evaluation guidance for moving from traces to repeatable grading of agent workflows.

Use OpenAI agent evals to measure progress and gate completion with repeatable evidence.

Primary official documentation from developers.openai.com; use it for current product or standard behavior.

Learn how to evaluate agent workflows with traces, graders, datasets, and evaluation runs on the OpenAI platform.


Promptfoo OpenAI Agents provider

Execution isolation and permission boundaries are part of the design. Testing and assertions for multi-turn agent workflows, tools, state, handoffs, sandboxes, and traces.

Use Promptfoo OpenAI Agents provider to measure progress and gate completion with repeatable evidence.

Working implementation, framework, runtime, or repository; signal comes from usable code and ecosystem adoption.

Test OpenAI Agents with tools, handoffs, sessions, sandbox workflows, and tracing in promptfoo.


Inspect AI

Evaluation data is used as the feedback signal for improving loop behavior. UK AISI evaluation framework with solvers, scorers, sandboxing, tool use, MCP, and log viewing.

Use Inspect AI to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (2,492 stars; 632 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - UKGovernmentBEIS/inspect_ai: Inspect: A framework for large language model evaluations · GitHub

Inspect: A framework for large language model evaluations - UKGovernmentBEIS/inspect_ai


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Portable tracing conventions for model calls, tool calls, and agent workflows.

Use OpenTelemetry Semantic Conventions for Generative AI Systems to measure progress and gate completion with repeatable evidence.

Primary official documentation from opentelemetry.io; use it for current product or standard behavior.

Moved: Generative AI semantic conventions | OpenTelemetry The OpenTelemetry Logo

Important GenAI semantic conventions have moved to the OpenTelemetry GenAI semantic conventions repository. This page has moved and is no longer maintained in this repository.


AgentOps

Durable execution and replay are treated as first-class loop infrastructure. Monitoring, replay, cost tracking, benchmarking, and tracing for agent sessions.

Use AgentOps to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (5,757 stars; 613 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - AgentOps-AI/agentops: Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI · GitHub

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI - AgentOps-AI/agentops


Langfuse

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Open-source LLM engineering platform with tracing, evaluations, and metrics that loops can read back as feedback signals.

Use Langfuse to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (32,689 stars; 3,505 forks; NOASSERTION license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - langfuse/langfuse: 🪢 Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. Integrates with OpenTelemetry, LangChain, OpenAI SDK, LiteLLM, and more. 🍊YC W23 · GitHub

🪢 Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. Integrates with OpenTelemetry, LangChain, OpenAI SDK, LiteLLM, and more. 🍊YC W23 - GitHub - langfuse/langfuse: 🪢 Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. Integrates with OpenTelemetry, LangChain, OpenAI SDK, LiteLLM, and more. 🍊YC W23


LangSmith

Evaluation data is used as the feedback signal for improving loop behavior. Tracing, evaluation, and monitoring platform for inspecting and grading agent runs across iterations.

Use LangSmith to measure progress and gate completion with repeatable evidence.

LangSmith: Agent & LLM Observability Platform

Complete AI agent and LLM observability platform with tracing and real-time monitoring. Debug agents, find failures fast, and track costs and latency.


Arize Phoenix

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Open-source AI observability for tracing, evaluating, and debugging agent behavior from real runs.

Use Arize Phoenix to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (10,929 stars; 1,043 forks; NOASSERTION license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - Arize-ai/phoenix: AI Observability & Evaluation · GitHub

AI Observability & Evaluation. Contribute to Arize-ai/phoenix development by creating an account on GitHub.


Braintrust

Evaluation data is used as the feedback signal for improving loop behavior. Evaluation and observability platform with experiments, datasets, and CI integration for gating agent changes.

Use Braintrust to measure progress and gate completion with repeatable evidence.

Braintrust - The active observability platform for agents

Ship quality agents at scale. Braintrust is the AI observability platform for tracing production, running evals, and catching regressions before they reach users.


Weave

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Weights & Biases toolkit for tracing, evaluating, and monitoring agent applications over time.

Use Weave to measure progress and gate completion with repeatable evidence.

W&B Weave - Weights & Biases Documentation

Track, test, and improve language model apps with W&B Weave


agentops (boshu2)

Verification is promoted from a final check to a loop-control signal. Independent verification layer for coding agents where a change only counts as done after a different model or a real test checks it, with the verdict recorded in the repo via a tamper-evident ledger.

Use agentops (boshu2) to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (416 stars; 40 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - boshu2/agentops: The operating loop a coding agent follows — and skills to orchestrate multi-agent systems. · GitHub

The operating loop a coding agent follows — and skills to orchestrate multi-agent systems. - boshu2/agentops


SkillSpec

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. CLI that makes agent skills followable, testable, and provable by converting prose skills into structured contracts, scoring follow-through risk, and generating execution traces of which steps ran, were skipped, and what evidence exists.

Use SkillSpec to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (820 stars; 59 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - modiqo/skillspec: SkillSpec makes agent skills followable, testable, and provable with Doctor risk reports, guided imports, structured contracts, and alignment proof. · GitHub

SkillSpec makes agent skills followable, testable, and provable with Doctor risk reports, guided imports, structured contracts, and alignment proof. - modiqo/skillspec


Shepherd

Durable execution and replay are treated as first-class loop infrastructure. Python runtime that records agent execution as reversible, Git-like traces so meta-agents or humans can observe, fork, replay, and revert any run before results touch files, with copy-on-write forking, roughly 95% cache reuse on replay, and syscall-level permission enforcement.

Use Shepherd to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (1,707 stars; 135 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - shepherd-agents/shepherd: A runtime substrate that turns an agent's execution into a reversible, Git-like trace, so meta-agents can observe, fork, replay, and revert any run. Couples agent and environments in a copy-on-write fork ~5x faster than docker commit, with ~95% KV-cache reuse on replay. Framework built for meta-agents to supervise, optimize, and train other agents · GitHub

A runtime substrate that turns an agent's execution into a reversible, Git-like trace, so meta-agents can observe, fork, replay, and revert any run. Couples agent and environments in a copy-on-write fork ~5x faster than docker commit, with ~95% KV-cache reuse on replay. Framework built for meta-agents to supervise, optimize, and train other agents - shepherd-agents/shepherd


grill-for-unknowns

Verification is promoted from a final check to a loop-control signal. Portable SKILL.md agent skill that gates long-running subagent and coding-agent launches behind plan interrogation: it inspects the real territory (docs, source, tests, config) first, sorts what's known into facts, decisions, domain language, and unknowns across known/unknown quadrants, then emits a launch packet with assumptions, verification steps, and rollback risks before dispatch.

Use grill-for-unknowns to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (195 stars; 7 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - nicobailon/grill-for-unknowns: Agent skill for finding unknowns, grilling plans, and reaching shared understanding before implementation · GitHub

Agent skill for finding unknowns, grilling plans, and reaching shared understanding before implementation - nicobailon/grill-for-unknowns


Fable Harness

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Drop-in behavior protocol kit (hooks, a skill, and sub-agents auto-injected into every Claude Code session) enforcing a verify-first process: gather evidence before answering and verify changes before declaring done.

Use Fable Harness to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (200 stars; 35 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - Miguok/fable-harness: Make Claude Code work like a disciplined engineer: OODA, multi-party adversarial review, tiered model routing, fail-then-pass — token-efficient by design (route heavy work to smaller models, isolate sub-agent context). Distilled from Fable to reinforce the Opus harness. · GitHub

Make Claude Code work like a disciplined engineer: OODA, multi-party adversarial review, tiered model routing, fail-then-pass — token-efficient by design (route heavy work to smaller models, isolate sub-agent context). Distilled from Fable to reinforce the Opus harness. - Miguok/fable-harness


Mindwalk

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Local visualization tool that replays Claude Code and Codex session logs as light moving across a 3D map of the repository, making long agent runs inspectable after the fact.

Use Mindwalk to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (1,150 stars; 89 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - cosmtrek/mindwalk: A visualization tool that replays coding-agent sessions on a 3D map of your codebase. · GitHub

A visualization tool that replays coding-agent sessions on a 3D map of your codebase. - cosmtrek/mindwalk


Waggle

Context is managed as durable loop state rather than a single prompt payload. MCP-native reference layer for agent handoffs: instead of pasting full context between agents, it passes a compact attributed, resolvable reference token that the receiving agent expands on demand.

Use Waggle to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (754 stars; 97 forks; Apache-2.0 license; updated 2026-08-05); popularity is context, not proof of reliability.

GitHub - modiqo/waggle: Attributed, resolvable artifact references for agent handoffs — a ~30-byte token instead of pasted context. MCP-native; the reference layer for the agent-harness world. · GitHub

Attributed, resolvable artifact references for agent handoffs — a ~30-byte token instead of pasted context. MCP-native; the reference layer for the agent-harness world. - modiqo/waggle


Jacquard

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Research language designed around the machine-writes, human-verifies contract, using effect-typed signatures so an agent's generated code carries checkable declarations of what it is allowed to touch.

Use Jacquard to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (112 stars; 3 forks; Apache-2.0 license; updated 2026-08-02); popularity is context, not proof of reliability.

GitHub - jbwinters/jacquard-lang: Jacquard is a small programming language designed for a regime in which most code is written by machine-learning models and reviewed by people. · GitHub

Jacquard is a small programming language designed for a regime in which most code is written by machine-learning models and reviewed by people. - jbwinters/jacquard-lang


Verification is promoted from a final check to a loop-control signal. Pairs a coding agent with a theorem prover (AutoRocq) in a generate-and-validate loop, turning formal proof into the exit gate for trusted automatic programming.

Use Agentic Verification of Software Systems to measure progress and gate completion with repeatable evidence.

Research source arXiv:2511.17330; inspect its method and evaluation before treating results as production evidence.

[2511.17330] Agentic Verification of Software Systems

Automatically generated code is gaining traction recently, owing to the prevalence of Large Language Models (LLMs). Further, the AlphaProof initiative has demonstrated the possibility of using AI for general mathematical reasoning. Reasoning about computer programs (software) can be accomplished via general mathematical reasoning; however, it tends to be more structured and richer in contexts. This forms an attractive proposition, since then AI agents can be used to reason about voluminous code that gets generated by AI. In this work, we present a first LLM agent, AutoRocq, for conducting program verification. Unlike past works, which rely on extensive training of LLMs on proof examples, our agent learns on-the-fly and improves the proof via an iterative refinement loop. The iterative improvement of the proof is achieved by the proof agent communicating with the Rocq (formerly Coq) theorem prover to get additional context and feedback. The final result of the iteration is a proof derivation checked by the Rocq theorem prover. In this way, our proof construction involves autonomous collaboration between the proof agent and the theorem prover. This autonomy facilitates the search for proofs and decision-making in deciding on the structure of the proof tree. Experimental evaluation on SV-COMP benchmarks and on Linux kernel modules shows promising efficacy in achieving automated program verification. As automation in code generation becomes more widespread, we posit that our proof agent can be potentially integrated with AI coding agents to achieve a generate and validate loop, thus moving closer to the vision of trusted automatic programming.

Haoxin Tu; Huan Zhao; Yahui Song; Mehtab Zafar; Ruijie Meng; Abhik Roychoudhury

Published in Proceedings of the ACM on Software Engineering 3 (FSE); the linked arXiv record remains available for open access.


Orchestration and control flow are made explicit and inspectable. Treats execution traces as the assurance substrate, pairing machine-checkable contracts, testing, and governance so recurring agent orchestration stays verifiable and auditable.

Use A Trace-Based Assurance Framework for Agentic AI Orchestration: Contracts, Testing, and Governance to measure progress and gate completion with repeatable evidence.

Research source arXiv:2603.18096; inspect its method and evaluation before treating results as production evidence.

[2603.18096] A Trace-Based Assurance Framework for Agentic AI Orchestration: Contracts, Testing, and Governance

In Agentic AI, Large Language Models (LLMs) are increasingly used in the orchestration layer to coordinate multiple agents and to interact with external services, retrieval components, and shared memory. In this setting, failures are not limited to incorrect final outputs. They also arise from long-horizon interaction, stochastic decisions, and external side effects (such as API calls, database writes, and message sends). Common failures include non-termination, role drift, propagation of unsupported claims, and attacks via untrusted context or external channels. This paper presents an assurance framework for such Agentic AI systems. Executions are instrumented as Message-Action Traces (MAT) with explicit step and trace contracts. Contracts provide machine-checkable verdicts, localize the first violating step, and support deterministic replay. The framework includes stress testing, formulated as a budgeted counterexample search over bounded perturbations. It also supports structured fault injection at service, retrieval, and memory boundaries to assess containment under realistic operational faults and degraded conditions. Finally, governance is treated as a runtime component, enforcing per-agent capability limits and action mediation (allow, rewrite, block) at the language-to-action boundary. To support comparative evaluations across stochastic seeds, models, and orchestration configurations, the paper defines trace-based metrics for task success, termination reliability, contract compliance, factuality indicators, containment rate, and governance outcome distributions. More broadly, the framework is intended as a common abstraction to support testing and evaluation of multi-agent LLM systems, and to facilitate reproducible comparison across orchestration designs and configurations.

Ciprian Paduraru; Petru-Liviu Bouruc; Alin Stefanescu

Published in Proceedings of the 21st International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE); the linked arXiv record remains available for open access.


Verification is promoted from a final check to a loop-control signal. Confines self-modification to a small steering adapter around a frozen base model and gates each change with anytime-valid statistical tests that emit auditable certificates, reporting solve-count gains and logged regression prevention on a SWE-bench Verified subset.

Use Self-Evolving Agents with Anytime-Valid Certificates to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.00871; inspect its method and evaluation before treating results as production evidence.

[2607.00871] Self-Evolving Agents with Anytime-Valid Certificates

Self-evolving agents violate the assumption behind most learning-theoretic guarantees: the data, evaluator, components, and hypothesis space are produced by the policy being updated. We present \textbf{SEA}, an architecture that confines self-modification to a small steering adapter and a versioned harness around a \emph{frozen} base model and admits each modification only through an anytime-valid gate that emits an auditable certificate against a fixed error budget. Five loop controllers compose published guarantees; because such gates can only \emph{select} among behaviors the frozen base already produces, five verifier-in-the-loop mechanisms -- best-of-$N$, micro-step search, self-authored reproduction oracles, search-layer control, and self-repair -- supply the dense, grader-free signal the gates require, computed from the issue text alone. On a $52$-instance SWE-bench Verified subset across four base models, base capability is the dominant, confound-free effect, and on two strong base models a deliberate no-op-composite control isolates the suite's contribution at $+4$ and $+5$ (\textsc{Glm}~5.2 $24\to28$; \textsc{Gpt} $29\to34$, the $65\%$ best), with event logs confirming that its mechanisms fire and prevent regressions. Results are single-run on expensive evaluations; confirming run-to-run variance and adapting the per-task algorithm mix are future work.


Verification is promoted from a final check to a loop-control signal. Models verifier-corrector loops in multi-agent LLM systems as delayed consensus, deriving a stability threshold where verification that is too strong or too late turns factual consensus into oscillation, plus a greedy corrector-placement algorithm validated on five open models.

Use Delayed Verification Destabilizes Multi-Agent LLM Belief to measure progress and gate completion with repeatable evidence.

Research source arXiv:2606.27409; inspect its method and evaluation before treating results as production evidence.

[2606.27409] Delayed Verification Destabilizes Multi-Agent LLM Belief: Instability Thresholds and Optimal Corrector Placement

Multi-agent large language model (LLM) systems often rely on verifier and critic agents to suppress hallucinations, but verification is delayed. During this delay, false claims can propagate through the agent network. We model this process as delayed consensus on a graph with grounded corrector nodes. Spectral decomposition by the grounded Laplacian yields a closed-form stability threshold for the verification dose: correction that is too strong or too delayed can turn consensus into oscillation. The most unstable regime occurs when the communication and verification delays coincide; for delay two, the threshold is the inverse golden ratio. The same framework gives a supermodular placement objective and a greedy (1-1/e)-approximation rule for assigning a limited corrector budget to influential nodes. Experiments across five open models confirm the predicted dose-delay oscillations. By contrast, grounded factual answering makes truth an absorbing boundary and eliminates the effect, suggesting that the instability is specific to signed-belief tasks while grounded verification remains stabilizing

20 pages, 5 figures, 1 table. Code and data: https://github.com/YehudaItkin/delayed-verification-llm


Verification is promoted from a final check to a loop-control signal. Models agent workflows and trajectories in Lean 4 dependent types so semantic consistency is machine-checked rather than judged by an LLM, with verification-passing workflows outperforming failing ones by an average of 11.94% on software-engineering benchmarks.

Use Lean4Agent: Formal Modeling and Verification for Agent Workflow and Trajectory to measure progress and gate completion with repeatable evidence.

Research source arXiv:2606.06523; inspect its method and evaluation before treating results as production evidence.

[2606.06523] Lean4Agent: Formal Modeling and Verification for Agent Workflow and Trajectory

Equipping Large Language Models (LLMs) to execute reliable multi-step workflows has become a central challenge in artificial intelligence. Despite recent advances in LLMs' agentic capabilities, most agent systems still lack formal methods for specifying, verifying, and debugging their workflow and execution trajectories. This challenge mirrors a long-standing problem in mathematics, where the ambiguity of natural languages (NLs) motivates the development of formal languages (FLs). Inspired by this paradigm, we propose **Lean4Agent**, to the best of our knowledge, the first framework that uses Lean4, a dependent-type FL to model and verify agent behavior. **Lean4Agent** launches **FormalAgentLib**, an extensible Lean4 library for formally modeling and verifying agent workflows' semantic consistency under explicit assumptions, and enabling localization of execution-time failures revealed by trajectories. Building on **FormalAgentLib**, we further develop **LeanEvolve**, which applies results in **FormalAgentLib** to revise workflows to enhance its capability. Extensive experiments on a hard problem subset of SWE-Bench-Verified and a subset of ELAIP-Bench across 5 leading LLMs indicate that the verification-passing workflows outperform the failing ones by an average of **11.94%**, and **LeanEvolve** further improves SWE performance by **7.47%** on average. Furthermore, **Lean4Agent** establishes a foundation for a new field of using expressive dependent-type FL to formally model and verify agent behavior.

Ruida Wang; Jerry Huang; Pengcheng Wang; Xuanqing Liu; Luyang Kong; Tong Zhang


Evaluation data is used as the feedback signal for improving loop behavior. Event-sourced agent runtime whose self-improvement loop gates every proposed repair behind static checks, sandbox execution, and held-out evaluation before adoption, keeping the full decision trail replayable.

Use Regimes: An Auditable, Held-Out-Gated Improvement Loop to measure progress and gate completion with repeatable evidence.

Research source arXiv:2606.10241; inspect its method and evaluation before treating results as production evidence.

[2606.10241] Regimes: An Auditable, Held-Out-Gated Improvement Loop Demonstrated on LongMemEval with ActiveGraph

Autonomous improvement loops are hard to trust because the improvement process is usually external scaffolding bolted onto the agent: failures go unlogged, diagnoses cannot be replayed, and promote-or-discard decisions land in a side database rather than the agent's own history. We show that an event-sourced agent runtime removes that friction and turns controlled improvement into a first-class workflow. When the agent's state is a deterministic projection of an append-only event log, failures are recorded, a run replays exactly from its log, candidate patches scope to typed pipeline seams, gates are auditable, and every promotion or discard is itself an event. We demonstrate this with Regimes, a loop on the ActiveGraph runtime that diagnoses failed evaluations, proposes a repair at a pipeline point, and promotes it only after static checks, sandbox execution, in-sample evaluation, and held-out validation. The loop is target-agnostic: the same control flow runs against different tasks through a common interface. On LongMemEval-S the dominant failure is not retrieval but reconciliation: the evidence is already in the assembled context, yet the reader answers incorrectly. Across five seeded held-out splits, Regimes discovers reader-prompt repairs that improve final held-out accuracy by +0.05 to +0.10 in four splits and +0.01 in one over-promotion split; two splits are individually significant (seed 5 unadjusted for its sequential promotion structure), and the pooled count is descriptive only, since the splits share one 500-question pool. The durable contributions are ActiveGraph as an auditable substrate that makes controlled improvement loops tractable, the held-out-gated loop it supports, the failure-regime taxonomy routing each failure to a pipeline location (whose marginal value over an unrouted baseline is the primary open question), and the prompt-as-discovery-probe hypothesis.

30 pages, 5 figures. Code and committed runs: https://github.com/yoheinakajima/regimes


Evaluation data is used as the feedback signal for improving loop behavior. Automated evaluation framework from IBM Research that grades agent behavior at system, trace, and node granularity without predefined error taxonomies, producing feedback aligned with human-annotated errors and predictive of task success.

Use Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2605.22608; inspect its method and evaluation before treating results as production evidence.

[2605.22608] Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

Agentic systems are becoming more capable: agents define strategies, take actions, and interact with different environments. This autonomy poses serious challenges for overseeing and assessing agent behavior. Most current tools are limited, focusing on observability with basic evaluation capabilities or imposing static, hand-crafted error taxonomies that cannot adapt to new domains. To address this gap, we present Agentic CLEAR, an automatic, dynamic, and easy-to-use evaluation framework. It produces textual insights into the agent behavior on three levels of granularity: system, trace, and node. Agentic CLEAR operates above the observability layer, enabling seamless integration and featuring an intuitive UI that makes agent evaluation highly accessible. In our experiments on four benchmarks, seven agentic settings, and tens of thousands of LLM calls, we show that Agentic CLEAR produces high-quality, data-driven, insightful feedback. Our analysis shows strong alignment with human-annotated errors and the ability to predict task success rate.

Asaf Yehudai; Lilach Eden; Michal Shmueli-Scheuer

Published in Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL); the linked arXiv record remains available for open access.


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. FlowFixer converts runs of platform-built agentic workflows (Dify, Coze, n8n) into symbolic traces, infers correctness specs and node dependencies to localize root-cause failures, and generates targeted repairs at a 71.3% success rate.

Use Diagnosis-Driven Automatic Repair for Agentic Workflow via Symbolic Inference to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.02882; inspect its method and evaluation before treating results as production evidence.

[2607.02882] Diagnosis-Driven Automatic Repair for Agentic Workflow via Symbolic Inference

Platform-orchestrated agentic workflows have become a popular paradigm for developing LLM-based applications. However, their reliability remains a major challenge due to the uncertainty of LLM outputs, complex inter-node dependencies, and heterogeneous tool interactions. Existing agentic workflow optimization and agent enhancement methods primarily rely on trajectory-level feedback. Without explicitly identifying the underlying failure root causes, their resulting repair plans are often insufficiently targeted. We propose FlowFixer, a diagnosis-driven automated repair framework for agentic workflows. FlowFixer first transforms workflow executions into unified symbolic traces and performs symbolic inference to derive executable behavioral specifications that capture node correctness, temporal dependencies, and causal relationships. Based on specification verification, it conducts failure attribution and root cause analysis, and then generates targeted repair patches. To reduce verification costs, FlowFixer further employs a multi-dimensional pre-execution assessment to filter infeasible repairs before dynamic verification. We evaluate FlowFixer on workflow failures collected from three popular development platforms: Dify, Coze and n8n. Results show that FlowFixer achieves a repair success rate of 71.3%, outperforming state-of-the-art baselines by 11.9% to 27.6%. It also improves failure attribution accuracy by 4.8% to 33.1% and root cause analysis accuracy by 15.3% to 38.8%. This work offers a new perspective on reliable diagnosis and repair of agentic workflows through symbolic modeling and inference.

Xuyan Ma; Yawen Wang; Junjie Wang; Xiaofei Xie; Boyu Wu; Mingyang Li; Dandan Wang; Qing Wang


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Self-evolving rubric framework that scores agent trajectories on skill selection, following, composition, and reflection, exposing failures that pass/fail outcome checks miss and beating outcome-only filtering as a training signal.

Use SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.01874; inspect its method and evaluation before treating results as production evidence.

[2607.01874] SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use

Skills are becoming a reusable operational layer for LLM agents, encoding SOPs, domain rules, tool workflows, scripts, and validation routines. In realistic skill repositories, overlapping skills make reliable skill-use difficult. Final verifier success is too coarse for both evaluation and training, since an agent may pass through trial and error while selecting distractor skills, skipping required steps, composing workflows incorrectly or omitting final checks. We introduce SkillCoach, a self-evolving rubric framework for evaluating and enhancing agentic skill-use. SkillCoach derives skill-grounded process rubrics from real rollouts and evaluates trajectories along four dimensions: skill selection, skill following, skill composition, and skill-grounded reflection. It keeps the external verifier as a separate outcome signal, allowing process quality to be distinguished from accidental task success. The evolved rubrics further serve as process supervision for selecting high-quality training trajectories. Experiments show that evolved rubrics substantially improve evaluation quality, expose failures hidden by final accuracy, and provide stronger supervision signals than outcome-only filtering for enhancing agentic skill-use.

Jiayin Zhu; Kelong Mao; Yudong Guo; Dengbo He; Sulong Xu; Simiu Gu; Yutao Yue


Verification is promoted from a final check to a loop-control signal. Shows that naively feeding bug-reproduction tests to software-engineering agents can mislead them, and instead pipes runtime diagnosis from multi-faceted reproduction tests into patch generation, reaching 75.7% on SWE-bench Verified.

Use SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Bug Reproduction Tests to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.00990; inspect its method and evaluation before treating results as production evidence.

[2607.00990] SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests

Large language model (LLM)-based software engineering agents are increasingly developed to resolve software issues by generating patches from issue reports and code repositories. Bug reproduction tests (BRTs) are an important building block for such agents and have been shown useful for patch validation. However, it remains unclear whether BRTs can also help the more central stage of patch generation. We first conduct a preliminary study and find that directly using advanced BRT generators to guide patch generation is not beneficial: fail-to-fail BRTs can mislead agents, while even fail-to-pass BRTs bring limited or negative gains. Our analysis reveals two reasons: fail-to-pass BRTs may cover only one manifestation of the reported issue, leading to partial patches, whereas fail-to-fail BRTs are unreliable as direct patch-generation targets. Motivated by these insights, we propose SWE-Doctor, a software issue resolution agent that guides patch generation with runtime diagnoses derived from multi-faceted BRT executions. SWE-Doctor first generates multi-faceted BRTs for different behavioral requirements stated in the issue, then executes and debugs these BRTs to construct runtime-grounded diagnosis records, and finally uses the diagnoses together with localization information inferred during BRT generation to guide patch generation and reduce partial patches. We evaluate SWE-Doctor on Python bug-fixing issues from the widely adopted SWE-bench Verified and SWE-bench Pro across five LLM backends. SWE-Doctor consistently outperforms existing agents across all 10 LLM-benchmark combinations, achieving average resolution rates of 75.7% on SWE-bench Verified and 59.4% on SWE-bench Pro. In particular, on the more challenging SWE-bench Pro, SWE-Doctor improves the average resolution rate by 8.0-8.9 percentage points over the baseline agents.

Yaoqi Guo; Yang Liu; Jie M. Zhang; Yun Ma; Yiling Lou; Zhenpeng Chen


Control flow is represented as an inspectable graph rather than an opaque prompt loop. Runtime repair layer that abstracts agent runs into a dependency-aware critical-transition graph, localizes failure-critical subtrajectories after a run, and guides recovery on re-execution without modifying the underlying agent.

Use AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.06273; inspect its method and evaluation before treating results as production evidence.

[2607.06273] AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation

Large language model (LLM) agents are increasingly used for multi-step, stateful tool-use tasks, yet production reliability remains limited. Unlike static software repair, agent repair must recover dynamic trajectories whose early decisions can propagate into later errors and external state changes. Existing automatic remedies address only part of this problem: blind retry adds no diagnosis, outcome feedback says whether a run failed but not where or why, and self-reflection often lacks grounded evidence to prevent the same failure from recurring. We present AgentTether, a run-time repair framework that automates post-run diagnosis and guided recovery without modifying the underlying agent or environment. AgentTether abstracts each run into Transition Units, links them through a dependency-aware Critical Transition Graph, and localizes failure-critical subtrajectories by combining an offline normal-behavior model with a run-local graph detector. It then converts the localized cause into behavior-scoped guidance backed by cross-iteration Repair Memory, and can optionally apply guarded run-time intervention to keep the correction active during re-execution. The same design can be deployed as an offline diagnostic-and-guidance tool or as an online repair layer. We evaluate AgentTether on 261 tau-bench tasks across three domains with Qwen3.7-max, and test cross-model transfer on Banking with GPT-5.4. On the hardest Banking domain, AgentTether repairs 59.04% (49/83) of initially failed Qwen3.7-max tasks and 65.12% (56/86) of initially failed GPT-5.4 tasks. Overall, AgentTether improves repair effectiveness while reducing agent turns and end-to-end approach tokens, suggesting a practical reliability layer that can wrap existing agent deployments, reduce wasted re-execution, and improve recovery without retraining the agent.

Chenyu Zhao; Shenglin Zhang; Wenwei Gu; Yongqian Sun; Dan Pei; Chetan Bansal; Saravan Rajmohan; Minghua Ma


Packages the evidence as queryable CSV and JSONL rather than only a rendered page. Replaces one-shot PR generation with a generate-review-revise loop in which a reviewer agent explores the repository, accepts or rejects the PR, and feeds structured feedback into revision, with an accompanying benchmark and trajectory dataset.

Use SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.06065; inspect its method and evaluation before treating results as production evidence.

[2607.06065] SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review

Coding agents increasingly generate pull requests (PRs) for real-world software issues, yet one-shot PR generation remains open-loop: the PR is proposed without systematic review, diagnosis, or revision. We introduce \textbf{SWE-Review}, a framework for closing this loop with agentic code review. Given an issue and an AI-generated PR, a reviewer agent explores the repository, decides whether the PR should be accepted, and provides structured feedback for revision. We evaluate this setting with our proposed \textbf{SWE-Review-Bench} to measure both review correctness and downstream revision usefulness. We further curate \textbf{SWE-Review-Traj} dataset to study broader applications of agentic review and fill the data-scarcity gap for open reviewer training. Experiments show that agentic review continuously improves PRs through a generate-review-revise loop, outperforms single-turn fixed-context review in both decision accuracy and resolve rate after revision, transfers beyond review to improve issue-resolution models, and enables effective and efficient test-time scaling. These results position agentic code review as a practical mechanism for moving AI coding agents from one-shot PR generation toward closed-loop issue resolution.

Ruoyu Wang; Jierun Chen; Shaowei Wang; Chaofan Tao; Sidi Yang; Yuxin Jiang; Kim-Hui Yap; Lifeng Shang; Xiaohui Li; Haoli Bai


State persistence is explicit enough for repeated runs and handoff. Finds that 78% of observed agent failures in a tau^2-bench domain are silent wrong-state failures invisible to both the tool and the agent's self-report, and that deterministic read-only pre-execution gates in the loop recover them.

Use Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.07405; inspect its method and evaluation before treating results as production evidence.

[2607.07405] Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode in Tool-Using LLM Agents

Tool-using LLM agents can violate the very policies they are deployed to enforce while appearing to complete the task successfully. In policy-permissive environments, a tool may execute any well-formed call even when the corresponding state transition is forbidden by domain policy. The result is a silent wrong state (a booking cancelled, a passenger count changed, a claim acted on without verification) that neither the tool nor the agent's self-report exposes. We study this failure mode in the $\tau^2$-bench airline domain. On a budget agent, 78% of observed failures are silent wrong-state failures with no tool error, and the aggregate failure rate is reproducible across disjoint seeds, not sampling noise. We then evaluate a lightweight intervention: deterministic, read-only pre-execution gates that inspect the proposed call and current state before allowing a write. A four-gate suite raises full-benchmark success from 29.6% to 42.0% on gpt-4o-mini (+12.4pp; paired task-level bootstrap P=0.0012), and the lift reproduces on a disjoint 15-seed set (+12.3pp; P=0.0008). The effect is concentrated where the gates fire: on the 26/50 firing tasks, success rises by +19.2pp, while movement on the 24 non-firing tasks does not exclude zero. Two negative controls (a self-enforcing retail domain and BFCL) bound the mechanism: gates help when tools are policy-permissive and add little where tools already self-enforce. As suggestive evidence, not a central claim, the same failure mode persists at the frontier: gpt-5.2 at default reasoning still attempts policy-violating writes, and the same suite improves success from 61.2% to 71.6% (+10.4pp; P=0.020; n=5, no replication). The contribution is a bounded evaluation and reliability result: deterministic gates do not guarantee task success, but they can deterministically prevent a known class of silent policy-violating writes at the action boundary.

Vikas Reddy; Sumanth Reddy Challaram; Abhishek Basu


Verification is promoted from a final check to a loop-control signal. Wraps a general code agent in a verification harness and lets it run until every targeted Coq lemma is proved, beating fixed human-designed proof strategies and reaching full lemma coverage with no expert intervention.

Use Harnessing Code Agents for Automatic Software Verification to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.06341; inspect its method and evaluation before treating results as production evidence.

[2607.06341] Harnessing Code Agents for Automatic Software Verification

Formal verification offers the strongest guarantee of software correctness, but it does not scale: the proofs demanded by interactive theorem provers such as Coq require enormous expert effort. Large language models (LLMs) promise to generate these proofs automatically, yet existing approaches wire a fixed, human-designed proof strategy into the system and constrain the model to follow it (retrieving premises and predicting tactics one step at a time, or splitting goals by divide-and-conquer), and still prove only a fraction of their target theorems. We show that imposing such a strategy is unnecessary and limiting. Handing the whole lemma to a general LLM code agent (for example, Claude Code), free to choose its own approach, and wrapping it in a verification harness is both simpler and more effective, achieving full coverage: every targeted lemma proved, with no failures and no Coq expert intervention. The agent writes the proofs under feedback and hard constraints from the harness that keep each one sound (accepted only when the prover's kernel closes it), complete (no obligation left unproved or silently dropped), and terminating (no divergent tactics). We evaluate this harness plus code agent along three dimensions. (1) Core logic: on Iris, the state-of-the-art separation logic for concurrent and memory-manipulating programs, Aria proves all 4,257 lemmas of the four core modules and the 217 lemmas verifying Rust's standard libraries built on it, fully automatically. (2) Comparison with prior LLM provers: on reglang, where prior provers manage barely one in eight, Aria proves all 318. (3) Generality: on iris-lean, the unfinished Lean 4 port of Iris, it proves 72 not-yet-ported lemmas, showing the approach is not specific to Coq. A state-of-the-art model (Claude Opus 4.7) can write proofs for verified software development fully and automatically.

Shuangxiang Kan; Shuanglong Kan; Sebastian Ertel


Verification is promoted from a final check to a loop-control signal. Treats verification as a scaling axis and builds a training-free framework that computes continuous scores from token logits for fine-grained agentic feedback, scaled via score granularity, repeated evaluation, and criteria decomposition.

Use LLM-as-a-Verifier: A General-Purpose Verification Framework to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.05391; inspect its method and evaluation before treating results as production evidence.

[2607.05391] LLM-as-a-Verifier: A General-Purpose Verification Framework

Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of LLMs. In this work, we identify verification, the ability to determine the correctness of a solution, as a new scaling axis. To unlock this and demonstrate its effectiveness, we introduce LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training. Unlike standard LM judges that prompt LLMs to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring token logits to generate continuous scores. This probabilistic formulation enables verification to scale along multiple dimensions: (1) score granularity, (2) repeated evaluation, and (3) criteria decomposition. In particular, we show that scaling the scoring granularity leads to better separation between positive and negative solutions, resulting in more calibrated comparisons. Moreover, scaling repeated evaluation and criteria decomposition consistently lead to additional gains in verification accuracy through variance and complexity reduction. We further introduce a cost-efficient ranking algorithm for selecting the best solution among candidates using the verifier's continuous scores. LLM-as-a-Verifier achieves state-of-the-art performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%). Beyond verification, the fine-grained signals from LLM-as-a-Verifier can also serve as a proxy for estimating task progress. We build an extension for Claude Code, enabling developers to monitor and improve their own agentic systems. Finally, we show that LLM-as-a-Verifier can provide dense feedback for RL, improving the sample efficiency of SAC and GRPO on robotics and mathematical reasoning benchmarks.

Jacky Kwok; Shulu Li; Pranav Atreya; Yuejiang Liu; Yixing Jiang; Chelsea Finn; Marco Pavone; Ion Stoica; Azalia Mirhoseini

Code: https://github.com/llm-as-a-verifier/llm-as-a-verifier Website: https://llm-as-a-verifier.com


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Moves deterministic agent behavior out of prompts into code, schemas, and behavior contracts, wrapping validation around a replaceable model boundary so enterprise agents remain auditable and safe across model substitutions.

Use From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.08028; inspect its method and evaluation before treating results as production evidence.

[2607.08028] From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents

Enterprise large language model (LLM) applications often begin as prototypes whose behavior is carried by prompts and retrieval context. Productization adds requirements for source boundaries, entity routing, answer contracts, and reproducible traces. We present a harness-engineering approach that reconstructs this pattern into a traceable, auditable LLM-agent architecture: deterministic behavior moves into code, manifests, schemas, and validation artifacts around a replaceable composition boundary, while source-backed claims remain the authority for runtime answers. We instantiate it on a public-data slice of five Korean corporate groups (25 listed companies) and evaluate three research questions. (1) The harness preserves its source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts across the fixed validation scenarios; a fault-injection control confirms the validators flag deliberately broken contracts. (2) The checks the harness enforces held under model substitution: across three hosted models, they passed on all 270 composition-boundary runs; failures were confined to the model-composed side and were caught and recorded. (3) The code-owned guarantees are load-bearing, not reproducible by prompting alone: holding the model fixed and varying only the enforcement layer, prompt instructions alone let recommendation-language and internal-trace-leakage violations reach the reader, which the harness blocks entirely. A bolt-on external guardrail prevents such violations too but over-refuses, dropping utility to 88/120 where the harness preserves full utility (120/120); in this ablation, only code-owned enforcement preserves both safety and utility. The result is a reusable engineering pattern for turning exploratory prototypes into auditable applications with versioned source, control, and validation artifacts.

32 pages, 6 figures, 16 tables. Reference implementation and evaluation artifacts: https://github.com/hammerbaki/enterprise-llm-agent-harness (archived at https://doi.org/10.5281/zenodo.21269426)


Control flow is represented as an inspectable graph rather than an opaque prompt loop. STRACE structures redundant, heterogeneous agent execution traces by mining batch-level failure patterns and performing causal localization over a textual dependency graph, handing root causes rather than noisy trajectories to the reflection-based optimizer and lifting success on a formal verification task from 42.5% to 58.5%.

Use From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.07702; inspect its method and evaluation before treating results as production evidence.

[2607.07702] From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization

The optimization of long-horizon agents increasingly relies on reflection-based mechanisms, where a large language model (LLM) acts as an optimizer to diagnose agent failures and improve agent policies. However, real execution traces are difficult to use directly for optimization: large trace collections are often redundant and heterogeneous, making optimization inefficient and prone to overfitting to low-value failures; meanwhile, each individual trajectory also contains many irrelevant steps, while naive context reduction methods such as truncation or sliding windows can discard causally important evidence and produce misleading optimization signals. To resolve this dilemma, we introduce STRACE (Structural TRajectory Analysis and Causal Extraction), a framework that constructs high signal-noise optimization contexts for more precise and effective optimization. At the batch level, STRACE mines failure patterns to filter redundant traces and retain representative failures; within each selected trace, it performs causal localization over a textual dependency graph to remove non-causal steps and identify the true root-cause module for optimization. Empirical results demonstrate that STRACE significantly outperforms standard context-filtering baselines. Notably, on a challenging formal verification task (VeruSAGE-Bench), it successfully optimizes human-expert designed agents, delivering $1.4\times$ success-rate improvement (42.5% to 58.5%). The code is available at https://github.com/moomight/STRACE .

Ying Chang; Jiahang Xu; Xuan Feng; Chenyuan Yang; Peng Cheng; Yuqing Yang


Verification is promoted from a final check to a loop-control signal. AgentLocate attributes failures in LLM multi-agent trajectories to both the responsible agent and the earliest decisive step, pairing LLM-based evaluation with independent assessor verification and confidence-weighted aggregation to outperform prior attribution methods on two benchmarks, the diagnose side of the verify step for dispatched-agent loops.

Use Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.07989; inspect its method and evaluation before treating results as production evidence.

[2607.07989] Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems

Large language model (LLM) based multi-agent systems enable complex problem solving through coordinated reasoning and action, but their distributed structure also introduces new challenges in diagnosing system-level failures. When an execution fails, identifying which agent is responsible and at what point the trajectory first becomes irreversibly misdirected is difficult due to long-horizon interactions and tightly coupled agent behaviors. In this paper, we study the problem of failure localization in LLM-based multi-agent systems and present AgentLocate, a framework that attributes failures to both a specific agent and the earliest decisive step. AgentLocate combines an LLM-based judging mechanism with multi-perspective verification by independent evaluators, whose assessments are aggregated using a confidence-aware strategy. The resulting feedback is further used to adapt the judge through lightweight fine-tuning, improving attribution quality. We evaluate AgentLocate on two complementary benchmarks covering diverse tasks, agent configurations, and trajectory lengths. Experimental results show that AgentLocate consistently outperforms existing failure localization methods in identifying both responsible agents and failure steps, while remaining efficient in terms of token usage and running time.

Yufei Xia; Anjun Gao; Yueyang Quan; Zhuqing Liu; Minghong Fang

Accepted at Conference on Language Modeling (COLM); the linked arXiv record is the available paper version.

Official COLM accepted-papers list and current arXiv note


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Builds a causal theory of 26 constructs and 67 relationships from 3,100 coded practitioner documents on how AI-authored pull requests reshape code review, arguing review is the control point through which a coding agent's effect on software is decided and that outcomes hinge on team expertise and review process structure rather than AI itself.

Use 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.07980; inspect its method and evaluation before treating results as production evidence.

[2607.07980] 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

Coding agents now author entire pull requests, and practitioners sharply disagree about what this does to code review: whether it becomes the bottleneck, whether human review is still necessary, and whether it quietly erodes the understanding that it once built. Repository-mining studies measure surface trends but seldom explain the mechanisms beneath them, and the trends themselves prove unstable. A motivating observational analysis of public GITHUB activity finds that agent-authored pull requests are reviewed less often, merged several times faster, and discussed less than human-authored ones, yet the direction of these trends flips under different but equally defensible analysis choices, so the traces establish what is changing without explaining why. To recover the mechanisms, we synthesize practitioner discourse at scale into an explanatory theory: we collect 38,709 grey-literature documents (engineering blogs and Reddit threads), filter to those substantively about code review, and code a stratified random sample of 3,100 with an LLM-assisted pipeline, from which we build a causal model of 26 constructs and 67 relationships (64 directed, 3 contested). Its organizing claim is that review is the control point through which a coding agent's effect on software is decided, and that AI does not fix the sign of that effect: the team sets it, through the expertise its humans bring and how it structures the review process. The theory makes the competing positions explicit and turns "AI is changing code review" into falsifiable propositions with named constructs and moderators. As a secondary contribution, we offer the underlying LLM-assisted, grey-literature theory-building method as a scalable template for software-engineering research, with a public implementation.

Shyam Agarwal; Courtney Miller; Christian Kästner; Bogdan Vasilescu


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Stress-tests chain-of-thought monitoring as an in-loop safety gate: adversarial agents arguing for policy-violating proposals turn the scratchpad into a persuasion channel, with monitor access to the agent's reasoning increasing approval of harmful actions by 9.5% on average, while pairing monitor and fact-checker from different model families cuts violating approvals by up to 45%.

Use Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.08066; inspect its method and evaluation before treating results as production evidence.

[2607.08066] Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring

Chain-of-thought (CoT) monitoring is a promising safety mechanism for AI agents, based on the premise that visible reasoning traces can surface misaligned or deceptive behavior. While effective in standard scenarios, recent work highlights that LLMs remain vulnerable to persuasion-based jailbreaks, where natural-language arguments override model constraints. We stress-test whether this vulnerability extends to monitoring LLMs: can an adversarial agent persuade its CoT monitor to approve proposed actions that violate the monitor's policy? We design an evaluation framework with 40 tasks and analyze thousands of agent-monitor interactions, where agents are instructed to argue for policy-violating proposals. We find that in such adversarial settings, monitor access to the agent's CoT reasoning increases rather than decreases approval of harmful actions on average by 9.5%, as the scratchpad provides an additional persuasion channel. To address this, we introduce a fact-checking monitoring framework. We find that a fact-checker and monitor pairing from different model families, for example a Claude 3.7 Sonnet monitor paired with a GPT-4.1 fact-checker, reduces approval of policy-violating actions by up to 45%, compared to only 6%, when using the same model for both fact-checking and monitoring roles. Our results demonstrate that CoT monitoring alone may be insufficient against adversarial persuasion, and that model-diverse fact-checking provides a robust mitigation.

Jennifer Za; Julija Bainiaksina; Nikita Ostrovsky; Tanush Chopra; Victoria Krakovna


Verification is promoted from a final check to a loop-control signal. Verification-first workflow (PA-SciML) for agentic model discovery in scientific ML: fixes the scoring evaluator before search, derives machine-checkable physics requirements (boundary conditions, superposition, stiffness scaling, causality), audits every trained candidate's predicted fields against them, and separately searches prescribed input ranges for high-violation cases, reporting a surrogate as verified only under the stated checks; a domain-specific case of verification-gated agentic search.

Use Physics-Audited Agentic Discovery in Scientific Machine Learning to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.07379; inspect its method and evaluation before treating results as production evidence.

[2607.07379] Physics-Audited Agentic Discovery in Scientific Machine Learning

In agentic scientific machine learning (SciML), large language model (LLM) agents can discover surrogate models and select one by an automated score, typically an error metric. A low error, however, does not establish that the predicted fields satisfy the physics that matter for mechanics, such as boundary conditions, superposition, stiffness scaling, or causality. We introduce Physics-Audited Agentic SciML (PA-SciML), a verification-first workflow for agentic SciML discovery. The workflow fixes a scoring evaluator before search, derives reviewable machine-checkable physics requirements, checks each trained candidate on its outputs, and separately searches prescribed input ranges or measured load-history spans for high-violation cases without reference solution fields. A surrogate is reported as verified only under the stated checks. When enabled, the workflow also adds advisory numerical probes before training and tests one modeling change at a time to record which isolated edits are associated with score gains before reuse. In the reported computational-solid-mechanics numerical examples, the static elasticity run selects a surrogate with lower validation error than the error-only baseline while both selected models pass the common linear-elastic checks. In the transient elastodynamics run, an error-only baseline with similar mean error fails a stricter causality check by responding to future parts of the loading history, while the selected surrogate passes the stated checks. The main distinction is per-candidate physics evidence on predicted fields, not a richer aggregate score.

Diab W. Abueidda; Bilal Ahmed; Panos Pantidis; Mostafa E. Mobasher


The resource is directly reusable as a starting artifact. TrajSpec runs a trajectory-collection agent over the pre-fix repository and mines the unverified trajectory for specification evidence, refining vague bug reports into structured specifications that guide automated program-repair loops.

Use Bug Report Specification Refinement with Trajectory Guidance for Automated Program Repair to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.07882; inspect its method and evaluation before treating results as production evidence.

[2607.07882] Bug Report Specification Refinement with Trajectory Guidance for Automated Program Repair

Bug reports serve as task specifications for repository-level automated program repair (APR) agents, but they often describe only the observed failure and omit repair-relevant information such as the failure-inducing behavior, behavioral requirement, and implementation scope. As a result, a repair agent may inspect irrelevant code, infer an incorrect requirement, or generate a patch that addresses the reported symptom without restoring the intended repository behavior. We present TrajSpec, a trajectory-guided approach for repository-supported bug report specification refinement. Given an original report and a pre-fix repository, TrajSpec runs a trajectory-collection agent and uses the resulting unverified trajectory as a source of trajectory-derived specification evidence. It organizes this evidence into a three-level representation consisting of a high-level interpretation of the issue, diagnostic findings supporting that interpretation, and concrete repository observations. TrajSpec then generates a draft refined report and applies repository-based review to remove unsupported claims, revise uncertain claims, and add repository-supported details. We evaluate TrajSpec on all 300 SWE-Bench Lite instances using Mini-SWE-Agent V2. TrajSpec's refined reports improve Pass@1 from 41.00% to 59.67% with GPT-5-mini and from 54.67% to 64.33% with MiniMax M2.5. On a stratified sample of 100 instances, TrajSpec's refined reports also improve Pass@1 from 41.00% to 71.00% with Agentless and from 47.00% to 72.00% with AutoCodeRover. Ablation results show that removing repository-based review or the hierarchical evidence representation reduces Pass@1 from 59.67% to 48.00% and 47.67%, respectively. Overall, TrajSpec provides actionable repository-supported context that consistently improves repair performance.

S M Farah Al Fahim; Md Nakhla Rafi; Md Ahasanuzzaman; Zeyang Ma; Dong Jae Kim; Shaowei Wang; Tse-Hsun; Chen