Evaluation data is used as the feedback signal for improving loop behavior. Empirical anatomy of 3,843 CLI coding-agent trajectories across seven models and three scaffolds (OpenHands, MiniSWE, Terminus2), with 1,794 fully annotated over 63,000+ manually reviewed steps; models failure as a temporal process of onset, evolution, and recovery and finds failures dominated by epistemic errors that begin within the first few steps yet stay undetected until recovery is impossible, arguing for in-loop validation and intervention over final-outcome evaluation.

Use Failure as a Process: An Anatomy of CLI Coding Agent Trajectories to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.09510; inspect its method and evaluation before treating results as production evidence.

[2607.09510] Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

Large language model (LLM) coding agents are increasingly deployed to autonomously perform software engineering tasks in terminal-based environments, making their reliability a growing concern. Existing empirical studies investigate why coding agents fail, yet they largely treat failure as a final outcome rather than a temporal process, providing limited insight into how failures emerge, evolve, and become unrecoverable. We present the first large-scale empirical study of CLI coding-agent failure trajectories, introducing a process-oriented framework that analyzes failure through its onset, evolution, and recovery across execution trajectories. We first collect 3,843 execution trajectories generated by seven frontier models across three coding-agent scaffolds (OpenHands, MiniSWE, and Terminus2) on Terminal-Bench, then carefully filter them to obtain 1,794 complete and valid trajectories for manual annotation (over 63,000 execution steps), from which we derive 14 findings spanning failure occurrence, root causes, recovery, and cross-system consistency. Our findings show that coding-agent failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is no longer possible, suggesting that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.

Xiangxin Zhao; Han Li; Shuaiting Li; Tianyi Zhao; Earl T. Barr; Federica Sarro; He Ye


Verification is promoted from a final check to a loop-control signal. Dual-track verification-in-the-loop for AI-generated code: shared property templates drive both formal proof in Lean 4 and executable property-based tests for PySpark, raising agentic proof success up to 2.6x and cutting proof hallucinations by 59%.

Use Agentic Proof and Property-Based Testing via Property-Templates to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.09072; inspect its method and evaluation before treating results as production evidence.

Gate progress with tests, evals, and evidence.

[2607.09072] Agentic Proof and Property-Based Testing via Property-Templates in Data-Intensive Computing

As the cost of code generation becomes cheaper with AI, the new bottleneck in software engineering has shifted to intent specification and validation. Overcoming this durability crisis of AI-driven coding requires more than traditional fuzzing: each candidate property must be proven correct over a model and shown to hold on the real implementation, making formal proof and systematic property-based testing (PBT) complementary. However, validating properties this way at scale requires solving two subproblems: verifying candidate properties and operationalizing PBT without AI hallucination. We hypothesize that recurring property patterns, cast as property templates--abstract, parameterized forms with holes--address both at once. This paper investigates recurring property patterns in Apache Spark. In data-intensive scalable computing systems, correctness properties arise from the principles of data partition, computation decomposition, and dataflow computation. For instance, aggregation decomposition relates a global function executed on the entire dataset to a local function followed by a recombiner. We design an agentic, dual-track validation framework that uses property templates to formally verify correctness in the Lean 4 theorem prover and instantiate PBT templates as executable PySpark tests. Our evaluation shows that property templates increase agentic proof engineering success by up to 2.6x (1.6x on average) and reduce proof hallucinations by 59%. Template-guided PBT synthesis reduces intent misalignments from 22 to 1 and cuts synthesis cost by up to 5.7x (3.8x on average). Template-guided synthesis further exceeds a state-of-the-art Spark fuzzer and approaches unguided LLM-based PBT on code coverage. Finally, comparing the two tracks is informative: when a proof succeeds yet a PBT finds a counterexample, the mismatch identifies a gap between the formal model and implementation.

12 pages, 7 figures, 4 tables; supplementary material included as ancillary file


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Workbench that reproduces an agent failure, intervenes at the point it went wrong, and tests mitigations, turning one-off agent bugs into a repeatable diagnose-and-fix loop over MCP tool use.

Use AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.11098; inspect its method and evaluation before treating results as production evidence.

[2607.11098] AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

Tool-using LLM agents are mostly evaluated assuming all tools work. When a tool times out, returns a week-stale value, or has its description poisoned in deployment, the developer needs a controlled way to reproduce the failure, test a fix, and confirm the fix worked before deployment. We present AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface. AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector. Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges. This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away. Scoring has two parts: deterministic pass/fail rules, plus an LLM judge for interpretive labels, validated against human annotations. Across five agents, the best passes 105/120 scenarios and the weakest only 77. The failures are usually silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent, a retry mitigation raises success on timeout error faults from as few as 30% of cases to 100%, whereas stale-data faults remain near 3-4 of 10 regardless of the mitigation. AgentCheck makes these failure modes reproducible, comparable, and verifiable before deployment.


Verification is promoted from a final check to a loop-control signal. Shows a coding agent's hidden states linearly encode program properties like correctness and test outcomes and predict future edits up to 25 steps ahead, a latent signal that could gate or steer verification loops before edits materialize.

Use Latent Programming Horizons in Coding Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.05188; inspect its method and evaluation before treating results as production evidence.

[2607.05188] Latent Programming Horizons in Coding Agents

A coding agent solving a software-engineering task spends dozens of steps reasoning, editing code, and running tests, yet little is known about what the underlying language model internally represents about the program it is working on. We show that the residual streams of language models under coding agents linearly encode properties of the evolving program: a logistic-regression probe on hidden states is able to decode whether the current code parses, passes its test suite, reduces the number of failing tests, and introduces regressions, reaching AUC up to 0.83 for correctness across two models and two benchmarks. Our second finding is more surprising: these representations run ahead of the agent's own edits. Probes trained to predict the outcome of future edits (before they are materialized and written on disk) achieve performance above chance up to roughly 25 steps in advance. We call this the agent's latent programming horizon. As a proof of external validity, we show that the probes transfer across benchmarks without retraining. Our positive results open calls for more research in mechanistic interpretability of coding agents.


Why evaluate agents

Primary-source operational guidance rather than commentary. Official ADK guide to evaluating final responses and trajectories, defining test cases, selecting criteria, and running repeatable agent evaluations locally or in CI.

Use Why evaluate agents to measure progress and gate completion with repeatable evidence.

Primary official documentation from adk.dev; use it for current product or standard behavior.

Why evaluate agents - Agent Development Kit (ADK) Agent Development Kit (ADK)

Build powerful multi-agent systems with Agent Development Kit (ADK)


The contribution is machine-readable and validation-friendly. In 50 paired TextWorld tasks under a four-call budget, feedback containing the failure location, observed value, and admissible alternatives raises repair success from 14/50 to 36/50 for one model and 8/50 to 29/50 for another; ablations identify alternatives, not JSON syntax, as the main driver.

Use Structured Feedback Improves Repair in an LLM Agent Loop to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.14167; inspect its method and evaluation before treating results as production evidence.

[2607.14167] Structured Feedback Improves Repair in an LLM Agent Loop

LLM agents often retry after external validation rejects a candidate, but the interface between validation and the next model call remains underspecified. We introduce VeriHarness, a code-controlled agent loop in which models generate candidates while external validators control acceptance, budgets, and traces. We use it to compare raw diagnostics with feedback that identifies the failure location, observed value, and admissible alternatives. Across 50 paired TextWorld games under a four-call cap, feedback containing all three fields raises terminal success from 14/50 to 36/50 for Qwen2.5-Coder-14B (+44 percentage points) and from 8/50 to 29/50 for Llama-3.1-8B (+42 points). Ablations locate most of the gain in the admissible alternatives: feedback containing only the location and observed value remains near the raw diagnostic baseline. Presenting the complete repair information in prose instead of a keyed JSON record yields nearly the same success, providing no evidence that JSON syntax itself improves repair. The ordering persists across the tested call budgets and one sampled-decoding setting.


State persistence is explicit enough for repeated runs and handoff. Uses PostgreSQL copy-on-write isolation to let an agent modify a realistic application state while a scorer evaluates the resulting operations safely; the Plane case study also exposes tool-surface defects that simpler task checks miss.

Use Copy-on-Write Scoring: Application-Specific Agent Evaluations to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.14336; inspect its method and evaluation before treating results as production evidence.

[2607.14336] Copy-on-Write Scoring: Application-Specific Agent Evaluations

Trustworthy deployment of LLM-based agents in software systems requires evaluating how they perform on application-specific workflows, with enough granularity to localize where they succeed and fail. Yet existing agent evaluation mechanisms are limited: benchmarks have low construct validity for application-specific workflows and environments, and replica evaluation environments are expensive and prone to drift. We propose Copy-on-Write (CoW) Scoring, a framework that evaluates agent operations directly within application environments using a PostgreSQL-level Copy-on-Write mechanism to isolate agent writes. CoW Scoring produces session- and operation-level scores that highlight where agents' database write operations succeed and fail in a given application environment, enabling inexpensive evaluation and iteration on agent harnesses and tool surfaces. We demonstrate the framework on Plane, an open-source project-management platform, where analysis surfaced specific issues in the tool surface, and corresponding fixes produced measurable improvements on affected models. Python library: https://github.com/trail-ml/agent-cow-python

International Conference on Machine Learning

Accepted at ICML Workshop on Agents in the Wild: Safety Security and Beyond; the linked arXiv record is the available paper version.

Current arXiv acceptance note and official workshop page


Verification is promoted from a final check to a loop-control signal. Places formal proof obligations inside a coding-agent repair loop and reports 49,280 discharged obligations with 20-40x less supervision; the paper also states that proofs must be paired with known-answer tests, interoperability checks, and human specification review.

Use The Prover Is the Judge: Verified Security Software from AI Coding Agents in Ada/SPARK to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.14340; inspect its method and evaluation before treating results as production evidence.

[2607.14340] The Prover Is the Judge: Verified Security Software from AI Coding Agents in Ada/SPARK

AI coding agents produce code faster than humans can review it. In our approach, the prover is the judge of whether the code is correct. Under a verifier-driven loop, AI agents wrote and verified bare-metal security software in Ada/SPARK spanning classical and post-quantum cryptography, TLS 1.3, IKEv2, X.509, and a Matrix client. GNATprove discharged 49,280 proof obligations, established functional correctness for selected primitives, and proved the absence of run-time errors for the rest, at roughly 20-40 times lower supervision cost than comparable hand verification. GNATprove alone was insufficient: some defects could not be detected and were resolved using known-answer tests, interoperability, or human review of specifications. Given weak checks, the agent tried to bypass them and reported success. We report where each layer caught faults and draw the central lesson: what an agent can be trusted to establish is bounded by the strength of its feedback.


Verification is promoted from a final check to a loop-control signal. Combines formal concept-and-reaction semantics with a counterexample-guided LLM synthesis loop whose candidates must satisfy machine-checked invariants; experiments on three applications show scenarios steer the loop more consistently than natural-language prompts, while sparse scenarios overfit and nondeterministic omissions still limit coverage.

Use Verified LLM-Driven Synthesis for Concept Design to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.15718; inspect its method and evaluation before treating results as production evidence.

[2607.15718] Verified LLM-Driven Synthesis for Concept Design

Concept Design structures software systems around concepts: user-facing, self-contained units of functionality with a focused purpose. Concepts are composed into applications using synchronization rules called reactions, which specify how actions in one concept trigger actions in others. This paper first gives a formal semantics for concepts and reactions, enabling automatic verification of safety invariants in applications developed with this methodology. It then presents a CEGIS-style, LLM-driven synthesis procedure for generating reaction designs that satisfy such invariants. Because many different designs can satisfy the same invariant, we study two ways of steering synthesis toward the user's intended design: natural-language prompts and positive/negative scenarios. We also propose an LLM-driven scenario elicitation technique to support early design exploration. In an evaluation on three applications and twelve design variants using one LLM configuration, invariant-only synthesis reached verified designs quickly but often produced inconsistent designs across runs, some of which were implausible, showing that invariants alone underconstrain the design task. Scenario-guided synthesis recovered intended designs more consistently than natural-language prompting, although minimal scenarios can lead to overfitting. LLM-driven scenario elicitation, where the user classifies proposed scenarios rather than authoring them from scratch, recovered the intended designs in most variants when enough scenarios were elicited, but missed behaviors and non-determinism prevented reliable coverage in all cases.


The contribution is machine-readable and validation-friendly. Pairs preflight LLM checks against machine-readable assay rules with runtime visual monitoring of physical execution; reports adjusted F1 0.97 for protocol validation and average precision 0.89 for trajectory monitoring, while explicitly surfacing the small-pipette and transparent-liquid limits.

Use AEGIS: Assay-Aware Protocol Validation and Runtime Monitoring for Open-Source Liquid Handling Robots to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.15620; inspect its method and evaluation before treating results as production evidence.

[2607.15620] AEGIS: Assay-Aware Protocol Validation and Runtime Monitoring for Open-Source Liquid Handling Robots

Self-driving laboratories increasingly rely on low-cost liquid handlers such as the Opentrons OT-2, which ship without the pressure-based aspiration monitoring of Hamilton or Tecan systems and are typically run open-loop. Two failure modes go undetected: protocols that are syntactically valid but violate assay-specific invariants (e.g., tip reuse between a PCR template and a no-template control), and physical execution failures (partial dispense, air bubbles, missing tips) at runtime. We present AEGIS, a two-layer guardian for both. Layer 1 pairs a curated machine-readable assay rule database with an LLM that reasons over OT-2 Python code, reaching an adjusted F1 of 0.97 on a 24-protocol benchmark across five assay families and beating rules-only and LLM-only ablations across five backends; a free open-weight model ties the best proprietary one, so no paid API is required. Layer 2 fits a PCA world model to YOLO-cropped four-frame pipette trajectories; under a leakage-free leave-one-plate-out evaluation it reaches average precision 0.89 and operating-point F1 0.71 (AUROC 0.80), a deployment-faithful number that matches the live demonstration, and we characterize the small-pipette (p20) resolution limit (F1 0.47). A live demonstration on a physical OT-2 (five replicates per condition) catches planted no-tip failures deterministically and partial dispense on coloured dyes, with an always-VLM self-vote gate lifting partial-dispense recall to 5/5; transparent water is a principled limit of any front-view-only monitor, which AEGIS surfaces as low-confidence VLM reasoning rather than a wrong verdict. Cascade triage holds VLM cost near $1.63 per plate versus $10.33 for an always-VLM baseline. AEGIS is open source and, to our knowledge, the first system to unify pre-flight assay-aware validation with runtime visual monitoring for an open-source liquid handler.

Priyanka V. Setty; Arvind Ramanathan; Ian Foster; Rick Stevens


Verification is promoted from a final check to a loop-control signal. Compiles expert guidelines into step-wise trajectory checks, calibrates accumulated evidence into correctness probabilities, and triggers additional verification when uncertainty remains high.

Use GLEAN: Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification to measure progress and gate completion with repeatable evidence.

Research source arXiv:2603.02798; inspect its method and evaluation before treating results as production evidence.

[2603.02798] Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification

As LLM-powered agents have been used for high-stakes decision-making, such as clinical diagnosis, it becomes critical to develop reliable verification of their decisions to facilitate trustworthy deployment. Yet, existing verifiers usually underperform owing to a lack of domain knowledge and limited calibration. To address this, we establish GLEAN, an agent verification framework with Guideline-grounded Evidence Accumulation that compiles expert-curated protocols into trajectory-informed, well-calibrated correctness signals. GLEAN evaluates the step-wise alignment with domain guidelines and aggregates multi-guideline ratings into surrogate features, which are accumulated along the trajectory and calibrated into correctness probabilities using Bayesian logistic regression. Moreover, the estimated uncertainty triggers active verification, which selectively collects additional evidence for uncertain cases via expanding guideline coverage and performing differential checks. We empirically validate GLEAN with agentic clinical diagnosis across three diseases from the MIMIC-IV dataset, surpassing the best baseline by 12% in AUROC and 50% in Brier score reduction, which confirms the effectiveness in both discrimination and calibration. In addition, the expert study with clinicians recognizes GLEAN's utility in practice.

Yichi Zhang; Nabeel Seedat; Yinpeng Dong; Peng Cui; Jun Zhu; Mihaela van de Schaar


The work targets tasks that exceed a single context window or prompt session. Defines semantic livelock as continued agent activity without progress and proposes an independent embedding-based convergence monitor that detected the pattern in 25% of the analyzed long-duration SWE-agent failures.

Use Zombie Agents: Detecting Semantic Livelock in Long-Horizon Autonomous Software to measure progress and gate completion with repeatable evidence.

Research source; inspect its method and evaluation before treating results as production evidence.

Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware '26)

Published at AIware 2026; metadata verified from the author-supplied camera-ready paper because the DOI landing page restricted automated access.


Evaluation data is used as the feedback signal for improving loop behavior. Every self-improvement loop rests on an evaluation metric that can itself drift or be gamed; the Double Ratchet framework co-evolves metrics alongside agent skills with anchor discipline and independent audits, retaining 88-110% of ground-truth lift across code generation, SQL, and report-writing loops.

Use Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.12790; inspect its method and evaluation before treating results as production evidence.

[2607.12790] Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

Self-evolving agent systems create, revise, and retire their own skills, but every such loop assumes a reliable evaluation metric already exists. In many real applications none does. We show the metric itself can be the evolving object: our loop searches compositions of small typed drawback detectors under a full evolutionary lifecycle, selecting for agreement with a ten-item anchored reference set and regularizing by consensus over unlabeled outputs. What evolves is the function that grades one output, never the fixed task sets it is scored on, and what comes out is an inspectable expression rather than an opaque judge. It is also valid: on code generation it gains 0.21 agreement with hidden ground truth on a locked set that metric selection never reads (paired $p=0.014$), beating the bare LLM judge it contains. Validity is where safety lives: removing the anchor guards collapses the metric into a vacuous always-pass detector while removing the detector lifecycle does not, inverting the lesson from skill evolution. That collapse warns this line of work that downstream task score cannot validate a self-evolved evaluator, since the collapsed metric trains skills just as well. Task score answers only sufficiency, and an evolved metric suffices: \emph{Double Ratchet}, co-evolving the metric with a lifecycle-managed skill loop, retains 88--110\% of the lift ground truth or a hand-written rubric buys, across MBPP+, Spider~2.0-Snow, and report generation. When evolved skills gamed the report rubric, an independent judge caught it and one added detector repaired it.

Xing Zhang; Guanghui Wang; Yanwei Cui; Ziyuan Li; Wei Qiu; Bing Zhu; Peiyang He


Verification is promoted from a final check to a loop-control signal. First-Order Linear Temporal Logic specification language for verifying that tool-using agents follow procedural rules across the whole trace, not just reach correct answers; supports real-time in-loop tool-call gating and dense-reward training, with compliance gains that generalize to unseen procedural variations.

Use AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.02599; inspect its method and evaluation before treating results as production evidence.

[2607.02599] AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agents

Tool-using LLM agents are usually evaluated by final-answer correctness or LLM judges. Neither captures how an answer was produced. In safety-critical settings, the procedure itself is part of correctness. In this paper, we introduce AgentLTL, a language derived from First-Order Linear Temporal Logic (FO-LTL) that expresses procedural rules over agent traces. It yields a deterministic, judge-free compliance score. In this framework, a single specification drives two usages. The first is harnessing: the constraints score completed traces, or gate tool calls by checking each prefix online, before execution. The second is finetuning: the score serves as a dense reward. On a benchmark spanning ordering, branching, iteration, and grounding, block-and-warn harnessing improves compliance on five of seven models. Finetuning with the same reward yields +38 and +17.5 percentage point gains in accuracy and compliance on held-out patterns, including unseen tool-name aliases. These findings are consistent with the model acquiring procedural structure rather than memorizing surface tool names and procedures.


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Trains lightweight linear probes on hidden states from an episode's earliest interactions to predict eventual failure, then aborts doomed runs through a calibrated cascade with user-specified recall guarantees, cutting generated tokens by roughly 54-60% at a 90% recall target on TextCraft and WebShop across three models, a when-to-stop gate that stops wasted runs instead of letting them burn to timeout.

Use Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.06503; inspect its method and evaluation before treating results as production evidence.

[2607.06503] Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade

Large language model (LLM) agents often waste inference compute by continuing multi-step trajectories that are already doomed to fail. We study early failure prediction and inference-time early stopping for LLM agents using hidden-state probes. Lightweight linear probes on internal activations predict eventual task failure from the first interaction round, substantially earlier than agent-monitoring methods based only on observable behavior. We turn this signal into a recall-controlled abort cascade for reducing LLM agent inference costs. The cascade applies a distribution-free calibrated failure detector at each early interaction round and jointly optimizes per-round recall budgets. This design ensures that eventually successful episodes survive all early-stopping gates at a user-specified global recall rate. After selection, the cascade is frozen and certified on independent data, providing an exact post-selection recall guarantee. We evaluate the method on TextCraft and WebShop with Qwen-2.5-7B, Llama-3.2-3B, and Qwen3-1.7B. The proposed LLM agent early-stopping cascade outperforms the best single-gate baseline in every model-environment pair, saving 1.5-8.8 times more compute at a 90% recall target. Achieved recall remains within one standard deviation of its target in all 24 configurations. The strongest settings reduce generated tokens by 60.2% on TextCraft and 54.9% on WebShop at 90% recall, while retaining savings of 45.0% and 41.5% at 95% recall. Behavior-only monitoring is consistently weaker, and adding behavioral features to hidden-state probes provides no further gain. We also characterize the sample complexity required to certify high-recall early-stopping policies. The code will be released soon.

Kai Ruan; Zihe Huang; Ziqi Zhou; Qianshan Wei; Jinghao Lin; Xuan Wang; Hao Sun


Verification is promoted from a final check to a loop-control signal. Ablates a production enterprise agent (Leni) across three benchmarks to decompose reliability gains among verification loops, specialist models, and scaffolding: 7-15pp improvements come mostly from scaffolding, routing, and specialist models, while the verification step's isolated gain is small (+1.5pp) yet rescues otherwise-failing tasks, and swapping the small trained verifier for the generating frontier model eliminates most rescues.

Use Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.17044; inspect its method and evaluation before treating results as production evidence.

[2607.17044] Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent

Multi-step enterprise agent tasks fail in a characteristic way: single-pass inference has no checkpoint between deciding an answer and committing to it. We study one production system (Leni) whose architecture installs such checkpoints: verification loops (execute, observe, compare, correct) staffed by lightweight task-specialized post-trained models. We evaluate the unmodified production configuration on three public benchmarks stressing distinct failure modes: SpreadsheetBench Verified (silent computation error), BullshitBench v2 (premise confabulation), and the GAIA validation split (cascade error over long tool chains). The full system improves over its frontier base model by +11.0 percentage points on SpreadsheetBench (91.25% vs 80.25%, n=400, p<0.001), +7 to +10 percentage points on BullshitBench (98% vs 91%, n=100), and roughly +15 points on GAIA validation (75.2% pass@1, n=165; 83.0% best-of-k). Our central contribution is a decomposition of that uplift: most of it comes from scaffolding, routing, and specialist models rather than from the verification step itself, whose isolated contribution is small (+1.5 points) but concentrated at the top of the score distribution, where it converts otherwise-failing tasks. We instrument the loop end-to-end, yielding an empirical verifier confusion matrix (catch rate about 0.20, fix rate 0.75, no false-alarm regressions) that grounds a compounding-reliability model. Specialist-swap ablations suggest that the loop's value depends on who observes it: replacing the small trained verifier with the generating frontier model eliminates most rescues. A valid-premise control shows zero over-rejections in 100 expert-level questions.

19 pages, 5 figures, 5 tables. Evaluations conducted March-April 2026. Run-level evaluation record and audit scripts: https://github.com/arnabdastidar/leni-agent-evals


Packages the evidence as queryable CSV and JSONL rather than only a rendered page. Mines 4,882 agent-generated PRs (five coding agents) from the AIDev dataset and finds agents ship tests in only 49.6% of code-changing PRs, existing tests cover just 27% of changed Python lines, and error-handling code goes unexecuted at up to 86% miss rates, empirical evidence that unattended agent loops under-verify their own output (to appear at ICSME 2026).

Use Test Coverage Analysis of Agentic Pull Requests to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.18057; inspect its method and evaluation before treating results as production evidence.

[2607.18057] Test Coverage Analysis of Agentic Pull Requests

AI coding agents increasingly submit complete pull requests (PRs) with minimal human intervention, shifting software development from AI-assisted to autonomous workflows. As these agents become more prevalent, ensuring the code they generate is adequately tested, by existing tests or by tests the agents write, is critical to preventing regressions, yet little is known about testing in agentic PRs. To address this gap, we analyze 4882 agent-generated PRs from the AIDev dataset (532 Java and 4350 Python PRs) produced by five coding agents. We study (i) how often agents include test changes and (ii) how well covered are code changes by existing and agent-written tests. Agents include test changes in only 49.6% of PRs that change code under test files. Existing tests provide an incomplete safety net: they cover 61.5% of agents' changed executable lines in Java and only 27.0% in Python, where 64.8% of PRs have no changed line executed by any existing test. Agent-written tests improve coverage over existing tests, but only in a minority of PRs: 35.9% of Java and 22.5% of Python Code + Tests PRs show a coverage gain. Across both languages, error-handling constructs (e.g., try and catch blocks) are the most consistently under-tested, with miss rates reaching 86.0% in Java and 81.0% in Python. These findings motivate coverage-aware development practices, coverage feedback loops for coding agents, and evaluation benchmarks that measure test quality to better help agents reliably test their own code.

Atish Kumar Dipongkor; Talank Baral; Wing Lam; Kevin Moran

12 pages, to appear 42nd International Conference on Software Maintenance and Evolution


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Attributes verbose "CodeSlop" to the agent loop's own search process, speculative edits, abandoned hypotheses, and temporary changes that survive into the final diff, and removes 17.9-32.9% of redundant code across agentic scaffolds by minimizing the trajectory rather than post-editing the code, with negligible performance regression (Columbia and Google).

Use TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimization to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.18161; inspect its method and evaluation before treating results as production evidence.

[2607.18161] TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimization

Coding agents are increasingly used to accelerate code generation in many downstream tasks, such as fixing bugs, building applications, and prototyping. However, despite their value as coding assistants, agent-generated code tends to be larger and more verbose than the corresponding human-written implementation. In this work, we show that the cause lies in the agent's own search process: while iterating toward a passing solution, an agent accumulates speculative edits, abandoned hypotheses, and temporary changes that persist into the final patch. This may seem harmless for a single patch, but the problem compounds as agents take responsibility for ever-larger portions of a codebase-a codebase that was once minimal and well-maintained slowly accumulates redundancy faster than it can be cleaned up, drifting to a state that is harder to maintain. Given the magnitude of this problem, we take a step towards alleviating this issue. First, we formally define this phenomenon as CodeSlop-the residual and functionally unnecessary edits commonly seen in AI-generated code. We then introduce our algorithm TRIM (Trajectory-guided Redundancy Identification and Minimization). Rather than minimizing CodeSlop directly, TRIM instead minimizes agent trajectories. As we show empirically, this indirect technique of minimizing CodeSlop is highly effective: TRIM cuts CodeSlop by 17.9%-32.9% across agentic scaffolds, with negligible performance regression. TRIM is also highly efficient, requiring roughly half the validation cost of algorithmic baselines such as Delta Debugging.

Alex Mathai; Shobini Iyer; Aleksandr Nogikh; Petros Maniatis; Franjo Ivancic; Junfeng Yang; Baishakhi Ray


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Trajectory-level study of terminal-based code-review agents that give feedback before pull-request creation, finding they achieve higher review precision but incur substantial exploration and validation overhead, and that successful reviews are associated with stronger planning and less downstream validation, behavior and cost evidence for designing review-agent gates.

Use Agentic Code Review in the Terminal: A Trajectory-Level Analysis of Behavior, Cost, and Human-Alignment to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.16740; inspect its method and evaluation before treating results as production evidence.

[2607.16740] Agentic Code Review in the Terminal: A Trajectory-Level Analysis of Behavior, Cost, and Human-Alignment

Agentic code review in terminal-based environments enables early feedback during local development before pull request creation. However, existing evaluations remain performance-centric and fail to capture the dynamic behaviors of repository-grounded agentic reviewers. Understanding these behaviors is critical for identifying how agentic reviewers succeed, fail, and incur hidden operational costs in practice. Then, we analyze the reviewers' behavior based on their trajectories. Our results show that agentic reviewers achieve higher review precision but incur substantial exploration and validation overhead, while successful reviews are associated with stronger planning and less downstream validation. These findings highlight the potential benefits of trajectory-aware and cost-sensitive evaluation of future agentic code review systems.

Wachiraphan Charoenwet; Kla Tantithamthavorn; Patanamon Thongtanunam; Hong Yi Lin; Minwoo Jeong; Ming Wu


brain0

Control flow is represented as an inspectable graph rather than an opaque prompt loop. Black-box decision graph for AI-written code that records why an agent made each change, making agent-authored diffs inspectable after the fact.

Use brain0 to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (367 stars; 13 forks; Apache-2.0 license; updated 2026-08-05); popularity is context, not proof of reliability.

GitHub - Brain0-ai/brain0: The black box for AI-written code. Passive decision graph linking every commit to the agent prompts behind it: drift detection, DLP audit of what agents read, evidence-driven risk, MCP memory for coding agents, signed provenance attestations. One command, offline by default. · GitHub

The black box for AI-written code. Passive decision graph linking every commit to the agent prompts behind it: drift detection, DLP audit of what agents read, evidence-driven risk, MCP memory for coding agents, signed provenance attestations. One command, offline by default. - Brain0-ai/brain0


Watch Skill

The agent workflow includes explicit self-checking or gated completion. Video understanding and self-verification layer that turns videos, streams, and agent screen recordings into searchable evidence an agent can check its own work against.

Use Watch Skill to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (266 stars; 38 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - oxbshw/watch-skill: Video understanding and self-verification for AI agents. Turn videos, streams, and agent screen recordings into searchable, timestamped evidence—then use THE LOOP to inspect, fix, and verify the work. MCP, CLI, REST, local-first. · GitHub

Video understanding and self-verification for AI agents. Turn videos, streams, and agent screen recordings into searchable, timestamped evidence—then use THE LOOP to inspect, fix, and verify the work. MCP, CLI, REST, local-first. - oxbshw/watch-skill


Auto

Verification is promoted from a final check to a loop-control signal. Compiles recorded LLM-agent behavior into verified, capability-confined WebAssembly binaries, proving which parts of a loop are secretly symbolic and distilling them into deterministic code.

Use Auto to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (121 stars; 10 forks; Apache-2.0 license; updated 2026-08-05); popularity is context, not proof of reliability.

GitHub - RightNow-AI/auto: the agi compiler: records llm agent behavior, proves what repeats, and compiles it into verified, sandboxed wasm binaries that run for microdollars. nothing figured out twice, paper: https://arxiv.org/abs/2607.04542 · GitHub

the agi compiler: records llm agent behavior, proves what repeats, and compiles it into verified, sandboxed wasm binaries that run for microdollars. nothing figured out twice, paper: https://arxiv.org/abs/2607.04542 - RightNow-AI/auto


Verification is promoted from a final check to a loop-control signal. Training-free verification-in-the-loop: a hindsight reviewer labels which steps of completed trajectories were genuinely productive, building an experience bank that calibrates the critic's step-level confidence on future runs, improving when a loop should trust, retry, or escalate an action.

Use Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.12397; inspect its method and evaluation before treating results as production evidence.

[2607.12397] Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents

LLM agents act in external environments where each action changes the state that later decisions condition on, and where a single wrong step can waste interaction budget or trigger irreversible side effects long before the final failure is observed. Reliable deployment therefore requires \emph{step-level confidence estimation}: a calibrated probability that each proposed action is productive, available \emph{before} the action is executed. Existing LLM confidence estimators are designed to score a response from the given prompt, but agent confidence also depends on execution consequences: whether similar actions in similar situations actually advanced the task after the environment responded. We introduce the \method (\methodshort), a self-evolving critic framework in which an LLM critic accumulates evidence from its own past judgments and their observed consequences. After each trajectory, a hindsight LLM that sees the full execution feedback votes on whether each step was productive. The resulting pseudo-labels populate a memory bank from which related productive and unproductive experiences are retrieved into the critic's prompt whenever a similar step recurs. \methodshort requires no training and uses no ground truth step labels. Across three agent benchmarks and three critic backbones, \methodshort attains the best calibration (ECE and Brier) and ranking (AUC) in every dataset--critic combination, reducing ECE by up to $54\%$ relative to the strongest training-free baseline.

Yaopei Zeng; Congchao Wang; JianHang Chen; Nan Wang; Yurui Chang; Lu Lin


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Failure attribution for agent loops that learns only from successful trajectories: OAT uses neural controlled differential equations to spot the error step in failed runs, 200-5000x faster than prompting-based attribution with ~20% better F1, needing just 100 success traces, makes in-loop failure localization cheap enough to run continuously.

Use Tracing Agentic Failure from the Flow of Success to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.12747; inspect its method and evaluation before treating results as production evidence.

[2607.12747] Tracing Agentic Failure from the Flow of Success

Failure attribution for LLM-based agentic systems, i.e., identifying which steps in a failure trajectory caused the task to fail, is critical for debugging and improving these systems. Existing approaches either rely on prompting-based pipelines, which are computationally expensive, or require post-training on failure trajectories with step-level error annotations, which are costly to collect and difficult to scale. We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data. To this end, we address unsupervised failure attribution, i.e., training exclusively on successful trajectories and identifying error steps at inference time given a failure trajectory. We propose OAT, which casts this problem as one-class learning with neural controlled differential equations, modeling the dynamical pattern of successful trajectories in latent space. At inference time, each step in a failure trajectory is assigned an anomaly score based on its deviation from the dynamics learned on successful trajectories, which is then used to form a set of error steps. With training on only 100 successful trajectories, experiments show that OAT is 200--5000 $\times$ faster than prompting-based baselines, and, at the same time, consistently outperforms them in both in-domain and out-of-distribution datasets with +20% and +7% F1 scores, respectively, demonstrating that OAT is a promising and efficient direction for diagnosing agentic system failures.

Samuel Yeh; Yiwen Zhu; Shaleen Deep; Sharon Li


Verification is promoted from a final check to a loop-control signal. Study of 204,673 test artifacts comparing agent-generated and human tests: agents win on edge-case and boundary coverage but produce flakier tests (file I/O, non-determinism), a caution that the verification step of a loop can itself become the unreliable component if agent-written checks go unaudited.

Use Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.12068; inspect its method and evaluation before treating results as production evidence.

[2607.12068] Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests in Open-Source Projects

The integration of AI-powered coding agents into Continuous Integration/Continuous Delivery (CI/CD) pipelines has fundamentally altered how software verification is conducted. While these agents successfully automate the test generation, current evaluation benchmarks (e.g., SWE-bench) largely focus on pass-rates rather than the intrinsic quality of the generated tests. This raises the possibility of "stealth technical debt", in which test suites pass execution but do not offer comprehensive coverage or semantic value. We address this methodological gap through a large-scale, empirical comparison of 204,673 test artifacts which comprises of 24,941 human-authored files and 179,732 agent-generated files; sourced from the AIDev dataset. Using the Abstract Syntax Tree (AST) parsing with Python's naive ast module, we implemented a "white-box" static analysis framework to evaluate three quality dimensions: Assertion Strength (RQ1), Edge-Case Coverage (RQ2), and Flakiness Potential (RQ3). Our results present a nuanced inversion of traditional assumptions. AI agents performed better than humans in Edge-Case Coverage, with almost twice the variety of boundary checks (Variety Score: 0.62 vs 0.32) and a higher frequency of null-safety testing (13.40% vs. 8.3%), even though human developers had a slight advantage in Assertion Strength (88.1% strong assertions vs. 85.37% for agents). But this thoroughness comes at a price: due mostly to their reliance on file I/O and non-deterministic logic, agent-generated tests exhibited a higher risk of flakiness (Candidate Rate: 0.41 vs. 0.30). These findings suggest that while AI agents excel at rigorous boundary testing, they lack the "environmental awareness" needed to write stable, hermetic tests.

Preet Jhanglani; Zeel Kaushal Desai; Vidhi Kansara; Eman Abdullah AlOmar


Verification is promoted from a final check to a loop-control signal. First-hand write-up measuring what wrapping a coding agent in an explicit verification loop changes in practice, comparing gated and ungated runs on the same tasks.

Use What a Verification Loop Adds to a Coding Agent: A First Look to measure progress and gate completion with repeatable evidence.

Contextual source from ironbee.medium.com; useful for practice signals or boundary conditions, not independent validation.


Control flow is represented as an inspectable graph rather than an opaque prompt loop. Inference-time selection framework replacing whole-response Best-of-N scoring with budgeted claim-level partial verification, using a signed candidate-factor graph so one checkable finding can support part of the candidate pool while contradicting the rest, with proven logarithmic-vs-linear query savings from shared evidence. Verification-gate primitive, though demonstrated on medical VQA rather than agent harnesses.

Use Best-of-Evidence: Best-of-N Selection under Partial Verification to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.20950; inspect its method and evaluation before treating results as production evidence.

[2607.20950] Best-of-Evidence: Best-of-N Selection under Partial Verification

BoN improves model outputs by sampling several candidates and selecting one with a proxy score, but it assumes that complete candidates can be evaluated reliably. Many vision-language tasks instead provide only partial verification: a finding, span, value, region, or relation may be checkable even when no dependable whole-response verifier exists. Moreover, the same claim may recur across candidates with opposing stances, allowing one observation to support part of the pool and contradict another. We introduce Best-of-Evidence (BoE), an inference-time selection framework that keeps the BoN candidate pool fixed, represents reusable claims with a signed candidate--factor graph, and allocates a limited budget to evidence actions that can change the final choice. BoE formalizes selection under partial verification and provides a practical score-based controller, with the zero-budget case recovering the underlying BoN decision. Theoretically, we show that residual evidence capacity limits any evidence-driven improvement and that shared factor queries can achieve an O(log K) versus {\Theta}(K) query separation in a factor-code model. Common-ledger experiments on four medical VQA settings show that BoE can improve fixed-pool selection and rescue some BoN failures when evidence is reliable, contrastive, and decision-relevant, while also revealing the channel-quality and candidate-generation limits that prevent universal gains.

Cenwei Zhang; Teng Fang; Yuxia Wang; Derek Li; Bryan Dai; Lei You


The trigger or cadence is explicit, making the workflow recurring rather than one-off. Position paper (Levy & Berger, Jul 2026) on the silent-degradation gap of low-code/no-code agents whose dependencies, models, tools, retrieval sources, permissions, schedules, drift after deployment; proposes continuous assurance via dependency mapping, readiness contracts, scheduled checks, and lifecycle governance, with an initial auditor prototype.

Use Toward Continuous Assurance for the Democratization of AI Agent Creation in Industry to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.21495; inspect its method and evaluation before treating results as production evidence.

[2607.21495] Toward Continuous Assurance for the Democratization of AI Agent Creation in Industry

AI agents are increasingly created inside organizations by non-engineering users through low-code, no-code, and conversational development environments. This democratization enables rapid local innovation, but it also creates a reliability gap: agents that appear to users as simple productivity artifacts may depend on changing models, tools, retrieval sources, permissions, prompts, schedules, and external services. These dependencies can cause silent degradation long after deployment, even when no user directly modifies the agent. This paper identifies the reliability challenge created by democratized AI agent creation and proposes a lightweight continuous-assurance framework for citizen-created organizational agents. The framework combines dependency mapping, readiness contracts, scheduled checks, diagnostics, and lifecycle governance to assess whether an agent remains operationally ready under expected conditions. We also present an initial prototype auditor and scenario-based assessment showing how the proposed taxonomy can be translated into practical checks and actionable remediation guidance.


Primary-source operational guidance rather than commentary. Official docs for Anthropic's security-guidance plugin (July 2026), which wires a verification loop into Claude Code's own lifecycle entirely via hooks: a deterministic per-edit pattern check with no model call, a background fresh-context model review of each turn's Git diff that re-prompts Claude with its findings, and a deeper agentic review on every commit or push that reads surrounding code, a worked example of layering independent, non-blocking verification gates inside the agent loop.

Use Catch Security Issues as Claude Writes Code to measure progress and gate completion with repeatable evidence.

Primary official documentation from code.claude.com; use it for current product or standard behavior.

Catch security issues as Claude writes code - Claude Code Docs

Install the security-guidance plugin to have Claude review its own code changes for vulnerabilities and fix them in the same session.


Checkpointed state makes long-running agent work recoverable across failures. Gated human-in-the-loop multi-agent architecture for domains lacking any cheap machine-checkable correctness signal, using economic theory development as the case study. Specialized diagnostic gates, human checkpoints, and a shared workspace of inspectable intermediate records replace full automation; across five tasks the gated variant beat an ungated baseline on failure severity (1.58 to 1.16) and usefulness (2.60 to 3.10). Useful as a design study of human gates where automated verifiers do not exist; single-domain case study with no independent traction yet (Jul 2026).

Use pAI-Econ-claude: A Gated Human-in-the-Loop Multi-Agent Architecture for AI-Assisted Economic Theory Development to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.21268; inspect its method and evaluation before treating results as production evidence.

[2607.21268] pAI-Econ-claude: A Gated Human-in-the-Loop Multi-Agent Architecture for AI-Assisted Economic Theory Development

In many social-science research tasks, such as economics, LLM-based agents must produce outputs for which no cheap, task-complete, machine-readable correctness signal exists. This creates a distinctive reliability problem for multi-agent systems: how should generation, critique, coordination, and human judgment be organized when no component can certify the final result? We address this problem through pAI-Econ-claude, a gated, human-in-the-loop multi-agent architecture for AI-assisted economic theory development. Agents coordinate through a shared workspace of inspectable intermediate records; specialized gates diagnose targeted failure modes and recommend loopbacks without certifying correctness; and human checkpoints retain authority over decisions that are costly to reverse. We evaluate the architecture on five matched economic-theory tasks against an ungated baseline. Two evaluators blinded to configuration agreed on all five pairwise rankings, preferring the gated architecture in four tasks and the baseline in one. Mean failure severity fell from 1.58 to 1.16, while overall usefulness rose from 2.60 to 3.10. The largest observed gain occurred when a reality check rejected a false market-structure premise and a proof review prompted revision of a false welfare claim. The negative case shows that scaffolding can also compress an economically important mechanism too aggressively. The results support a bounded claim: gated oversight improves the auditability of AI-assisted economic theory without substituting for formal verification, and the allocation of irreversible human judgment is a more informative design variable than pure agent autonomy. The workflow is publicly available at https://github.com/maxwell2732/pAI-Econ-claude.


Review Loop

Checkpointed state makes long-running agent work recoverable across failures. First-party extension from earendil-works, the org behind the pi coding agent (77k stars): keeps a native review window open while the agent works, and submitting a review records the current workspace as a session-backed checkpoint so the next pass diffs only the increment ("Since review" vs "vs HEAD" modes), turning one-shot PR review into a persistent human-verification loop over continuous agent output.

Use Review Loop to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (136 stars; 19 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - earendil-works/pi-review-loop: A persistent incremental diff review loop for pi · GitHub

A persistent incremental diff review loop for pi. Contribute to earendil-works/pi-review-loop development by creating an account on GitHub.


State persistence is explicit enough for repeated runs and handoff. Directly attacks the core Loop Engineering assumption that more generate-test-revise iterations equal more reliability. Over 900 three-revision trajectories on HumanEval repairs, current correctness DROPS from 0.820 after one revision to 0.673 after two even as ever-correct rises to 0.847 -- the loop finds the patch and then loses it.

Use Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.24604; inspect its method and evaluation before treating results as production evidence.

[2607.24604] Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair

Generate--test--revise loops are common in coding agents, but repetition alone provides no reliability guarantee. We study the gap between finding a correct patch and retaining, verifying, and submitting it. A sealed five-seed study over 30 HumanEval repairs produces 900 three-revision trajectories. Under forced revision, current correctness with current traces falls from 0.820 after one revision to 0.673 after two, although ever-correct rises to 0.847. Two common-state studies use 2,430 branches from identical frozen programs to remove post-treatment risk-set bias. In a prespecified 14B replication, stale traces harm 34/135 correct starts versus 4/135 with current traces, a 22.2-point increase (task-cluster 95\% CI $[8.9,37.0]$, exact Holm $p=0.0337$). A prospective 540-rollout policy eliminates observed correct-start harm but reduces wrong-start repair and fails its joint criterion. Repository experiments over 24 bugs and four coder stacks expose floor effects and component heterogeneity without Holm-significant effects. We therefore separate admission, preservation, grounded certification, competence, and liveness. We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation that binds verifier evidence to exact code states, preserves verified checkpoints, and emits auditable admission receipts. The implementation is an executable specification and conformance artifact, not evidence of improved repair competence or calibrated verifier dependence.


Verification is promoted from a final check to a loop-control signal. Names and measures the verifier-deployment gap: when a self-improving agent controls both the artifact it rewrites and the tests that judge the rewrite, self-assigned scores stay near-perfect while sealed deployment performance degrades. Introduces a Sealed Exogenous evaluation protocol and asks the practical question -- how little exogenous trust suffices to stop real regressions from shipping. This is the sharpest current statement of why closed self-improvement loops need an external verification anchor.

Use Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.24300; inspect its method and evaluation before treating results as production evidence.

[2607.24300] Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents

Self-improving agents accumulate capability by repeatedly rewriting procedural policies, controllers, or heuristic rules. They typically rely on self-authored tests or metrics to decide whether to accept subsequent edits. The agent controls both the optimized object and its verifier. As a result, self-assigned scores can remain near perfect while real deployment performance degrades or stays low. We study this problem through the verifier--deployment gap. This gap refers to the discrepancy between an agent's self-authored verification signal and a sealed deployment evaluation that the agent cannot observe or access. We ask how self-authored verification fails under iterative policy-and-test rewriting, how the failure changes with capability, and how little exogenous trust is sufficient to prevent real regressions from being deployed. To address this problem, we introduce a Sealed Exogenous Acceptance Loop (SEAL). SEAL retains self-authored tests but compares each candidate with the incumbent through a fixed harness-side audit. The agent cannot author or inspect the audit, receives only accept/reject, and the whole incumbent state is retained after a clear regression. Our experiments show that this problem often appears in heuristic learning settings. These settings require trial-and-error discovery of the target objective. We further find that failures of self-written verification are stratified by capability. Weaker agents tend to damage previously acquired strategies behind easy self-tests. Stronger agents are more stable, but they still mismeasure the deployment distribution. Standard self-written constraints do not reliably close this gap. In contrast, SEAL outperforms unprotected baselines across six models and three random seeds. Reliable self-improvement need not abandon self-verification, but it requires at least one deployment-acceptance signal outside the agent's control.

Diandian Guo; Cong Cao; Fangfang Yuan; Yingqi Wang; Yueshan Wang; Dakui Wang


The work targets tasks that exceed a single context window or prompt session. Long-horizon web agents drift: a trajectory stays locally plausible long after the state or plan assumption stopped supporting the instruction. FCPAgent makes each plan step a Falsifiable Commitment Unit -- a subgoal grounded in a reusable skill plus confirming evidence, falsifying evidence, and a confidence score -- and runs a plan-test-repair loop whose hybrid commitment testing checks candidate actions before they mutate the browser and observations after execution. Writing the falsification condition into the plan is a genuinely transferable loop pattern.

Use Falsifiable Commitment Planning for Self-Correcting Web Agents to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.24167; inspect its method and evaluation before treating results as production evidence.

[2607.24167] Falsifiable Commitment Planning for Self-Correcting Web Agents

Long-horizon web agents often go off track before final failure: a trajectory can remain locally plausible even after the current state, reused skill, or plan assumption no longer supports the user instruction. Existing agents can plan, reflect, or reuse experience, but their plans rarely specify the evidence under which an active step should still be trusted. We propose FCPAgent, a falsifiable commitment planning framework for robust long-horizon web agents. FCPAgent represents each plan step as a Falsifiable Commitment Unit (FCU): a subgoal grounded in a reusable skill, together with confirming evidence, falsifying evidence, and a confidence score. Execution is organized as a plan-test-repair loop. The hybrid commitment testing module checks candidate actions before they modify the browser and checks observations after execution; for efficiency, it combines lightweight evidence matching with LLM-based diagnostic verification. When evidence falsifies a commitment, scope-aware repair localizes the contradiction to the execution, skill, or planning level and revises the smallest adequate part. On WebArena, FCPAgent achieves a 13.8% relative improvement in average success over the strongest baseline, with especially large gains on long-horizon tasks.


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. A Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those -- every verdict decided mechanically so no model judges another. The loop killed 105 mutants one-shot generation missed and lost none.

Use Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.23002; inspect its method and evaluation before treating results as production evidence.

[2607.23002] Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop

Large language models increasingly write both code and the tests meant to check it; coverage records what ran, not what was verified. We study an adversarial test-hardening loop under a mechanical oracle: a Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those, with every verdict decided mechanically, so no model judges another's output. In Experiment 1, on five Python subjects (one same-lineage-loop cell could not be scored), the loop killed 105 mutants that one-shot generation missed and lost none, and the cross-lineage-Critic question returned a pre-declared null. The central finding was an autopsy: an earlier analysis reported a cross-lineage effect at p = 9.5e-66 that was an instrument artifact, an output cap silently truncating the verbose model, caught only by adversarial review of the completed analysis. Review then found a further confound, each arm resampling its own initial suite; Experiment 2 removes it. Under a pre-registered frozen-shared-round-0 design (five replicates on each of four subjects, seeds committed in advance), same-lineage Critic rounds killed 78% of the survivors the frozen initial suite left standing (mean incremental kill rate 0.783, 95% cluster-bootstrap interval [0.592, 0.935]), a within-replicate causal estimate; the cross-provider configuration showed a positive pilot difference (rate gap 0.178, 95% interval [0.039, 0.347]; magnitude dominated by a single replicate) at 5.5x lower arm cost. This compares two named model-provider-harness configurations, not an isolated lineage effect: part of the gap is one configuration's receipted operational failures, including truncation recurrences, now detected and scored rather than laundered. Cross-model comparisons can inherit the asymmetries of the harness that runs them. We release both protocols, all receipts, and the analysis code.

26 pages. Two pre-registered experiments; protocols, all run receipts, and analysis code at https://github.com/Jott2121/crucible


Untrusted intake is treated as a loop-level security boundary. Tool-call arguments carry wildly different risk -- untrusted content may safely shape an email body but must never determine a recipient, account, command, or credential -- yet standard statistical control certifies the action as a whole, letting rare high-risk field failures hide behind benign arguments. Adds a calibration layer wrapping any per-field detector with separate thresholds and risk budgets per semantic argument role, avoiding the alpha*p_r effective-budget penalty that aggregate-only certification requires. Evaluated on AgentDojo and InjecAgent.

Use Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.24343; inspect its method and evaluation before treating results as production evidence.

[2607.24343] Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls

Language-model agents act through structured tool calls whose arguments carry very different risks: untrusted content may legitimately shape an email body but should never set a recipient, account, command, or credential. Existing conformal risk control methods certify a tool call as a whole, so a failure in one rare high-risk field can be averaged away by the many benign arguments around it, leaving the argument that causes harm uncertified. We introduce role-stratified per-field conformal risk control, a calibration layer that wraps any per-field detector and assigns a separate threshold and risk budget to each semantic argument role. We show that aggregate certification pays a price of coarseness, tightening a rare role's effective budget in proportion to how often that role appears, whereas role-stratified calibration certifies each sufficiently sampled role directly with a finite-sample guarantee and pools the rarest roles. Across AgentDojo and InjecAgent with six language models, our method achieves the most consistent role-specific budget compliance among the methods we evaluate under model and attack transfer, detector noise, gradual drift, unseen tool suites, and adaptive attacks, providing formal per-role guarantees under exchangeability or after recalibration. These results suggest that structured tool calls should be certified at the semantic-role level, not the whole action.

Md Ashikur Rahman; Md Arifur Rahman; Niamul Hassan Samin; Khandaker Rifah Tasnia; Md Hasibul Amin; Sifat Rahman Ahona; Juena Ahmed Noshin


Verification is promoted from a final check to a loop-control signal. RLVR works where correctness is deterministically checkable and stalls everywhere else, leaving open-ended tasks dependent on preference data, reward models, or LLM judges with their bias, capability ceilings, and inference cost. RLSVR borrows the self-supervised trick of constructing pretext tasks: transform an open-ended task into a verifiable proxy environment whose internal rules and interaction outcomes supply the reward signal directly. A route to closing the verification loop where no natural verifier exists.

Use From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.23802; inspect its method and evaluation before treating results as production evidence.

[2607.23802] From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verifiable. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference costs. Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a Self-PlaY Reinforcement Learning method inspired by social deduction game Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality. Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code have been released at https://github.com/wangqinsi1/RLSVR/tree/SpyRL.

Qinsi Wang; Jing Shi; Huazheng Wang; Kun Wan; Yiran Wu; Bo Liu; Qingyun Wu; Hai Helen Li; Yiran Chen; Handong Zhao; Wentian Zhao


Durable execution and replay are treated as first-class loop infrastructure. Verification-gated memoization taken to its limit: the model stays frozen while a persistent store accumulates only those solutions that passed an independent verification step, so later encounters with a seen task replay a checked answer instead of re-deriving it. Read the headline accuracy and token figures as properties of cache hits on verified work, not as general model capability.

Use A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.23806; inspect its method and evaluation before treating results as production evidence.

[2607.23806] A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever

Improving a language model today means retraining it: enormous compute, a new opaque model each cycle, non-deterministic output. We take the opposite path: the model stays frozen, and a persistent memory of verified solutions grows beside it. Once a problem family is solved and has passed an independent verification step that never consults the answer key, every new instance of that family is answered at zero generation tokens, bit-exact, deterministically. Across 180 fresh instances spanning nine problem families, four architectures from four vendors - dense and mixture-of-experts - each score 180/180 at zero generation tokens per answer: execution-bound capability decoupled from parameter scaling. A negative control attributes the capability fully to the memory: emptied, it solves nothing. The same verify-before-store contract holds for open-ended reasoning: 88/88 consistency-gated acceptances across all four models, machine-checked formal proof, and reasoning-method transfer at 77/80. Memory selection takes 1.4 microseconds; a full reuse completes in 6-23 ms at 36 mWh. Approximate similarity retrieval selects the wrong item 94.3% of the time on a 4,500-item verified store where exact addressing makes zero errors. The store also serves as working context at a scale no shipped engine matches: a 6,000,000-token movable window on a single 46 GB GPU at flat memory, where vLLM stops at 30,399 tokens and SGLang silently truncates past 32,000. On published benchmarks, frontier models remain far ahead of any 12B at raw from-scratch reasoning; on everything this system has solved and verified, the comparison inverts: a frontier API call pays a fresh generation pass on every query, forever, while verified reuse costs zero tokens and returns the identical bits every time. A public testbench with free, rate-limited access accompanies this report: https://corbenic-galahad-bench.hf.space

Industry experience report. 14 pages, 8 figures. Public testbench: https://corbenic-galahad-bench.hf.space; companion repository with SHA-256 provenance manifest: https://github.com/corbenicai/galahad-bench


The work targets tasks that exceed a single context window or prompt session. Pairs architecture and behavior specifications as executable artifacts so a code agent can verify design completeness and hold design-implementation consistency across a long repository-level feature build, beating Claude Code baselines. Makes the spec a runnable gate rather than a prompt preamble.

Use CodeSpec: Dual Executable Specifications for Agentic Long-Horizon Feature Development to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.26777; inspect its method and evaluation before treating results as production evidence.

[2607.26777] CodeSpec: Dual Executable Specifications for Agentic Long-Horizon Feature Development

LLM-based code agents have advanced repository-level software development through iterative interaction with codebases and tools. However, feature development requires integrating new behaviors into existing architectures through coherent cross-component functional chains. Existing agents typically derive such chains through free-form reasoning, often producing unreliable feature designs with incomplete functional chains. Moreover, textual designs are difficult to verify and enforce, making it challenging to maintain design-implementation consistency throughout long-horizon development. We propose CodeSpec, a dual executable specification method for repository-level feature development. It builds reliable functional chains from evidence pairing sub-requirement semantics with repository architectures, then compiles them into complementary architecture and behavior specifications that check chain completeness and correctness while preserving design-implementation consistency over long interactions. On FeatureBench, which targets feature development in existing repositories, CodeSpec achieves 70.7%, 55.0%, and 49.9% pass rates under DeepSeek-V4-Pro, outperforming representative baselines such as Claude Code. Results on the repository generation benchmark NL2Repo-Bench further demonstrate its generalizability.

Peiding Wang; Li Zhang; Fang Liu; Taichuan Li; Yinghao Zhu


The work turns loop quality into a measurable task or score. Splits from-scratch program synthesis into a behavioral-specification elicitation stage and a code synthesis stage, improving test pass rates and exploration coverage across models and benchmarks. Useful evidence that the specification step deserves its own loop iteration instead of being folded into generation.

Use SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.27167; inspect its method and evaluation before treating results as production evidence.

[2607.27167] SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.

Yihao Chen; Shi Chang; Feng Lin; Khaled Chawa; Boyuan Chen; Shaowei Wang; Ahmed E. Hassan


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Shows a competent agent silently converts an injected metadata-borne defect (e.g. a stale price) into a costly action about 60% of the time, and proposes a metadata-aware pre-action gate plus downstream remediation. Names a failure mode, the defect is invisible to the agent, so more capability does not help, that pure output-checking gates cannot catch.

Use SARC-DQ: Runtime Data-Quality Gating for Agentic AI: Silent Evidence Defects, the Incompetence Shield, and Downstream-Only Remediation to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.26313; inspect its method and evaluation before treating results as production evidence.

[2607.26313] SARC-DQ: Runtime Data-Quality Gating for Agentic AI: Silent Evidence Defects, the Incompetence Shield, and Downstream-Only Remediation

Agentic systems act, so a defect in the evidence they retrieve becomes a wrong action with a currency cost. The most dangerous enterprise defects are metadata-borne: a stale price or a superseded record, perfectly well-formed in the payload and betrayed only by freshness, lineage, or provenance. Such a defect never enters the agent's context, and an agent cannot doubt data it cannot see. On a priced replenishment benchmark, a competent agent silently converts an injected metadata-borne defect into a costly action about 60% of the time, with zero data-quality flags and behavioral doubt markers at chance (AUC <= 0.50). Across four model tiers spanning roughly 15x in inference price, the rate stays flat: capability does not buy skepticism. A metadata-aware pre-action gate with downstream-only remediation recovers the loss fully on the signals its predicates cover and not at all on those they miss. A model-free oracle derived from the task's decision geometry tracks the measured rates with MAE 0.015 (Pearson r = 0.876, interval coverage 15/16 cells), giving the flat ladder an analytical form. Evidence integrity is a systems axis distinct from model capability; mitigation depends on enforcement placement and predicate coverage. Code, frozen results, and a deterministic analysis pipeline: https://github.com/besanson/dqSarc


Verification is promoted from a final check to a loop-control signal. HalluProp estimates individual agent failure and emergent system-level hallucination risk before inter-agent interaction begins, by scoring agent-task semantic alignment and modeling propagation across the communication network. Shifts multi-agent verification from post-hoc detection to admission control.

Use Before Agents Speak: Pre-hoc Failure Risk Inference in Multi-Agent Systems to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.26836; inspect its method and evaluation before treating results as production evidence.

[2607.26836] Before Agents Speak: Pre-hoc Failure Risk Inference in Multi-Agent Systems

LLM-based multi-agent systems (MAS) have exhibited remarkable capabilities in collaborative reasoning and decision-making, yet their interconnected communications introduce new systemic risk: localized hallucinations can propagate along agent communication chain, amplify through interactions, and ultimately trigger cascading failures. Existing countermeasures predominantly follow a post-hoc paradigm, identifying failures only after unsafe behaviors emerge, by which time harmful effects may have already spread throughout the agent network. To tackle this problem, we investigate a complementary pre-hoc approach and propose HalluProp, a Propagation-aware Hallucination inference framework that estimates individual agent failures and emergent system-level hallucination risks before inter-agent interaction. First, we model intrinsic hallucination risks by identifying fine-grained semantic misalignment between agent roles and task queries. We then characterize inter-agent risk propagation by modeling both semantic influence and communication topology. Finally, we integrate these two risks via a differentiable Noisy-OR inference mechanism to derive a systemic diagnosis. Extensive experiments show that HalluProp accurately localizes faulty agents, achieving an average AUROC of 84.6%, while enabling sub-second diagnosis with over $65\times$ speedup over post-hoc methods. By facilitating early intervention through upstream screening, HalluProp effectively complements post-hoc methods, highlighting the potential of pre-hoc risk inference for building more reliable multi-agent systems.

Shi Lin; Chenpei Wang; Peng Qian; Dezhang Kong; Minghao Li; Yufeng Li; Xun Wang


Verification is promoted from a final check to a loop-control signal. Multi-agent harness that compiles natural-language regulations into auditable verification code, making compliance checks transparent and rerunnable, with open models matching frontier APIs at lower cost. The regulation-to-executable-check pipeline generalizes well beyond its building-code case study.

Use ARCHER: Agentic Rule and Compliance Harness for Executable Regulations to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.25566; inspect its method and evaluation before treating results as production evidence.

[2607.25566] ARCHER: Agentic Rule and Compliance Harness for Executable Regulations

Verifying building compliance requires validating thousands of rules against large Building Information Modeling (BIM) designs, which is laborious, capital-intensive, and unscalable. Existing Automated Compliance Checkers (ACCs) are often difficult to generalize across different scenarios, as they are typically developed for highly specific rule sets and use cases. In addition, many ACCs are proprietary, meaning the underlying verification code is not released to end users, so users cannot verify whether their regulatory intent can be accurately captured. We introduce ARCHER (Agentic Rule and Compliance Harness for Executable Regulations), a test-driven, deterministically orchestrated multi-agent program-synthesis harness that generates auditable verification code from regulatory Codes of Practice, enabling transparent, adaptable, and scalable compliance checking. To characterize what makes agentic synthesis work, we evaluate a taxonomy of six harnesses of increasing agentic sophistication across four backbone models, spanning realistic data-governance tiers (from frontier third-party APIs to a fully on-premise open-weights model) on a novel dataset derived from real-world compliance scenarios. ARCHER's deterministic multi-agent orchestration achieves the highest accuracy for every backbone, improving mean union accuracy by 82% over a naive single-pass prompting baseline. Our cost-accuracy analysis further shows that using the ARCHER harness, a self-hosted open-weights model can reach 97.8% of frontier-API accuracy at a quarter of the cost, making data-sovereign compliance checking practical.

Chiraag Singh Anand; Xue Wen Tan; Lionel Teo; Eric Tan


Verification is promoted from a final check to a loop-control signal. Packages provenance capture as an executable agent skill (aiprov) that records activities, claims, and sources during production while reserving verification authority for humans. Answers the audit question every long-running agent loop eventually faces: who checked this, and can you prove it.

Use F(AI)2R: Who Did What, and Who Checked? Verifiable AI Provenance as an Executable Skill to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.25637; inspect its method and evaluation before treating results as production evidence.

[2607.25637] F(AI)2R: Who Did What, and Who Checked? Verifiable AI Provenance as an Executable Skill

F(AI)2R is FAIR research with AI in the loop, twice: an AI-assisted authoring pass and a machine-readable audit pass over every artefact. AI systems now draft, refactor, and verify research artefacts, yet their contributions are rarely recorded in a form a later human or machine can audit. Building on the original F(AI)2R experiment, we generalize its provenance model beyond scholarly writing into aiprov, a PROV-O extension covering any AI-in-the-loop artefact, and we package the method as an executable skill that an AI agent operates itself: setup asks the human operator for their ORCID ID, resolves their identity from the public registry, and scaffolds continuous integration that gates every push on graph conformance and publishes the current build of this very paper. The paper is its own case study. Every activity, claim, and source in its production is recorded in the repository's provenance graph under two invariants: no parentless claim, and verification rungs that only humans may grant.


Codex Security

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. OpenAI's Codex Security CLI and TypeScript library, giving a coding agent a first-party path to run security review over its own changes instead of depending on an external scanner bolted onto the loop.

Use Codex Security to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (9,222 stars; 634 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - openai/codex-security: OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security · GitHub

OpenAI's Codex Security CLI and TypeScript SDK for finding, validating, and fixing security vulnerabilities. npm: https://www.npmjs.com/package/@openai/codex-security - openai/codex-security


Context is managed as durable loop state rather than a single prompt payload. GitHub changelog, 2026-07-29: the automated review gate becomes programmable and context-aware for all Copilot Pro, Pro+, Business, and Enterprise users.

Use Copilot Code Review: Agent Skills and MCP Now Generally Available to measure progress and gate completion with repeatable evidence.

Primary documentation from a platform, SDK, standard, or framework; strong implementation signal.

Copilot code review: Agent skills and MCP now generally available - GitHub Changelog LinkedIn icon Instagram icon YouTube icon X icon TikTok icon Twitch icon GitHub icon

Copilot code review support for agent skills and MCP servers is now generally available for all Copilot Pro, Pro+, Business, and Enterprise users. Previously announced in public preview, these capabilities…


old-coder

Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Markdown skill that makes a coding agent prove its work instead of asking you to read every line: the agent pushes each change through a spec, then a gauntlet, then evidence. Plain instructions rather than a runtime, so it works with Claude Code, Codex CLI, Cursor, Aider, or a custom loop.

Use old-coder to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (331 stars; 28 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - AmazingAng/old-coder: An old coder's strategy for the agent era: don't read the code — make it run the gauntlet. Evidence-first development skill for coding agents, inspired by Uncle Bob. · GitHub

An old coder's strategy for the agent era: don't read the code — make it run the gauntlet. Evidence-first development skill for coding agents, inspired by Uncle Bob. - AmazingAng/old-coder


Verification is promoted from a final check to a loop-control signal. Attacks the assumption every verification gate rests on: that a passing test is evidence about the reported defect.

Use Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug? to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.28871; inspect its method and evaluation before treating results as production evidence.

[2607.28871] Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?

When a repair agent runs a test and sees it pass, the result is treated as evidence about the reported defect. We measure how often that treatment is warranted. BSG-VA (buggy-state/candidate-state/gold-fix validation analysis) captures each validation command at its exact working-tree state, extracts a test-only patch, and replays the command on the original buggy code (B), the candidate state (S), and the developer gold fix (G). The captured outcome and the replay results assign every event an evidence role, from gold-aligned bug-discriminating through regression-only to misleading. Across 3,730 events in 643 rollouts on 110 tasks, 46.0% of positive comparable events carry no bug-discriminating information; 23.8% of baseline rollouts, with no feedback injected, close with a patch whose entire positive evidence base is of this kind. A three-arm experiment tests whether returning the B-replay outcome to the agent changes this pattern. Bug-contrast feedback reduces evidence-inadequate closure by 7.8 percentage points relative to an attention-matched reminder (p = 0.0029) and raises bug-discriminating evidence by 7.4 points (p = 0.011), with no detectable cost to repair success. Both estimates fall below the prespecified 10-percentage-point smallest effect size of interest, so practical magnitude remains uncertain. Roughly a third of the improvement traces to the reminder alone; across two exploratory replications, varying the scaffold and the model, the B-replay content adds a detectable increment only with gpt-5.6-sol under the unconstrained tool-use loop. BSG-VA applies post hoc to any replayable repair trajectory that preserves the required code states and execution environment. Keywords: program repair agents, validation evidence, test adequacy, large language models, software quality, controlled experiment.


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. Names premature commitment as a distinct failure pattern: coding agents edit source or submit patches before examining enough repository evidence to justify the change.

Use Preventing Premature Commitment in Coding Agents with an Evidence-Conditioned Execution Layer to measure progress and gate completion with repeatable evidence.

Research source arXiv:2607.28815; inspect its method and evaluation before treating results as production evidence.

[2607.28815] Preventing Premature Commitment in Coding Agents with an Evidence-Conditioned Execution Layer

LLM-based coding agents often edit source code or submit patches before examining enough repository evidence to justify the change, a failure pattern we call premature commitment. We present ECLoop, an execution layer that interposes between the agent and the repository to enforce evidence-conditioned execution. For each task, ECLoop uses the issue description and repository structure to compile a set of conditions specifying what the agent should observe before each type of code modification or patch submission. During execution, ECLoop tracks which conditions the agent's runtime trajectory has satisfied and postpones any proposed action whose required conditions remain unmet. Evaluated on all 500 instances of SWE-bench Verified with two language models and two agent scaffolds, ECLoop raises Pass@1 by 4.8-11.8 percentage points without model retraining or scaffold changes. Ablation experiments show that each of ECLoop's three operations contributes distinct value and that structured evidence conditions outperform an equivalent natural-language summary. These gains come at no additional inference cost: by redirecting the agent before it pursues unsupported actions, ECLoop lowers average token consumption by up to 12.1%.

Yisen Xu; Chenglin Li; Zehao Wang; Jinqiu Yang; Tse-Hsun Chen


Treats feedback, telemetry, and deterministic artifacts as loop-control gates. OpenAI's Aug 1 writeup of ten previously-open problems in high-dimensional geometry, coding theory, group theory, operator algebras, arithmetic circuit complexity, quantum complexity, lattices and extremal combinatorics, closed by an internal Astra model.

Use Ten advances in mathematics and theoretical computer science to measure progress and gate completion with repeatable evidence.

Contextual source from openai.com; useful for practice signals or boundary conditions, not independent validation.

Ten advances in mathematics and theoretical computer science | OpenAI

OpenAI shares new results on long-standing open problems in mathematics and theoretical computer science, including advances in geometry, cryptography, and complexity.