Amp: Custom Agents

The work separates roles across agents, verifiers, or orchestration layers. Amp's plugin-defined custom agents that run as the main agent or as subagents, spawn parallel workers, join tool pipelines, and use thread actions to build background review threads that report results back to a parent thread.

Use Amp: Custom Agents to choose an implementation surface for repeatable agent work.

Contextual source from ampcode.com; useful for practice signals or boundary conditions, not independent validation.

Plugins can now create agents, run them once, and keep talking to their threads.


Orchestration And Multi-Agent Delegation

AgentsMesh

Workspace isolation is part of the loop design, not an afterthought. Self-hosted control plane for running fleets of coding agents across your own machines, with scheduling, per-pod Git worktree isolation, Kanban work tracking, and merge-request integration.

Use AgentsMesh to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (2,308 stars; 236 forks; NOASSERTION license; updated 2026-08-07); popularity is context, not proof of reliability.

Choose runtimes, tools, and delegation surfaces.

GitHub - AgentsMesh/AgentsMesh: The AI Agent Workforce Platform. Run a hundred AI coding agents across your own machines — schedule, isolate, and steer them all from one console. · GitHub

The AI Agent Workforce Platform. Run a hundred AI coding agents across your own machines — schedule, isolate, and steer them all from one console. - AgentsMesh/AgentsMesh


Bernstein

Workspace isolation is part of the loop design, not an afterthought. Deterministic Python orchestrator that runs parallel CLI coding agents in isolated Git worktrees, gates merges on tests, lint, and type checks, and records every scheduling decision in a tamper-evident audit log.

Use Bernstein to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (799 stars; 93 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - sipyourdrink-ltd/bernstein: Deterministic orchestrator for CLI coding agents (Claude Code, Codex, Gemini CLI, +40 more). No model in the coordination loop, so parallel runs in per-task git worktrees replay byte-identically. Signed lineage plus an opt-in HMAC audit chain a reviewer checks offline, without rerunning it. Cluster mode, air-gap deploy. https://bernstein.run · GitHub

Deterministic orchestrator for CLI coding agents (Claude Code, Codex, Gemini CLI, +40 more). No model in the coordination loop, so parallel runs in per-task git worktrees replay byte-identically. Signed lineage plus an opt-in HMAC audit chain a reviewer checks offline, without rerunning it. Cluster mode, air-gap deploy. https://bernstein.run - sipyourdrink-ltd/bernstein


Aeon

Persistent memory is treated as an external runtime artifact. Autonomous agent framework that runs Claude Code unattended on GitHub Actions, dispatching skills on cron or reactive triggers with per-run quality scoring, persistent memory, and self-healing skill repair.

Use Aeon to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (610 stars; 218 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - aeonfun/aeon: The most autonomous agent framework. No approval loops. No babysitting. Configure once, forget forever. · GitHub

The most autonomous agent framework. No approval loops. No babysitting. Configure once, forget forever. - aeonfun/aeon


h5i

Workspace isolation is part of the loop design, not an afterthought. Gives each coding agent an isolated sandboxed Git worktree, dispatches one task to a team that peer-reviews each other's candidates, then replays and tests each candidate with a neutral verifier before merging the winner.

Use h5i to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (504 stars; 45 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - h5i-dev/h5i: Integrated sandbox for coding agents. Workspace, shell, dependencies, dev server, and browser in one disposable environment. Local-first. No hosted sandbox. No SaaS account required. · GitHub

Integrated sandbox for coding agents. Workspace, shell, dependencies, dev server, and browser in one disposable environment. Local-first. No hosted sandbox. No SaaS account required. - h5i-dev/h5i


Context is managed as durable loop state rather than a single prompt payload. A shepherd agent with global context steers a population of search agents that each work in their own Git branch with local context, countering the context accumulation of solo long-running agents and matching or beating baselines on 13 of 15 open-ended discovery tasks.

Use SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.02807; inspect its method and evaluation before treating results as production evidence.

[2607.02807] SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery

Long-running coding agents such as autoresearch can persistently discover optimizations for open-ended problems. However, they tend to converge onto a single high-level approach, then proceed with low-level edits while missing other superior approaches to the problem. We hypothesize two harness-level design choices contribute to this behavior: accumulating context in a single long-running agent and only exposing a single program state to edit. We introduce SwarmResearch, an orchestrator-subagent harness in which a Shepherd Agent uses global context to steer a population of Search Agents, each operating with local context in their respective git branch. On open-ended optimization tasks, SwarmResearch discovers better or comparable solutions to state-of-the-art LLM-guided evolution and multi-agent techniques on 13/15 tasks, driven by higher-level exploration. Compared with fixed scaling of serial and parallel agents, SwarmResearch's orchestrator-guided scaling discovers better-performing solutions by adapting parallelism at different search depths.

Yuvraj Virk; Zack Edds; Chunqiu Steven Xia; Lingming Zhang


Scaling Long-Running Autonomous Coding

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Cursor traces the coordination designs behind week-long autonomous coding runs, from flat agents with locking to optimistic concurrency to a planner/worker/judge hierarchy, letting hundreds of concurrent workers push to one branch on projects exceeding a million lines.

Use Scaling Long-Running Autonomous Coding to choose an implementation surface for repeatable agent work.

Contextual source from cursor.com; useful for practice signals or boundary conditions, not independent validation.

Scaling long-running autonomous coding · Cursor

We've been experimenting with running coding agents autonomously for weeks at a time.


babysitter

Durable execution and replay are treated as first-class loop infrastructure. Harness-agnostic orchestration framework that runs agent workflows as process-as-code with mandatory enforcement stops after every step, quality-convergence loops that re-run verify-and-refine until thresholds pass, human-approval breakpoints, and an immutable event-sourced journal for deterministic replay across 12 coding harnesses.

Use babysitter to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (1,653 stars; 96 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - a5c-ai/babysitter: Babysitter enforces obedience on agentic workforces and enables them to manage extremely complex tasks and workflows through deterministic, hallucination-free self-orchestration · GitHub

Babysitter enforces obedience on agentic workforces and enables them to manage extremely complex tasks and workflows through deterministic, hallucination-free self-orchestration - a5c-ai/babysitter


claude-code-merge-queue

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Local FIFO merge queue that serializes parallel Claude Code agents landing on a shared codebase, with a machine-wide build lock and a landing gate that blocks the integration branch until a configured check command passes, wired into the WorktreeCreate hook.

Use claude-code-merge-queue to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (120 stars; 3 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - funador/claude-code-merge-queue: The local merge queue for parallel Claude Code agents · GitHub

The local merge queue for parallel Claude Code agents - funador/claude-code-merge-queue


Devin can now manage Devins

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Cognition's multi-Devin architecture where a coordinator Devin scopes a task into pieces, delegates each to a managed Devin running in its own isolated VM with terminal, browser, and dev environment, monitors progress, resolves conflicts, compiles the results, and reads workers' full trajectories to learn what worked and where they got stuck.

Use Devin can now manage Devins to choose an implementation surface for repeatable agent work.

Contextual source from cognition.com; useful for practice signals or boundary conditions, not independent validation.

Devin can now break down large tasks and delegate them to a team of managed Devins, with each running in its own isolated VM in parallel.


pilotfish

Verification is promoted from a final check to a loop-control signal. Multi-model orchestration layer for Claude Code where the frontier model plans, decides, and reviews while cheaper models execute volume work through six global subagent roles, with quality guarded by fresh-context verifier subagents rather than expensive models everywhere and graceful degradation when the frontier model is unavailable, shipped as three config files with no runtime code.

Use pilotfish to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (584 stars; 40 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - Nanako0129/pilotfish: Multi-model orchestration layer for Claude Code — the frontier model plans, cheaper models execute, verification guards quality. One-prompt install. · GitHub

Multi-model orchestration layer for Claude Code — the frontier model plans, cheaper models execute, verification guards quality. One-prompt install. - Nanako0129/pilotfish


fable-advisor

Verification is promoted from a final check to a loop-control signal. Claude Code plugin implementing an architect pattern where Fable 5 owns specs, decomposition, and verification while routing implementation to cheaper lanes (Grok 4.5 by default, GPT-5.6 via Codex CLI, or Sonnet/Opus fallback), with cross-vendor review and optional racing of two implementers on the same spec.

Use fable-advisor to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (702 stars; 65 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - DannyMac180/fable-advisor: Claude Fable as an orchestrator for Opus, GPT and Grok · GitHub

Claude Fable as an orchestrator for Opus, GPT and Grok - DannyMac180/fable-advisor


agent-chief

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Local-first chief-of-staff layer that guards human attention across a fleet of agents and alerts: hard rules kill noise fast while a cache-stable LLM judge batches, blocks, or escalates what remains.

Use agent-chief to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (1,016 stars; 4 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - SmileLikeYe/agent-chief: Attention is your scarcest resource. Chief is the local-first layer that guards it — turning every agent, alert, and feed into one honest call: interrupt, or not. · GitHub

Attention is your scarcest resource. Chief is the local-first layer that guards it — turning every agent, alert, and feed into one honest call: interrupt, or not. - SmileLikeYe/agent-chief


OpenTag

Context is managed as durable loop state rather than a single prompt payload. Turns an existing work thread into a governed agent work loop: @-mention a coding agent from Slack, GitHub, GitLab, Linear, Lark, Telegram, or Discord, and OpenTag curates the context and manages the run.

Use OpenTag to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (1,378 stars; 78 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - amplifthq/opentag: Mention any ACP coding agent from Slack, GitHub, GitLab, Linear, or Lark. OpenTag runs Claude Code, Codex, Cursor and more on your own machine, then replies in-thread with verified, evidence-backed results. · GitHub

Mention any ACP coding agent from Slack, GitHub, GitLab, Linear, or Lark. OpenTag runs Claude Code, Codex, Cursor and more on your own machine, then replies in-thread with verified, evidence-backed results. - amplifthq/opentag


herdr

State persistence is explicit enough for repeated runs and handoff. Single-binary terminal multiplexer purpose-built for running fleets of coding agents (Claude Code, Codex, Copilot CLI, Cursor Agent, and 15+ others) with per-agent state tracking.

Use herdr to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (25,446 stars; 1,786 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - herdrdev/herdr: the runtime your coding agents live on · GitHub

the runtime your coding agents live on. Contribute to herdrdev/herdr development by creating an account on GitHub.


Orca

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Open-source agent development environment for orchestrating a fleet of parallel coding agents (20+ backends) on your own subscriptions, with per-agent workspaces and review flow.

Use Orca to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (39,364 stars; 2,765 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - stablyai/orca: Orca is the ADE for working with a fleet of parallel agents. Run any coding agent with your own subscription. Available on desktop, mobile and VPS. · GitHub

Orca is the ADE for working with a fleet of parallel agents. Run any coding agent with your own subscription. Available on desktop, mobile and VPS. - stablyai/orca


State persistence is explicit enough for repeated runs and handoff. Argues model routing must live inside the execution harness rather than in single-turn cost-quality tradeoffs, making step-level model selections from execution state and logging each decision as structured telemetry that feeds back to improve both routers and models.

Use Agentic Routing: The Harness-Native Data Flywheel to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.11399; inspect its method and evaluation before treating results as production evidence.

[2607.11399] Agentic Routing: The Harness-Native Data Flywheel

Large language model agents are increasingly executed not by a single model call, but by an execution harness that manages observation, context, control, action, state, and verification. At the same time, frontier and open models are becoming structurally specialized: a model that is strong at code editing, long-context recovery, tool use, mathematical reasoning, or low-latency response may not dominate on the other axes. This makes model selection inside an agent a core systems problem rather than a per-query serving trick. Existing routing methods mostly optimize single-turn cost-quality trade-offs and therefore miss the execution state, intermediate failures, and feedback loops that make agents different from chat completion. We propose Harness-Native agentic routing, a step-level routing paradigm that selects either a single best-fit model for cost-effective execution or multiple complementary models for ensemble-style accuracy improvement, conditioned on the full harness state. The key insight is that every routing decision naturally produces a structured data record -- consisting of the query, harness state, model choice or model set, execution trace, outcome, and cost -- whose labels are supplied by the environment rather than by the router itself. These records form a harness-native data flywheel: execution traces train better routers and harness-native models, which improve cost-quality trade-offs and generate more traces under the same budget. We instantiate this idea in OpenSquilla with a four-layer routing stack, an open LightGBM cold-start ranker, and a staged router-model path that turns logged arena records into progressively stronger routing policies. The report studies singleton and multi-model routing on agentic benchmarks including DRACO and PinchBench, and argues that agentic routing is not merely cost control, but a data engine for agent-native training.

Xinchen Liu; Hang Zhou; Yingjie Zong; Yuchuan Tian; Liuyang Song; Shuo Zhang; Yulong Li; Wei He; Mengyu Zheng; Runke Liu; Siyang Cheng; Xiang Kuang; Hailin Hu; Kai Han; Yunhe Wang


The work separates roles across agents, verifiers, or orchestration layers. Formal orchestration architecture that runs agent workflows on a call stack with lazy capability discovery, giving multi-agent loops explicit, inspectable control flow instead of implicit prompt-driven handoffs.

Use A Formal Hierarchical Architecture for Agentic Orchestration with Stack-Based Execution to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.11138; inspect its method and evaluation before treating results as production evidence.

[2607.11138] A Formal Hierarchical Architecture for Agentic Orchestration with Stack-Based Execution and Lazy Discovery

The rapid expansion of capabilities in Large Language Model (LLM) agents has exposed a critical architectural bottleneck: when agents are given access to a flat, monolithic registry of tools, the model must evaluate hundreds or thousands of options simultaneously. This leads to decision-space explosion, context window saturation, and degraded routing accuracy. To address these limitations, this paper presents a hierarchical, skill-based architecture for agentic orchestration. Capabilities are organized as a rooted tree where internal nodes make routing decisions and leaf nodes execute deterministic tasks. The runtime enforces a single-step execution loop governed by a Last-In-First-Out (LIFO) stack, giving the agent a form of memory akin to a Pushdown Automaton, therefore enabling it to track nested execution contexts and resume deterministically from any depth. Capability discovery follows a manifest-driven, lazy-loading protocol: only the immediate children of the active node are loaded, so memory and prompt costs scale with the explored path rather than the global registry. By replacing global memory with localized stack frames, the architecture prevents outputs from one execution branch from leaking into another, establishing the isolation guarantees required for deployment in regulated enterprise environments. We also discuss UPI Help, an AI-powered digital payments support product, as a motivating production deployment context. We provide a mathematical formalization of the orchestration state, detailed algorithmic analysis of the execution loop, and controlled benchmarks comparing flat and hierarchical routing under increasing tool catalogs, multi-step workflow pressure, and visible schema-token exposure per LLM call.

Prashant Devadiga; Abhishek; Adithya Mishra; Alok Singh; Amisha Sinha; Asit Desai; Gaurang Dahad; Harshit Bhushan; Mandati Pramod Reddy; Prakhar Gupta; Rupesh Patil; Siddhi Behere


Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Formalizes how protocol-compliant reasoning agents can still agree on a semantically invalid transition, especially when they share models, prompts, or tools; derives separate safety and liveness budgets and shows that adding agents helps only when measured error correlation falls.

Use The Honest Quorum Problem: Epistemic Byzantine Fault Tolerance for Agentic Infrastructure to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.16109; inspect its method and evaluation before treating results as production evidence.

[2607.16109] The Honest Quorum Problem: Epistemic Byzantine Fault Tolerance for Agentic Infrastructure

State machine replication (SMR) and Byzantine fault-tolerant (BFT) consensus guarantee agreement despite a bounded number of arbitrary, colluding faulty participants. However, these guarantees rely on participants outside this set correctly executing the protocol's transition semantics. Agentic validators expose a weaker boundary: an authenticated, responsive, non-equivocating, and protocol-compliant reasoning participant may still endorse a semantically invalid transition due to reasoning errors. We call this failure mode an epistemic fault, and the collective phenomenon the Honest Quorum Problem (where "honest" means protocol-compliant, not semantically correct). Such a quorum can satisfy ordinary checks while forming a certificate for an invalid transition. Thus, agreement alone does not guarantee semantic validity or execution safety. Furthermore, because agentic validators often share model weights, training distributions, prompts, or toolchains, they are highly susceptible to correlated epistemic faults. We define Epistemic Byzantine Fault Tolerance (EBFT), a fault-tolerance model for agentic infrastructure and post-deterministic distributed systems. EBFT augments the conventional Byzantine fault bound with two separate, confidence-indexed quantities: $e_\delta$ bounds coherent invalid endorsements outside the Byzantine set, and $u_\epsilon$ bounds unusable validator support that degrades liveness. These quantities characterize semantic safety risk and liveness degradation independently. We derive quorum-threshold conditions for semantic validity, consensus agreement, liveness, and feasible threshold selection, and outline a calibration methodology for estimating these budgets. We show that adding nominally distinct agents improves fault tolerance only when it measurably reduces the upper-tail concentration of invalid endorsements or unusable support.


Graph-based agent workflows

Primary-source operational guidance rather than commentary. Official ADK 2.0 graph runtime for explicit, deterministic workflows that mix agents, tools, and code with branching, state, human input, and bounded cycles.

Use Graph-based agent workflows to choose an implementation surface for repeatable agent work.

Primary official documentation from adk.dev; use it for current product or standard behavior.

Graph-based agent workflows - Agent Development Kit (ADK) Agent Development Kit (ADK)

Build powerful multi-agent systems with Agent Development Kit (ADK)


Flows

Primary-source operational guidance rather than commentary. Official event-driven workflow layer with typed state, branching and loops, persistence decorators, and resume or fork operations for long-running executions.

Use Flows to choose an implementation surface for repeatable agent work.

Primary official documentation from docs.crewai.com; use it for current product or standard behavior.

Learn how to create and manage AI workflows using CrewAI Flows.


Graph

Primary-source operational guidance rather than commentary. Official deterministic graph pattern for multi-agent dependencies and cycles, with shared execution state and explicit execution limits.

Use Graph to choose an implementation surface for repeatable agent work.

Primary official documentation from strandsagents.com; use it for current product or standard behavior.

Graph Multi-Agent Pattern | Strands Agents

Orchestrate multi-agent systems with the Strands Graph pattern: deterministic DAG or cyclic execution, conditional edges, and nested agents.


The work turns loop quality into a measurable task or score. Controls 260 configurations across six agentic benchmarks to show how coordination, model capability, task parallelism, and tool load shape performance and error amplification.

Use Towards a Science of Scaling Agent Systems to choose an implementation surface for repeatable agent work.

Research source arXiv:2512.08296; inspect its method and evaluation before treating results as production evidence.

[2512.08296] Towards a Science of Scaling Agent Systems

Agents, language model-based systems capable of reasoning, planning, and acting are widely adopted in real-world tasks, yet how their performance changes as these systems scale across key dimensions remains underexplored. We introduce quantitative scaling principles for agent systems as a predictive model, capturing how performance varies with coordination, model capability, and measurable system and task factors. Across 260 configurations spanning six agentic benchmarks, five canonical architectures (Single-Agent and four Multi-Agent: Independent, Centralized, Decentralized, Hybrid), and three LLM families, we perform controlled evaluations, standardizing tools, prompts, and compute to isolate architectural effects. The resulting model achieves a cross-validated R^2=0.373 across all six benchmarks (R^2=0.413 with a task-grounded capability metric). We identify a robust capability-saturation effect and additional patterns: (1) a coordination yields diminishing returns once single-agent baselines exceed certain performance; (2) tool-heavy tasks appear to incur multi-agent overhead; and (3) architectures without centralized verification tend to propagate errors more than those with centralized coordination. Relative performance change compared to single-agent baseline ranges from +80.8% on decomposable financial reasoning to -70.0% on sequential planning, demonstrating that architecture-task alignment determines collaborative success. The framework identifies the best-performing architecture for 87% of held-out configurations and shows consistent relative architecture preferences on unseen frontier models. Agent effectiveness depends on alignment between coordination and task structure, and that mismatched coordination degrades the performance.

Yubin Kim; Ken Gu; Chanwoo Park; Chunjong Park; Samuel Schmidgall; A. Ali Heydari; Yao Yan; Zhihan Zhang; Yuchen Zhuang; Yun Liu; Mark Malhotra; Paul Pu Liang; Hae Won Park; Yuzhe Yang; Xuhai Xu; Yilun Du; Shwetak Patel; Tim Althoff; Daniel McDuff; Xin Liu


Amp: From Agent to Agent

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Amp on agent-to-agent handoffs: one agent delegating bounded work to another with results returned to the parent thread, formalizing delegation as a first-class platform primitive.

Use Amp: From Agent to Agent to choose an implementation surface for repeatable agent work.

Contextual source from ampcode.com; useful for practice signals or boundary conditions, not independent validation.

Agents can spawn other agents, message them, and exchange files across Amp threads.


Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Cursor on the economics of agent swarms: how parallel fleets change the cost calculus of model selection, and what coordination overhead does to effective throughput per dollar.

Use Cursor: Agent Swarms and the New Model Economics to choose an implementation surface for repeatable agent work.

Contextual source from cursor.com; useful for practice signals or boundary conditions, not independent validation.

Agent swarms and the new model economics · Cursor

We compared old and new agent swarms building SQLite from scratch and found that better coordination delivers similar quality at a fraction of the cost.


Amp: Meet Puck

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Amp's conversational coordinator that spawns agents, investigates issues, organizes threads, and coordinates multiple agent tasks across projects through natural-language commands.

Use Amp: Meet Puck to choose an implementation surface for repeatable agent work.

Puck is your new meta-agent, your Amp helper


Warren

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Control plane for coding agents that operate in isolation, styled as a Coolify for agents: deploy, monitor, and manage long-running agent workloads on your own infrastructure.

Use Warren to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (281 stars; 66 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - jayminwest/warren: Coolify for coding agents. Control plane for your agents that operate in isolation, self-manage, self-repair, and self-improve all on your infrastructure. · GitHub

Coolify for coding agents. Control plane for your agents that operate in isolation, self-manage, self-repair, and self-improve all on your infrastructure. - jayminwest/warren


agent-talk

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Enables coding agents to work together by giving them a shared communication channel, so parallel agents can coordinate instead of colliding.

Use agent-talk to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (147 stars; 8 forks; MIT license; updated 2026-08-04); popularity is context, not proof of reliability.

GitHub - xhluca/agent-talk: Enabling coding agents to work together · GitHub

Enabling coding agents to work together. Contribute to xhluca/agent-talk development by creating an account on GitHub.


codex-model-routing-team

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Runs background Codex workers under a lead agent that verifies their output before it lands, a small-team pattern for supervised parallel delegation.

Use codex-model-routing-team to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (157 stars; 17 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - zjp1997720/codex-model-routing-team: Route complex Codex tasks to model-specific background workers with bounded concurrency and lead-agent verification. · GitHub

Route complex Codex tasks to model-specific background workers with bounded concurrency and lead-agent verification. - zjp1997720/codex-model-routing-team


Agent Orchestrator

Workspace isolation is part of the loop design, not an afterthought. Agent IDE and orchestrator for managing fleets of coding agents, with worktree isolation and coordination so many agents can work a backlog in parallel from one control surface.

Use Agent Orchestrator to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (8,843 stars; 1,297 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - Untrivial-ai/agent-orchestrator: Agent IDE that enables you to manage fleets of coding agents. It comes with an agentic orchestrator that plans tasks, spawns agents, and autonomously handles CI fixes, merge conflicts, and code reviews. · GitHub

Agent IDE that enables you to manage fleets of coding agents. It comes with an agentic orchestrator that plans tasks, spawns agents, and autonomously handles CI fixes, merge conflicts, and code reviews. - Untrivial-ai/agent-orchestrator


Buzz

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Self-hostable workspace from Block where humans and AI agents share the same rooms, giving a fleet of long-running agents a common coordination surface instead of isolated one-off sessions.

Use Buzz to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (24,627 stars; 2,837 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - block/buzz: A hive mind communication platform · GitHub

A hive mind communication platform. Contribute to block/buzz development by creating an account on GitHub.


open-kritt

Verification is promoted from a final check to a loop-control signal. AGPL-3.0 security-research platform that decomposes vulnerability hunting into small well-defined tasks, fans them out across parallel tool-enabled agents in disposable Docker sandboxes, then closes the loop with post-processing verification scripts, proof-of-concept generation, de-duplication, and severity ranking via reusable workflow playbooks. Built by the pseudonymous Blockian team (self-reported $1.5M in bug-bounty payouts on Immunefi/HackenProof), a worked example of a fan-out/verify agent pipeline in an adversarial domain.

Use open-kritt to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (1,560 stars; 278 forks; AGPL-3.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - Kritt-ai/open-kritt: Orchestrate AI agents to find real vulnerabilities in code. · GitHub

Orchestrate AI agents to find real vulnerabilities in code. - Kritt-ai/open-kritt


BossConsole

The work separates roles across agents, verifiers, or orchestration layers. Native JVM operator's console (Kotlin Multiplatform, explicitly not Electron) that runs Claude Code, Codex, Gemini CLI, and OpenCode side-by-side with a shared browser, terminal, editor, secrets manager, and ~100 MCP tools, adding per-tool RBAC governance, kill-switches, E2E-encrypted session sharing, and daemonized persistent sessions for supervised multi-agent operation.

Use BossConsole to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (216 stars; 7 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - risa-labs-inc/BossConsole: Open-source, multi-platform harness for AI agents — a native, multi-threaded operator's console (JVM, not Electron) to run Claude Code, Codex, Gemini or OpenCode with a real browser, terminal, editor, secrets & 100+ MCP tools. Built for enterprises, science & research. · GitHub

Open-source, multi-platform harness for AI agents — a native, multi-threaded operator's console (JVM, not Electron) to run Claude Code, Codex, Gemini or OpenCode with a real browser, terminal, editor, secrets & 100+ MCP tools. Built for enterprises, science & research. - risa-labs-inc/BossConsole


Fractal

Primary-source operational guidance rather than commentary. Arranges autonomous agent loops into a tree: each node iterates toward a goal in its own Git worktree and spawns child loop-nodes for separable subtasks, bounded by hard caps on iterations, depth, children, cost, and time, with run metadata in a local SQLite database and a live TUI for operator steering. A loop runner whose unit of composition is another loop; Apache-2.0, supports five agent backends, with an official research write-up at plasma.ai (Jul 21, 2026).

Use Fractal to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (678 stars; 50 forks; Apache-2.0 license; updated 2026-08-06); popularity is context, not proof of reliability.

GitHub - plasma-ai/fractal: Hierarchical agent loops with recursive self-organization. · GitHub

Hierarchical agent loops with recursive self-organization. - plasma-ai/fractal


The trigger or cadence is explicit, making the workflow recurring rather than one-off. Serving infrastructure for MCP-based agent loops, where disaggregated sandboxes force a choice between persistent reservations that waste memory at scale and lazy instantiation that inflicts cold-start penalties on multi-tenant multi-turn workloads. SpecBox does intent-driven sandbox prewarming: keyword matching plus streaming semantic embedding predict pending tool-execution demand mid-token-generation and overlap sandbox bootstrap with model inference. One of the few systems papers treating the agent loop's execution environment as the scheduling problem.

Use SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.23933; inspect its method and evaluation before treating results as production evidence.

[2607.23933] SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving

As LLM agents increasingly rely on the Model Context Protocol (MCP) to invoke isolated external sandboxes, disaggregated sandbox deployment introduces a fundamental tension between resource utilization and interactive tail latency. Persistent long-lived sandbox reservations incur excessive memory overhead at scale, while lazy on-demand instantiation generates severe cold-start penalties that degrade response performance under multi-tenant, multi-turn agent workloads. To resolve this dilemma, we present SpecBox, a runtime built around speculative sandbox preallocation tailored for dynamic LLM agent execution pipelines. At its core, SpecBox implements keyword matching and streaming semantic embedding to enable intent-driven sandbox prewarming, which identifies pending tool execution demands mid-LLM token generation and fully overlaps sandbox bootstrapping with model inference. To extend prewarming windows across sequential agent steps, the framework leverages context-aware stochastic prefetching atop a sandbox dependency graph to probabilistically forecast future sandbox switches ahead of execution. We complement these speculative mechanisms with two orthogonal optimizations: a semantic result cache that prunes redundant repeated sandbox invocations, and a dedicated out-of-band shared-memory transport plane that bypasses conventional network serialization to deliver zero-copy artifact transfers. Evaluated on high-concurrency multi-turn agent traces, our prototype demonstrates that SpecBox cuts P99 end-to-end latency by up to $2.9\times$ relative to the on-demand sandbox baseline, while slashing peak memory consumption by $45.9\%$ compared to permanently reserved sandbox deployments.

Yihui Zhang; Tianyu Wo; Jinghao Wang; Xiaoyang Sun; Menghao Zhang; Cangzhou Yuan; Li Li; Chunming Hu; Albert Y. Zomaya; Renyu Yang


The work separates roles across agents, verifiers, or orchestration layers. Implementation-grounded comparison rather than a survey: the same software-engineering task built twice, once on MCP and once on A2A, evaluated against requirements drawn from prior literature and industry partner discussions -- agent discoverability, multi-part messaging, multi-turn conversations, asynchronous communication, observability, interoperability, and access control. Concrete evidence for teams choosing a coordination substrate for multi-agent loops instead of reasoning from spec documents.

Use A Comparative Study of MCP and A2A for Inter-Agent Coordination in LLM-Based Systems to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.23884; inspect its method and evaluation before treating results as production evidence.

[2607.23884] A Comparative Study of MCP and A2A for Inter-Agent Coordination in LLM-Based Systems

Recent industry practice has seen the rapid emergence of agentic systems composed of heterogeneous, tool- and LLM-mediated agent components, raising practical questions about inter-agent coordination and protocol design. This paper presents an implementation-grounded comparison of the Model Context Protocol (MCP) and the Agent2Agent (A2A) protocol, from a multi-agent systems engineering perspective, using an inter-agent coordination scenario involving LLM-based agents. We evaluate an MCP-based and an A2A-based multi-agent implementation of the same software engineering task against a set of requirements derived from prior literature and discussions with industry partners, including agent discoverability, multi-part messaging, multi-turn conversations, asynchronous communication, observability, interoperability, and access control. The results evidence that MCP can support inter-agent coordination in constrained LLM-based systems through a comparatively lightweight implementation model with lower coordination complexity, although coordination concerns such as conversational state management and task lifecycle handling must be implemented explicitly at the application layer. In contrast, A2A provides richer native support for stateful, multi-turn coordination through protocol-level abstractions for tasks and lifecycle management, but this comes with substantially greater implementation and coordination complexity. Given the narrow scope of the evaluated coordination pattern, these findings are presented as design observations from an empirical experience report rather than general claims of protocol suitability or superiority across broader classes of MAS, highlighting trade-offs and how protocol abstractions shape the distribution of coordination responsibilities in contemporary agentic systems.

Ionut Predoaia; Tuong Manh Vu; Konstantinos Barmpis; Dimitris Kolovos; Antonio García-Domínguez


AgentENV

Shows how delegation, handoff, and workflow control turn one agent into a coordinated loop. Open-sourced 2026-07-27 by the Kimi/kvcache-ai team as the environment substrate behind Kimi K3's agentic RL training.

Use AgentENV to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (2,945 stars; 235 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - kvcache-ai/AgentENV: AgentENV (AENV) is a distributed platform for running agent environments at scale. · GitHub

AgentENV (AENV) is a distributed platform for running agent environments at scale. - kvcache-ai/AgentENV


Ruflo

Persistent memory is treated as an external runtime artifact. Agent meta-harness for Claude Code and Codex, built on the premise that an agent is a model plus a harness: it supplies tools, memory, loops, sandboxes, and controls so agents self-organize into swarms, learn across tasks, and persist state between runs. Formerly Claude Flow.

Use Ruflo to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (67,241 stars; 8,046 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - ruvnet/ruflo: 🌊 The original agent meta-harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, RAG integration, and native Claude Code / Codex / Hermes and many more Integrated · GitHub

🌊 The original agent meta-harness. Deploy intelligent multi-player swarms, coordinate autonomous workflows, and build conversational AI systems. Features adaptive memory, self-learning intelligence, RAG integration, and native Claude Code / Codex / Hermes and many more Integrated - ruvnet/ruflo


The work separates roles across agents, verifiers, or orchestration layers. IMACS separates multi-agent systems into organizational structure, coordination mechanism, and collaboration protocol, then shows a contextual-bandit meta-protocol picking per task beats any fixed protocol on quality-cost tradeoffs. The decoupling makes previously incomparable multi-agent results comparable.

Use Toward an Organizational Science of Multi-Agent LLM Systems: Decoupling Who, How, and Which Algorithm to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.25446; inspect its method and evaluation before treating results as production evidence.

[2607.25446] Toward an Organizational Science of Multi-Agent LLM Systems: Decoupling Who, How, and Which Algorithm

Multi-agent frameworks built on large language models (LLMs) routinely entangle three logically distinct concerns: who is on the team (organization), how members align (coordination), and which algorithm fuses their work (collaboration protocol). IMACS (Intelligent Multi-Agent Collaboration System) separates the three into orthogonal, independently swappable layers. Classic organizational theory (Belbin roles, Mintzberg coordination, RACI accountability) becomes executable, validated configuration, and the framework places six published collaboration algorithms behind a common interface while exposing roles, coordination, and accountability as independently configurable factors. We use this separation to conduct controlled comparisons in which organizational assignments vary while the collaboration protocol is held fixed. It also turns protocol choice into a variable that can be learned: Adaptive Org Routing, a contextual-bandit meta-protocol, selects a protocol per task under an explicit quality-cost tradeoff, outperforms every fixed protocol in a controlled study, and trains online on real benchmark and LLM-judge rewards. The ablations expose a mechanism. Accountability placement changes outcomes exactly when the protocol routes the deliverable through the accountable agent, and the winning placement flips across model families, so organizational design cannot be hard-coded; it must be revalidated, or learned, for each model binding.

Huan Chen; Xiang Song; Jian Jin; Pan Ren; Liang-Jie Zhang


Orchestration and control flow are made explicit and inspectable. Hierarchical orchestrator/sub-agent decomposition for long-horizon ML engineering that lets a 4B model reach performance comparable to much larger ones, with relative gains up to 36.7%. Good evidence that delegation structure substitutes for model scale on long tasks.

Use Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.25090; inspect its method and evaluation before treating results as production evidence.

[2607.25090] Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering

Machine learning engineering (MLE) tasks require long-horizon decision making over iterative solution debugging and refinement, under expensive and feedback-driven environment interactions. Developing and training a monolithic agent for such tasks is fundamentally challenging, as it must simultaneously manage extremely long and noisy contexts, explore vast solution spaces, and remain effective under limited model capacity and computational budgets. To address these challenges, we propose Matryoshka Agent, a unified hierarchical agent framework for complex long-horizon tasks. Matryoshka Agent decomposes agentic problem solving into a coordinated hierarchy of decision making and execution: a high-level Orchestrator maintains compact, long-horizon exploration states and issues strategic instructions, while lower-level Sub-Agents execute concrete solution attempts through direct environment interaction, mediated by standardized Tool interface. This design decouples strategic exploration from costly execution, substantially reducing the burden of long-context reasoning and enabling efficient iterative refinement. We further develop an efficient training paradigm for Matryoshka Agent. Experimental results on a broad range of MLE tasks with diverse model types and scales demonstrate that Matryoshka Agent is an effective and scalable paradigm for long-horizon MLE tasks and complex agentic problem solving. Notably, Matryoshka Agent enables Qwen3-4B-Instruct to reach Orchestrator performance comparable to o4-mini. Applying Matryoshka Agent to Qwen3-30B-Coder results in at most 36.7% relative performance gain.

Rushi Qiang; Changhao Li; Haotian Sun; Yuchen Zhuang; Chao Zhang; Bo Dai


The work separates roles across agents, verifiers, or orchestration layers. Systematic review of 141 multi-agent debate studies with a three-dimensional taxonomy over participants, interaction mechanisms, and agreement protocols, concluding the field has converged prematurely on one conventional design. Useful map for anyone choosing a consensus mechanism for a delegation loop.

Use Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.26212; inspect its method and evaluation before treating results as production evidence.

[2607.26212] Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges

Multi-Agent Debate (MAD) is a promising paradigm for improving the accuracy and robustness of Large Language Model (LLM)-based agentic systems. It enables multiple agents to exchange arguments, critique each other's outputs, and iteratively converge towards a solution. However, research remains fragmented, with inconsistent terminology and no rigorous synthesis of MAD design dimensions. We present a systematic literature review characterizing 141 primary studies on MAD. We derive a three-dimensional taxonomy covering debate participants, the interaction mechanisms structuring the exchange, and the agreement protocols governing debate resolution, supported by formal notations to render MAD configurations. Our analysis reveals that the field has implicitly converged on a narrow design pattern - static, fully connected topologies, verbatim exchange, short-term memory and voting resolution strategies - adopted by convention rather than systematic comparison, while promising alternatives remain marginal. Because any MAD setting reflects roughly a dozen interacting design decisions, cross-study comparison is unreliable when these are left implicit. We position the taxonomy as a descriptive map of the research landscape, a framework for controlled benchmarking, and potentially as a schema for machine-readable MAD specifications. As future work, we propose formalizing it into an executable specification, enabling cost-aware benchmarking and automated tuning of debate configurations.

Quim Motger; Marc Oriol; Jordi Marco; Xavier Franch


Agent Manager

State persistence is explicit enough for repeated runs and handoff. Terminal UI for operating a small agent fleet, surfaced on HN 2026-07-30. Each agent runs in an isolated tmux session that persists after the manager exits, and dead sessions revive with conversation history intact, so the fleet outlives the operator's terminal.

Use Agent Manager to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (276 stars; 17 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - YoanWai/agent-manager: Terminal UI to manage AI coding-agent sessions (Claude Code, OpenCode, Codex, Grok, Gemini CLI) in tmux: live status, group tree, quick prompts, diff review. macOS, Linux, Windows via WSL2. · GitHub

Terminal UI to manage AI coding-agent sessions (Claude Code, OpenCode, Codex, Grok, Gemini CLI) in tmux: live status, group tree, quick prompts, diff review. macOS, Linux, Windows via WSL2. - YoanWai/agent-manager


TrueDeck

Persistent memory is treated as an external runtime artifact. Created 2026-07-27. Terminal-first workbench that puts Grok, Codex, Cursor, Claude, and Gemini in split panes on one codebase, each keeping its real TUI rather than being wrapped in a chat webview. The loop-engineering claim is the abstraction: TrueMemory maintains per-repo and global context plus MCP wiring automatically, so operating several agents on a project needs no memory dashboard, note app, or Docker checklist to babysit, 'agentic programming, without the ops.' Useful as a counterpoint to memory systems that ask the operator to curate state by hand.

Use TrueDeck to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (108 stars; 2 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - WutIsHummus/TrueDeck: Free multi-agent coding workbench: Grok/Codex/Claude/Cursor panes + TrueMemory (per-repo + global) · GitHub

Free multi-agent coding workbench: Grok/Codex/Claude/Cursor panes + TrueMemory (per-repo + global) - WutIsHummus/TrueDeck


qm

Durable execution and replay are treated as first-class loop infrastructure. Y Combinator's open-source agent platform for whole organizations rather than one operator. Each person and each Slack room gets its own scoped memory, files, keychain view, permissions, crons, web apps, and durable sandbox, so background work keeps running "while nobody's watching" without workspaces bleeding into each other.

Use qm to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (12,175 stars; 1,385 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - yc-software/qm: Multiplayer agent harness for work · GitHub

Multiplayer agent harness for work. Contribute to yc-software/qm development by creating an account on GitHub.


The work turns loop quality into a measurable task or score. Adds asynchronous message passing so coding agents keep background awareness of each other instead of exchanging information only at handoff boundaries. Four coordinated agents reach 62.1 percent on a long-horizon codebase-comprehension benchmark against 32.3 percent solo, and beat the same harness running a newer model generation.

Use AgentRadio: Passive Awareness for Long-Horizon Multi-Agent Collaboration to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.28430; inspect its method and evaluation before treating results as production evidence.

[2607.28430] AgentRadio: Passive Awareness for Long-Horizon Multi-Agent Collaboration

Understanding large codebases is a long-horizon task for Large Language Model (LLM) agents: answering a single question can require building and running the software, tracing execution across files, and synthesizing evidence over tens of minutes. On SWE-Atlas QnA, a benchmark of long-horizon questions over production repositories, a single Claude Code agent (Opus 4.6) resolves only 32.3% of tasks. Dividing the work among agents with clean contexts mitigates this limitation. However, the subtasks of code comprehension are interdependent. One agent's findings can rewrite another's task, so agents must coordinate during execution, not only at phase boundaries. Existing multi-agent systems support such exchange only between phases, through staged handoffs or synchronized rounds. Communication and work remain mutually exclusive. A discovery made mid-execution cannot be shared until the next boundary. We present AgentRadio, an asynchronous message-passing layer that equips coding-agent harnesses with three primitives: threads, messages, and waiting for mentions. The last runs as a background task, surfacing teammates' messages without interrupting foreground work, so each agent remains passively aware of its peers and folds new findings into its ongoing task. Under a five-phase protocol of division of labor and negotiation, four agents organized by AgentRadio resolve 62.1% of tasks, 29.8 points above a single agent and above Claude Code with the newer Opus 4.8 (57.2%). Rubric-level analysis shows the gain growing with task difficulty, consistent with mid-course correction as the underlying mechanism. Our code is available at https://github.com/Coral-Protocol/AgentRadio.

Xinxing Ren; Qianbo Zang; Ziyan Wang; Caelum Forder; Suman Deb; Peter Carroll; Zekun Guo


The work separates roles across agents, verifiers, or orchestration layers. Existing multi-agent systems treat communication topology as a fixed design choice or an offline optimization target.

Use MANTA: Multi-Agent Network Topology Adaptation for Self-Evolving Multi-Agent Systems to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.28527; inspect its method and evaluation before treating results as production evidence.

[2607.28527] MANTA: Multi-Agent Network Topology Adaptation for Self-Evolving Multi-Agent Systems

Large language model-based multi-agent systems improve complex problem solving through task decomposition, agent specialization, information exchange, and intermediate validation. However, existing systems typically treat communication topology as a fixed design choice or an offline optimization target. We introduce MANTA, a framework for Multi-Agent Network Topology Adaptation that enables communication structures to self-evolve at inference time. Before execution, MANTA initializes a task-conditioned topology from prior structural experience. During deployment, it monitors collaboration traces and applies bounded structural updates when the current organization becomes insufficient. These updates can modify agent roles, communication links, execution order, information visibility, and validation pathways while preserving the task interface and agent budget. We evaluate MANTA against representative single-agent and multi-agent baselines on five benchmarks spanning information seeking, tool use, planning, workflow execution, and mathematical reasoning. MANTA achieves the highest average score of 74.0, outperforming the strongest baseline by 5.8 percentage points and obtaining the best result on PlanCraft. These results show that inference-time self-improvement can extend to the architecture of collaboration itself.

Mao-xun Huang; Jerry Wang; Yi-Cheng Lai; Zhengxin Zhang; Claire Cardie; Hen-Hsen Huang


sol-advisor

Orchestration and control flow are made explicit and inspectable. Codex-native orchestration plugin that splits an agent run into an architect session that owns requirements, specs and acceptance, and bounded implementation lanes (native Terra lane, or an opt-in Luna lane using visible Codex app tasks) that only execute.

Use sol-advisor to choose an implementation surface for repeatable agent work.

Inspectable GitHub source (1,747 stars; 126 forks; MIT license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - DannyMac180/sol-advisor: Codex-native architect orchestration with Luna and Terra implementation lanes and mandatory fresh Sol review. · GitHub

Codex-native architect orchestration with Luna and Terra implementation lanes and mandatory fresh Sol review. - DannyMac180/sol-advisor


The work separates roles across agents, verifiers, or orchestration layers. Scales the harness rather than the model: embedding-based skill retrieval, submodular anti-dilution routing, and adaptive skill evolution let multiple agents refine over repeated rounds. Notably honest about limits, gains are large on open-ended math but flat or negative on multiple-choice, and mostly land in the first refinement round.

Use SKIMIX: Multi-Agent Harness-Time Scaling with Skill Mixture for Dynamic Harness Engineering to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.27994; inspect its method and evaluation before treating results as production evidence.

[2607.27994] SKIMIX: Multi-Agent Harness-Time Scaling with Skill Mixture for Dynamic Harness Engineering

AI agents increasingly rely on large skill libraries, but selecting, combining, and maintaining skills remains difficult. We propose SKIMIX, a multi-agent framework in which agents with different skill portfolios collaborate through iterative refinement. SKIMIX combines embedding-based skill retrieval, submodular anti-dilution routing, and adaptive skill evolution. Across six reasoning benchmarks, multi-agent collaboration substantially improves open-ended mathematical reasoning but offers limited or negative gains on multiple-choice tasks. Agent-count scaling is non-monotonic, and most improvements arise during the first refinement round. These results show that task characteristics determine whether skill-level ensembles help and provide practical guidance for scalable agent design.


The work separates roles across agents, verifiers, or orchestration layers. Introduces MSEval plus the LegoGent execution engine and holds the model fixed while varying 10 collaboration topologies, finding that organizational topology rivals model capability, score swings above 30 points. Structured pipelines converge fastest; heavy managerial layers hurt.

Use An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.27877; inspect its method and evaluation before treating results as production evidence.

[2607.27877] An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding

Multi-agent vibe coding promises to accelerate software development, yet existing benchmarks rely on synthetic environments that ignore practical time and monetary costs, conflate reasoning with communication, and reward only superficial completion. We introduce multi-agent from-scratch evaluation benchmark, MSEval, evaluating multi-agent coding on real-world tasks. Grounded in 10 authentic, full-stack projects across 10 domains, MSEval scores performance using hierarchical requirements and deterministic rubrics. Its execution engine, LegoGent, tests 10 collaboration topologies where agents coordinate via periodic sync intervals and deploy through native CI/CD pipelines. Concurrently, the automated grader TAgent dynamically probes implementations to jointly measure functional success, latency, and prefix-cached token cost. Across 100 runs, MSEval reveals that organizational topology rivals model capability in shaping the speed--cost--quality trade-off. For identical tasks and models, varying the topology shifts scores by over 30 points and doubles wall-clock time. Structured pipelines converge fastest with the highest quality, whereas heavy managerial oversight degrades performance. Ultimately, MSEval establishes a rigorous, reproducible standard for measuring how multi-agent teams actually build software. The benchmark is released at https://github.com/robinren03/MSEval.

Yanyu Ren; Yunfeng Bai; Xizheng Wang; Li Chen; Dan Li


The work separates roles across agents, verifiers, or orchestration layers. Proposes four design principles, simplicity, elastic feedback, sequential workflows with optional loops, and summary-based communication, then tests them across four architecture configurations on standardized engineering tasks.

Use Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability Analysis to choose an implementation surface for repeatable agent work.

Research source arXiv:2607.27942; inspect its method and evaluation before treating results as production evidence.

[2607.27942] Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability Analysis

LLM-based multi-agent systems have the potential to enable collective intelligence and scale toward solving highly complex tasks through coordinated ensembles of specialized agents. However, despite their theoretical potential, the architectural design space remains largely non-systematized and lacks broadly established design principles. Furthermore, the scalability characteristics of such systems are only partially understood so far. This paper makes two contributions. We first distill four design principles for scalable MAS architectures from a structured analysis of prior work: simplicity, elastic feedback, sequential workflows with optional loops, and summary-based communication. We operationalize these principles in a reference architecture whose topology is formalized as a constrained directed workflow graph, and we evaluate four configurations of increasing complexity on a standardized benchmark of terminal-based system engineering tasks using two LLMs of differing capability. Our findings show that scaling yields measurable accuracy improvements with approximately linear cost growth, but only when the underlying LLM exceeds a minimum capability threshold. Performance peaks at intermediate complexity, then degrades due to timeouts and evaluation limitations. In addition, persistent consistency issues emerge as a central challenge across all scaling levels. These results provide concrete design guidance for practitioners and highlight consistency and evaluation standardization as key targets for future research.

Linus Sander; Fengjunjie Pan; Vahid Zolfaghari; Andre Schamschurko; Nenad Petrovic; Alois Knoll


SWE-bench

The work turns loop quality into a measurable task or score. Benchmark for resolving real GitHub issues through code editing and tests.

Use SWE-bench to measure progress and gate completion with repeatable evidence.

Benchmark or leaderboard source with repeatable tasks or scores; compare systems only after checking setup and scope.

Gate progress with tests, evals, and evidence.


Original SWE-bench paper.

Links loop design to measurable tasks where progress and failure can be compared. Original SWE-bench paper.

Use SWE-bench: Can Language Models Resolve Real-World GitHub Issues? to measure progress and gate completion with repeatable evidence.

Research source arXiv:2310.06770; inspect its method and evaluation before treating results as production evidence.

[2310.06770] SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.

Carlos E. Jimenez; John Yang; Alexander Wettig; Shunyu Yao; Kexin Pei; Ofir Press; Karthik Narasimhan

Published in International Conference on Learning Representations (ICLR); the linked arXiv record remains available for open access.


SWE-bench Goes Live

The work turns loop quality into a measurable task or score. Dynamic benchmark designed to reduce overfitting to static issue sets.

Use SWE-bench Goes Live to measure progress and gate completion with repeatable evidence.

Research source arXiv:2505.23419; inspect its method and evaluation before treating results as production evidence.

The issue-resolving task, where a model generates patches to fix real-world bugs, has emerged as a critical benchmark for evaluating the capabilities of large language models (LLMs). While SWE-bench and its variants have become standard in this domain, they suffer from key limitations: they have not been updated since their initial releases, cover a narrow set of repositories, and depend heavily on manual effort for instance construction and environment setup. These factors hinder scalability and introduce risks of overfitting and data contamination. In this work, we present SWE-bench-Live, a live-updatable benchmark designed to overcome these challenges. Our initial release consists of 1,319 tasks derived from real GitHub issues created since 2024, spanning 93 repositories. Each task is accompanied by a dedicated Docker image to ensure reproducible execution. Central to our benchmark is \method, an automated curation pipeline that streamlines the entire process from instance creation to environment setup, removing manual bottlenecks and enabling scalability and continuous updates. We evaluate a range of state-of-the-art agent frameworks and LLMs on SWE-bench-Live, revealing a substantial performance gap compared to static benchmarks like SWE-bench, even under controlled evaluation conditions. To better understand this discrepancy, we perform detailed analyses across repository origin, issue recency, and task difficulty. By providing a fresh, diverse, and executable benchmark grounded in live repository activity, SWE-bench-Live facilitates rigorous, contamination-resistant evaluation of LLMs and agents in dynamic, real-world software development settings.

Linghao Zhang; Shilin He; Chaoyun Zhang; Yu Kang; Bowen Li; Chengxing Xie; Junhao Wang; Maoquan Wang; Yufan Huang; Shengyu Fu; Elsie Nallipogu; Qingwei Lin; Yingnong Dang; Saravan Rajmohan; Dongmei Zhang

Neural Information Processing Systems Foundation

Published in Advances in Neural Information Processing Systems 38: Datasets and Benchmarks Track (NeurIPS); the linked arXiv record remains available for open access.


Terminal-Bench

The work turns loop quality into a measurable task or score. Benchmark for agents operating in terminal environments.

Use Terminal-Bench to measure progress and gate completion with repeatable evidence.


Terminal-Bench repository

The work turns loop quality into a measurable task or score. Open-source benchmark and harness for hard terminal tasks.

Use Terminal-Bench repository to measure progress and gate completion with repeatable evidence.

Inspectable GitHub source (454 stars; 366 forks; Apache-2.0 license; updated 2026-08-07); popularity is context, not proof of reliability.

GitHub - harbor-framework/terminal-bench: Measuring and evolving with the frontier of agent work · GitHub

Measuring and evolving with the frontier of agent work - harbor-framework/terminal-bench


AgentBench

The work turns loop quality into a measurable task or score. Multi-environment benchmark for evaluating LLMs as agents.

Use AgentBench to measure progress and gate completion with repeatable evidence.

Research source arXiv:2308.03688; inspect its method and evaluation before treating results as production evidence.

[2308.03688] AgentBench: Evaluating LLMs as Agents

The potential of Large Language Model (LLM) as agents has been widely acknowledged recently. Thus, there is an urgent need to quantitatively \textit{evaluate LLMs as agents} on challenging tasks in interactive environments. We present AgentBench, a multi-dimensional benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities. Our extensive test over \num API-based and open-sourced (OSS) LLMs shows that, while top commercial LLMs present a strong ability of acting as agents in complex environments, there is a significant disparity in performance between them and many OSS competitors that are no larger than 70B. We identify the typical reasons of failures in environments and LLMs, showing that poor long-term reasoning, decision-making, and instruction following abilities are the main obstacles for developing usable LLM agents. Improving instruction following and training on high quality multi-round alignment data could improve agent performance. And different from existing assumptions, training on code present ambivalent impacts on different agent tasks. Datasets, environments, and an integrated evaluation package for AgentBench are released at https://github.com/THUDM/AgentBench.

Xiao Liu; Hao Yu; Hanchen Zhang; Yifan Xu; Xuanyu Lei; Hanyu Lai; Yu Gu; Hangliang Ding; Kaiwen Men; Kejuan Yang; Shudan Zhang; Xiang Deng; Aohan Zeng; Zhengxiao Du; Chenhui Zhang; Sheng Shen; Tianjun Zhang; Yu Su; Huan Sun; Minlie Huang; Yuxiao Dong; Jie Tang


WebArena

Links loop design to measurable tasks where progress and failure can be compared. Realistic web environment for autonomous agents.

Use WebArena to measure progress and gate completion with repeatable evidence.

Research source arXiv:2307.13854; inspect its method and evaluation before treating results as production evidence.

[2307.13854] WebArena: A Realistic Web Environment for Building Autonomous Agents

With advances in generative AI, there is now potential for autonomous agents to manage daily tasks via natural language commands. However, current agents are primarily created and tested in simplified synthetic environments, leading to a disconnect with real-world scenarios. In this paper, we build an environment for language-guided agents that is highly realistic and reproducible. Specifically, we focus on agents that perform tasks on the web, and create an environment with fully functional websites from four common domains: e-commerce, social forum discussions, collaborative software development, and content management. Our environment is enriched with tools (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage human-like task-solving. Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions. The tasks in our benchmark are diverse, long-horizon, and designed to emulate tasks that humans routinely perform on the internet. We experiment with several baseline agents, integrating recent techniques such as reasoning before acting. The results demonstrate that solving complex tasks is challenging: our best GPT-4-based agent only achieves an end-to-end task success rate of 14.41%, significantly lower than the human performance of 78.24%. These results highlight the need for further development of robust agents, that current state-of-the-art large language models are far from perfect performance in these real-life tasks, and that WebArena can be used to measure such progress.

Shuyan Zhou; Frank F. Xu; Hao Zhu; Xuhui Zhou; Robert Lo; Abishek Sridhar; Xianyi Cheng; Tianyue Ou; Yonatan Bisk; Daniel Fried; Uri Alon; Graham Neubig


OSWorld

The work turns loop quality into a measurable task or score. Benchmark for multimodal agents operating full computer environments.

Use OSWorld to measure progress and gate completion with repeatable evidence.

Research source arXiv:2404.07972; inspect its method and evaluation before treating results as production evidence.

[2404.07972] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.

Tianbao Xie; Danyang Zhang; Jixuan Chen; Xiaochuan Li; Siheng Zhao; Ruisheng Cao; Toh Jing Hua; Zhoujun Cheng; Dongchan Shin; Fangyu Lei; Yitao Liu; Yiheng Xu; Shuyan Zhou; Silvio Savarese; Caiming Xiong; Victor Zhong; Tao Yu

Published in Advances in Neural Information Processing Systems 37: Datasets and Benchmarks Track (NeurIPS); the linked arXiv record remains available for open access.


ToolBench

Packages the evidence as queryable CSV and JSONL rather than only a rendered page. Tool-use benchmark and dataset for tool-augmented agents.

Use ToolBench to measure progress and gate completion with repeatable evidence.

Research source arXiv:2307.16789; inspect its method and evaluation before treating results as production evidence.

[2307.16789] ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

Despite the advancements of open-source large language models (LLMs), e.g., LLaMA, they remain significantly limited in tool-use capabilities, i.e., using external tools (APIs) to fulfill human instructions. The reason is that current instruction tuning largely focuses on basic language tasks but ignores the tool-use domain. This is in contrast to the excellent tool-use capabilities of state-of-the-art (SOTA) closed-source LLMs, e.g., ChatGPT. To bridge this gap, we introduce ToolLLM, a general tool-use framework encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is constructed automatically using ChatGPT. Specifically, the construction can be divided into three stages: (i) API collection: we collect 16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub; (ii) instruction generation: we prompt ChatGPT to generate diverse instructions involving these APIs, covering both single-tool and multi-tool scenarios; (iii) solution path annotation: we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To enhance the reasoning capabilities of LLMs, we develop a novel depth-first search-based decision tree algorithm. It enables LLMs to evaluate multiple reasoning traces and expand the search space. Moreover, to evaluate the tool-use capabilities of LLMs, we develop an automatic evaluator: ToolEval. Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction. Experiments show that ToolLLaMA demonstrates a remarkable ability to execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT. Our ToolLLaMA also demonstrates strong zero-shot generalization ability in an out-of-distribution tool-use dataset: APIBench.

Yujia Qin; Shihao Liang; Yining Ye; Kunlun Zhu; Lan Yan; Yaxi Lu; Yankai Lin; Xin Cong; Xiangru Tang; Bill Qian; Sihan Zhao; Lauren Hong; Runchu Tian; Ruobing Xie; Jie Zhou; Mark Gerstein; Dahai Li; Zhiyuan Liu; Maosong Sun


GAIA

The work turns loop quality into a measurable task or score. Benchmark for general AI assistants requiring reasoning, tool use, and multi-step work.

Use GAIA to measure progress and gate completion with repeatable evidence.

Research source arXiv:2311.12983; inspect its method and evaluation before treating results as production evidence.

[2311.12983] GAIA: a benchmark for General AI Assistants

We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins. This notable performance disparity contrasts with the recent trend of LLMs outperforming humans on tasks requiring professional skills in e.g. law or chemistry. GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer. We release our questions while retaining answers to 300 of them to power a leader-board available at https://huggingface.co/gaia-benchmark.

Grégoire Mialon; Clémentine Fourrier; Craig Swift; Thomas Wolf; Yann LeCun; Thomas Scialom


Tau-bench

The work turns loop quality into a measurable task or score. Benchmark for tool-agent-user interactions in realistic domains.

Use Tau-bench to measure progress and gate completion with repeatable evidence.

Research source arXiv:2406.12045; inspect its method and evaluation before treating results as production evidence.

[2406.12045] $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose $\tau$-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably.

Shunyu Yao; Noah Shinn; Pedram Razavi; Karthik Narasimhan


VisualWebArena

The work turns loop quality into a measurable task or score. Visually grounded web-agent benchmark extending WebArena.

Use VisualWebArena to measure progress and gate completion with repeatable evidence.

Research source arXiv:2401.13649; inspect its method and evaluation before treating results as production evidence.

[2401.13649] VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

Autonomous agents capable of planning, reasoning, and executing actions on the web offer a promising avenue for automating computer tasks. However, the majority of existing benchmarks primarily focus on text-based agents, neglecting many natural tasks that require visual information to effectively solve. Given that most computer interfaces cater to human perception, visual information often augments textual data in ways that text-only models struggle to harness effectively. To bridge this gap, we introduce VisualWebArena, a benchmark designed to assess the performance of multimodal web agents on realistic \textit{visually grounded tasks}. VisualWebArena comprises of a set of diverse and complex web-based tasks that evaluate various capabilities of autonomous multimodal agents. To perform on this benchmark, agents need to accurately process image-text inputs, interpret natural language instructions, and execute actions on websites to accomplish user-defined objectives. We conduct an extensive evaluation of state-of-the-art LLM-based autonomous agents, including several multimodal models. Through extensive quantitative and qualitative analysis, we identify several limitations of text-only LLM agents, and reveal gaps in the capabilities of state-of-the-art multimodal language agents. VisualWebArena provides a framework for evaluating multimodal autonomous language agents, and offers insights towards building stronger autonomous agents for the web. Our code, baseline models, and data is publicly available at https://jykoh.com/vwa.

Jing Yu Koh; Robert Lo; Lawrence Jang; Vikram Duvvur; Ming Chong Lim; Po-Yu Huang; Graham Neubig; Shuyan Zhou; Ruslan Salakhutdinov; Daniel Fried

Published in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL); the linked arXiv record remains available for open access.


AppWorld

Evaluation data is used as the feedback signal for improving loop behavior. Benchmark of interactive app tasks with state-based and execution-based evaluation.

Use AppWorld to measure progress and gate completion with repeatable evidence.

Research source arXiv:2407.18901; inspect its method and evaluation before treating results as production evidence.

[2407.18901] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

Autonomous agents that address day-to-day digital tasks (e.g., ordering groceries for a household), must not only operate multiple apps (e.g., notes, messaging, shopping app) via APIs, but also generate rich code with complex control flow in an iterative manner based on their interaction with the environment. However, existing benchmarks for tool use are inadequate, as they only cover tasks that require a simple sequence of API calls. To remedy this gap, we built $\textbf{AppWorld Engine}$, a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs and populated with realistic digital activities simulating the lives of ~100 fictitious users. We then created $\textbf{AppWorld Benchmark}$ (40K lines of code), a suite of 750 natural, diverse, and challenging autonomous agent tasks requiring rich and interactive code generation. It supports robust programmatic evaluation with state-based unit tests, allowing for different ways of completing a task while also checking for unexpected changes, i.e., collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our 'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least 16% fewer. This highlights the benchmark's difficulty and AppWorld's potential to push the frontiers of interactive coding agents. The project website is available at https://appworld.dev/.

Harsh Trivedi; Tushar Khot; Mareike Hartmann; Ruskin Manku; Vinty Dong; Edward Li; Shashank Gupta; Ashish Sabharwal; Niranjan Balasubramanian


Vending-Bench

The work turns loop quality into a measurable task or score. Benchmark for long-term coherence of autonomous agents; documents how small errors compound over very long loop horizons.

Use Vending-Bench to measure progress and gate completion with repeatable evidence.

Research source arXiv:2502.15840; inspect its method and evaluation before treating results as production evidence.

[2502.15840] Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents

While Large Language Models (LLMs) can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons. In this paper, we present Vending-Bench, a simulated environment designed to specifically test an LLM-based agent's ability to manage a straightforward, long-running business scenario: operating a vending machine. Agents must balance inventories, place orders, set prices, and handle daily fees - tasks that are each simple but collectively, over long horizons (>20M tokens per run) stress an LLM's capacity for sustained, coherent decision-making. Our experiments reveal high variance in performance across multiple LLMs: Claude 3.5 Sonnet and o3-mini manage the machine well in most runs and turn a profit, but all models have runs that derail, either through misinterpreting delivery schedules, forgetting orders, or descending into tangential "meltdown" loops from which they rarely recover. We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits. Apart from highlighting the high variance in performance over long time horizons, Vending-Bench also tests models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios. We hope the benchmark can help in preparing for the advent of stronger AI systems.


Vending-Bench leaderboard

The work turns loop quality into a measurable task or score. Live long-horizon coherence results from Andon Labs.

Use Vending-Bench leaderboard to measure progress and gate completion with repeatable evidence.

Vending-Bench: Testing long-term coherence in agents | Andon Labs

How do agents act over very long horizons? We answer this by letting agents manage a simulated vending machine business. The agents need to handle ordering, inventory management, and pricing over long context horizons to successfully make money.


The work turns loop quality into a measurable task or score. Release-note-derived evolution tasks where agents score far below isolated-issue benchmarks, quantifying the long-horizon gap loops must manage.

Use SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios to measure progress and gate completion with repeatable evidence.

[2512.18470] SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature. However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality. We introduce SWE-EVO, a benchmark for this long-horizon software evolution challenge. Constructed from release notes of seven mature open-source Python projects, SWE-EVO comprises 48 tasks requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance. Experiments reveal a striking capability gap: GPT-5.4 with OpenHands achieves only 25% on SWE-EVO versus 72.80% achieved by GPT-5.2 on SWE-Bench Verified, showing that current agents struggle with sustained, multi-file reasoning. We also propose Fix Rate, a metric capturing partial progress on these complex, long-horizon tasks.

Tue Le; Minh V. T. Thai; Dung Nguyen Manh; Huy Phan Nhat; Nghi D. Q. Bui


Verification is promoted from a final check to a loop-control signal. A skill generator and a co-evolving surrogate verifier improve multi-file skill packages over iterations, evaluated on the SkillsBench benchmark of structured skill bundles.

Use EvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification to measure progress and gate completion with repeatable evidence.

Research source arXiv:2604.01687; inspect its method and evaluation before treating results as production evidence.

[2604.01687] CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification

Anthropic proposes the concept of skills for LLM agents to tackle multi-step professional tasks that simple tool invocations cannot address. A tool is a single, self-contained function, whereas a skill is a structured bundle of interdependent multi-file artifacts. Currently, skill generation is not only label-intensive due to manual authoring, but also may suffer from human--machine cognitive misalignment, which can lead to degraded agent performance, as evidenced by evaluations on SkillsBench. Therefore, we aim to enable agents to autonomously generate skills. However, existing self-evolving methods designed for tools cannot be directly applied to skills due to their increased complexity. To address these issues, we propose CoEvoSkills, a self-evolving skills framework that enables agents to autonomously construct complex, multi-file skill packages. Specifically, CoEvoSkills couples a Skill Generator that iteratively refines skills with a Surrogate Verifier that co-evolves to provide informative and actionable feedback without access to ground-truth test content. On SkillsBench, CoEvoSkills achieves the highest pass rate among five baselines on both Claude Code and Codex, and also exhibits strong generalization capabilities to six additional LLMs.

Hanrong Zhang; Shicheng Fan; Henry Peng Zou; Yankai Chen; Zhenting Wang; Jiayu Zhou; Chengze Li; Wei-Chieh Huang; Yifei Yao; Kening Zheng; Xue Liu; Xiaoxiao Li; Philip S. Yu


Evaluation data is used as the feedback signal for improving loop behavior. Benchmark for agents on multi-dependency, interactive enterprise tasks, with automated evaluation that probes where long-horizon loops break down.

Use SaaSBench: Coding Agents in Long-Horizon Enterprise SaaS Engineering to measure progress and gate completion with repeatable evidence.

[2605.17526] SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering

As autonomous coding agents become capable of handling increasingly long-horizon tasks, they have gradually demonstrated the potential to complete end-to-end software development. Although existing benchmarks have recently evolved from localized code editing to from-scratch project generation, they remain confined to structurally simplified, single-stack applications. Consequently, they fail to capture the heterogeneous environments, full-stack orchestration, and system-level complexity of real enterprise Software as a Service (SaaS) systems, leaving a critical gap in assessing agents under realistic engineering constraints. To fill this gap, we introduce SaaSBench, the first benchmark designed to explore the boundaries of AI agents in enterprise SaaS engineering. Spanning 30 complex tasks across 6 SaaS domains with 5,370 validation nodes, it incorporates 8 programming languages, 6 databases, and 13 frameworks to meticulously mirror real-world software heterogeneity. Furthermore, we design a dependency-aware hybrid evaluation paradigm tailored for complex systems with long horizons and multi-component coupling, enabling fine-grained, reproducible assessment. Crucially, our extensive experiments reveal a striking insight: the primary bottleneck for state-of-the-art agents is not generating isolated code logic, but successfully configuring and integrating a multi-component system. Over 95\% of task failures occur before agents even reach deep business logic, with models often falling victim to overconfidence and prematurely halting during foundational system setup, or getting trapped in ineffective debugging loops. We hope SaaSBench serves as a practical and challenging testbed to drive the evolution of reliable, system-level coding agents. The code is available at \url{https://github.com/ShadeCloak/SaaSbench}.

Qingnan Ren; Shun Zou; Shiting Huang; Ziao Zhang; Kou Shi; Zhen Fang; Yiming Zhao; Yu Zeng; Qisheng Su; Lin Chen; Yong Wang; Zehui Chen; Xiangxiang Chu; Feng Zhao


The work targets tasks that exceed a single context window or prompt session. 115 real version-upgrade tasks across 17 repositories requiring multi-file changes (median ~3,700 lines), stressing how far agent loops sustain coherent, large-scale work.

Use RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades to measure progress and gate completion with repeatable evidence.

[2605.15846] RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades

Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem.

Xinbo Xu; Ruihan Yang; Haiyang Shen; Wendong Xu; Bofei Gao; Ruoyu Wu; Kean Shi; Weichu Xie; Xuanzhong Chen; Ming Wu; Jason Zeng; Michael Heinrich; Elvis Zhang; Liang Chen; Kuan Li; Baobao Chang