arxiv:2610.20424

BizBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business

Published on Oct 14

· Submitted by

Yongqi Tong on Oct 19

ant-international

Upvote

Authors:

Abstract

BizBench evaluates long-horizon business agents on intermediate execution validity using real-world payment procedures, revealing that frontier models frequently violate constraints despite achieving partial goal success.

Generated by thinkingmachines/Inkling-Small

Existing long-horizon agent benchmarks often emphasize final task success while providing limited assessment of intermediate execution validity. Yet real-world business operations require agents to navigate extensive, potentially outdated or conflicting policies while satisfying critical constraints throughout execution. These requirements are particularly consequential in high-stakes domains such as payments, where intermediate violations can incur financial losses or regulatory consequences regardless of the final outcome. To bridge this evaluation gap, we introduce BizBench, a multi-turn benchmark grounded in real-world procedures from a leading payments company. Its executable environment integrates 789 knowledge documents and 601 discoverable business tools, with consistent policies, states, and tool behavior. The benchmark comprises 50 tasks involving resource constraints, temporal dependencies, prior commitments, and conflicting or outdated guidance. Each task is validated through reference execution, while deterministic checks assess both goal attainment and execution validity without prescribing a single solution path. Experiments reveal substantial limitations even among frontier models: Claude Opus 5 and GPT-6 Sol achieve strict success rates of only 64.0% and 61.3%, respectively. For Qwen3.8-27B, goal attainment reaches 42.0%, but strict success drops to 32.7%, demonstrating how outcome-based evaluation can mask invalid execution. Beyond success rates, DeepSeek-V4.1-Flash averages 191.3 tool calls per run, approximately 70% more than reference executions, revealing substantial execution overhead. Overall, BizBench establishes a testbed for evaluating and advancing long-horizon agents beyond task completion, toward reliable and constraint-compliant execution in complex business environments.

View arXiv page View PDF GitHub 1 Add to collection

Community

Yooki

Paper submitter about 6 hours ago

AI agents are moving from answering questions to executing business workflows, where actions change business states, commit resources, and shape later decisions. We introduce BizAgentBench, a long-horizon benchmark grounded in workflows from a global payments company. Agents complete tasks through customer dialogue, document retrieval, and tool use in a stateful simulated environment. Its design centers on three business insights.
First, rule applicability depends on the business event. Guidance is scattered across manuals, FAQs, and operational notices. Historical and current instructions coexist, and apparent conflicts may reflect differences in product, region, or effective period. An expired insurance policy, for example, may still cover an incident that occurred during its coverage period. Agents must establish which sources are authoritative and which rules govern the event they are handling.
Second, compliance depends on authorization and business state at the moment of execution. A customer’s general consent does not authorize every operation: approval must cover the specific action and its terms, come from someone with the required authority, and precede execution. A tool may accept an operation that breaches a reserve requirement or interrupts required coverage. Restoring the balance later does not erase the earlier violation; critical constraints must hold throughout the workflow.
Third, waiting can make a task infeasible. Reviews, refunds, or settlements may finish after another obligation requires action. A priority delivery booking window, for example, may close before warehouse confirmation arrives, making timely delivery impossible if the agent waits. Agents must coordinate information gathering and action while accounting for shared resources and existing commitments. A locally reasonable decision can consume funds or time needed elsewhere.
BizAgentBench translates these insights into deterministic checks of goal attainment, process compliance, and prohibited side effects, while accepting different valid strategies. GPT-6 Sol achieves only 61.3% strict success, with just 40% of tasks completed successfully in all three runs. The benchmark brings evaluation closer to the demands of business execution: identifying the right rules, acting at the right time, and maintaining valid execution throughout.