| --- |
| title: README |
| emoji: 😻 |
| colorFrom: pink |
| colorTo: indigo |
| sdk: static |
| pinned: false |
| license: apache-2.0 |
| --- |
| |
| # AutoBench |
|
|
| ## Organization Description |
|
|
| **[AutoBench](https://autobench.org/)** is the premier LLM evaluation and routing infra for the Agentic Era. This is not just about LLM benchmarking, but real-time, AI-trained LLM routing for agents (delivering up to 90% inference cost savings). |
|
|
| We are solving the LLM evaluation crisis by moving the industry beyond static, domain-rigid, and easily gameable benchmarks. AutoBench uses massive pools of LLMs to dynamically generate tasks, execute multi-turn workflows, and granularly evaluate LLM performance. Our benchmarks correlate 80-90% with industry standards, but they remain strictly un-gameable, unbiased, granular, flexible. At a fraction of the cost. |
|
|
| And that is just the beginning. We leverage the massive synthetic datasets generated by our benchmarks to train **next-gen Agentic LLM Routers**, helping agentic frameworks optimize for both quality and economics. |
|
|
| Our vision sees AutoBench as the essential, universal layer that will soon intermediate between all AI agents and underlying LLMs. |
|
|
| ## The AutoBench Ecosystem |
|
|
| ### 1. AutoBench Agentic (Latest Evolution) |
| Current agentic benchmarks (like Gval-AA or Terminal-bench) are static, allowing models to simply "train to the test." AutoBench Agentic fundamentally changes this by dropping LLMs into dynamically generated, multi-turn **Agentic Virtual Environments**. |
| * **Technical Complexity:** Our infrastructure combines deterministic procedures and LLM generation to build complex business-flavored agentic task payloads via a native Universal Intermediate Representation (UIR). We inject stateful "memory lines" of previous workflow failures, and force models to navigate complex native JSON `tools[]` arrays filled with randomly injected "distractor" tools. |
| * **10 Granular Task Types:** We evaluate true orchestration under pressure, measuring specific capabilities like *Adaptive Replanning*, *Parameter Complexity*, *Single Tool Call*, and *Failure Recovery*. |
| * **Cost vs. Performance Tracking:** Tracks exact P99 Latency and strict USD/run costs to help developers define their efficiency frontier. |
|
|
| ### 2. Agentic LLM Routing (alpha) |
| Benchmarking is just the first step. AutoBench uses the millions of execution traces and granular performance data generated by our runs to train dynamic **Agentic LLM Routers**. Instead of passive gateways or superficial semantic heuristic routers, AutoBench empowers active pipeline optimization—routing complex edge cases to frontier models and standard tasks to open-weight models, saving agent applications and enterprises up to 90% in API costs. |
|
|
| ### 3. AutoBench 2.0 & Domain Benchmarks |
| The core engine powering our latest generalist and domain-specific runs (such as our Agronomy vertical). AutoBench 2.0 introduces three major technical breakthroughs to the Collective-LLM-as-a-Judge framework: |
| * **Random Score Pooling:** Instead of prefixed judging models, we pool random models for every scoring session, expanding exploration of the "LLM performance space" while reducing required compute. |
| * **Nonlinear Weighting:** Replaces simple linear averaging with advanced weighting functions (exponential, power-law, Boltzmann) to compensate for variance and improve convergence among highly capable frontier models. |
| * **Parallel Iteration:** Reduces evaluation cycles from days to mere hours. |
|
|
| ### 4. Bot Scanner (Consumer/Dev Platform) |
| Powered by AutoBench's evaluation methodology, [**Bot Scanner**](https://botscanner.ai) is the "skyscanner for LLM responses." It is a live platform that allows users to route a single prompt to multiple "responder" LLMs simultaneously, and then uses AutoBench's "judge" LLMs to evaluate, rank, and deliver the absolute best answer instantly, ending LLM guesswork. |
|
|
| ### 5. AutoBench 1.0 (Open Source) |
| The foundational open-source framework that proved the Collective-LLM-as-a-Judge concept. It remains free and available for researchers and developers to run local evaluations and explore the core architecture. |
|
|
| ## Key Differentiators & Industry Correlations |
|
|
| AutoBench solves the traditional tradeoff between scalability, cost, and accuracy: |
| * **Strictly Un-gameable:** Because tasks and environments are dynamically generated at runtime, test-set contamination is impossible. Models cannot "memorize" the benchmark. |
| * **Highly Correlated (Scientific Validation):** Despite its dynamic nature, AutoBench achieves massive correlation with rigid, human-verified industry standards: |
| * **Agentic Correlations:** 85.15% with the Artificial Analysis Intelligence Index, 84.56% with GDPval-AA, and 83.00% with Terminal-Bench Hard. |
| * **Generalist Correlations:** 89.38% with the Artificial Analysis Index, 82.21% with MMLU-Pro, and 71.84% with LMSYS Chatbot Arena (Human Preference). |
| * **High Granularity & Adaptability:** Unlike one-size-fits-all tests, AutoBench's architecture easily adapts to highly specialized, domain-specific verticals. We provide granular, topic-specific performance insights—such as our recent **[Agronomic Benchmark](https://huggingface.co/blog/PeterKruger/autobench-run-agronomy-1)**, allowing enterprises to test models on their exact proprietary schemas and niche industry knowledge. |
| * **Highly Scalable & Cost-Effective:** A comprehensive benchmark evaluating 30+ models costs a fraction of human-annotated alternatives (often under $100 in raw compute). |
|
|
| ## Scientific Validation & Acknowledgements |
|
|
| Our methodology is scientifically validated and continuously peer-reviewed. We extend our immense gratitude to our partners and supporters: |
| * **Translated:** Global leader in Professioal AI-enabled translations and high-quality training human data generation for their continued support in compute resources and strategic insight. |
| * **DIAG, Sapienza Università di Roma:** The team led by **Prof. Fabrizio Silvestri** for providing the rigorous scientific validation that underpins our methodology. |
| * **eZecute:** The venture builder for enabling the industrialization and scaling of this platform. |
| * **AWS Startups:** For compute credits. |
|
|
| ## Explore, Connect, and Contribute |
|
|
| Whether you are an AI researcher, a prompt engineer, or an enterprise IT architect deploying autonomous agents, AutoBench has the data you need to stop flying blind. |
|
|
| * 🌐 **Official Website & Data Archive:** [autobench.org](https://autobench.org/) |
| * 🏆 **Interactive Agentic Leaderboard:** [AutoBench-Leaderboard](https://huggingface.co/spaces/AutoBench/AutoBench-Leaderboard) |
| * 🤖 **Test Bot Scanner:** [botscanner.ai](https://botscanner.ai/) |
| * 📖 **Read Our Official Blog:** [AutoBench Blog](https://autobench.org/blog) |
| * 💻 **Explore the OS Code:** [AutoBench 1.0 Repository](https://huggingface.co/AutoBench/AutoBench_1.0) |
| * 📝 **Read the Scientific Paper:** [arXiv:2510.22593](https://arxiv.org/abs/2510.22593) |
|
|
| *Inference Support: Running a compute-intensive benchmark like AutoBench can be expensive. We welcome all inference API providers to support us with free inference credits to expand the scope of our evaluations.* |
|
|
| ### Citation |
| If you use AutoBench in your research, please cite our validation paper: |
|
|
| ```bibtex |
| @misc{autobench2025, |
| title={AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment}, |
| author={AutoBench}, |
| year={2025}, |
| eprint={2510.22593}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={[https://arxiv.org/abs/2510.22593](https://arxiv.org/abs/2510.22593)}, |
| } |
