File size: 3,684 Bytes
a5a3893
dc09410
 
 
 
a5a3893
 
 
dc09410
 
 
a5a3893
dc09410
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
---
title: Benchgen
emoji: 📊
colorFrom: yellow
colorTo: red
sdk: static
pinned: false
---
<p align="center">
  <a href="https://benchgen.com/?utm_source=huggingface&utm_medium=org_card&utm_campaign=org&utm_content=banner"><img src="https://huggingface.co/spaces/benchgen/README/resolve/main/banner.png" alt="Benchgen: the benchmarking infrastructure for AI agents" width="100%"></a>
</p>

<p align="center">
  <a href="https://benchgen.com/?utm_source=huggingface&utm_medium=org_card&utm_campaign=org&utm_content=badge-website"><img src="https://img.shields.io/badge/WEBSITE-benchgen.com-F97316?style=for-the-badge&labelColor=111827" alt="Website: benchgen.com"></a>
  <a href="https://benchgen.com/benchmarks/platform/router-fidelity-benchmark?utm_source=huggingface&utm_medium=org_card&utm_campaign=org&utm_content=badge-benchmark"><img src="https://img.shields.io/badge/RUN%20THE%20BENCHMARK-ROUTER%20FIDELITY-F97316?style=for-the-badge&labelColor=111827" alt="Run the benchmark on Benchgen"></a>
  <a href="https://benchgen.com/docs?utm_source=huggingface&utm_medium=org_card&utm_campaign=org&utm_content=badge-docs"><img src="https://img.shields.io/badge/DOCS-benchgen.com%2Fdocs-F97316?style=for-the-badge&labelColor=111827" alt="Docs"></a>
</p>

<p align="center">
  <a href="https://github.com/benchgen-ai"><img src="https://img.shields.io/badge/GitHub-benchgen--ai-181717?style=flat&logo=github&logoColor=white" alt="GitHub"></a>
  <a href="https://x.com/BenchGenAI"><img src="https://img.shields.io/badge/X-%40BenchGenAI-000000?style=flat&logo=x&logoColor=white" alt="X"></a>
  <a href="https://www.linkedin.com/company/benchgen"><img src="https://img.shields.io/badge/LinkedIn-benchgen-0A66C2?style=flat" alt="LinkedIn"></a>
  <a href="https://huggingface.co/benchgen"><img src="https://img.shields.io/badge/Hugging%20Face-benchgen-FFD21E?style=flat&logo=huggingface&logoColor=black" alt="Hugging Face"></a>
</p>


**Nobody can prove an AI agent actually works.** Benchgen helps you discover benchmarks, run evaluations on any model or agent, find where and why it fails, and check the fix with the same tests before and after.

## What we publish here

| Release | What it is | Link |
|---|---|---|
| **Router Lite** (model) | A 10,245-parameter linear router head on a frozen Qwen3-1.7B. Picks which of 5 models should answer a query | [benchgen/benchgen-router-lite](https://huggingface.co/benchgen/benchgen-router-lite) |
| **Router Pilot rewards** (dataset) | Per-question accuracy of 5 agents over 3 attempts (46 tasks) | [benchgen/router-pilot](https://huggingface.co/datasets/benchgen/router-pilot) |
| **Router Pilot tasks** (dataset) | 1,110 normalized questions from public benchmarks, with per-source licences | [benchgen/router-pilot-tasks](https://huggingface.co/datasets/benchgen/router-pilot-tasks) |

Every card states its method, licence, limits and a link to the relevant benchmark on Benchgen.

## Run it yourself

1. Open the [Router Fidelity Benchmark](https://benchgen.com/benchmarks/platform/router-fidelity-benchmark?utm_source=huggingface&utm_medium=org_card&utm_campaign=org&utm_content=router-fidelity) (sign-in required to run).
2. Register your own model or router head, pick it on the Evaluate tab and run.
3. Compare with our published run. The [Router Lite guide](https://benchgen.com/docs/guides/router-head/benchgen-router-lite?utm_source=huggingface&utm_medium=org_card&utm_campaign=org&utm_content=guide) has the steps.

## Honest limits

Our releases are pilots. Router Lite was trained on 46 labelled tasks, and its results are directional, not a large-sample benchmark. Each card says exactly what the numbers do and do not show.