README / README.md
benchgen's picture
Organization card: banner, links, what we publish
dc09410 verified
|
Raw History Blame Contribute Delete
3.68 kB
metadata
title: Benchgen
emoji: 📊
colorFrom: yellow
colorTo: red
sdk: static
pinned: false

Benchgen: the benchmarking infrastructure for AI agents

Website: benchgen.com Run the benchmark on Benchgen Docs

GitHub X LinkedIn Hugging Face

Nobody can prove an AI agent actually works. Benchgen helps you discover benchmarks, run evaluations on any model or agent, find where and why it fails, and check the fix with the same tests before and after.

What we publish here

Release What it is Link
Router Lite (model) A 10,245-parameter linear router head on a frozen Qwen3-1.7B. Picks which of 5 models should answer a query benchgen/benchgen-router-lite
Router Pilot rewards (dataset) Per-question accuracy of 5 agents over 3 attempts (46 tasks) benchgen/router-pilot
Router Pilot tasks (dataset) 1,110 normalized questions from public benchmarks, with per-source licences benchgen/router-pilot-tasks

Every card states its method, licence, limits and a link to the relevant benchmark on Benchgen.

Run it yourself

  1. Open the Router Fidelity Benchmark (sign-in required to run).
  2. Register your own model or router head, pick it on the Evaluate tab and run.
  3. Compare with our published run. The Router Lite guide has the steps.

Honest limits

Our releases are pilots. Router Lite was trained on 46 labelled tasks, and its results are directional, not a large-sample benchmark. Each card says exactly what the numbers do and do not show.