posts/universal-eval.html

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>One Script, Every Benchmark</title>
<link rel="stylesheet" href="../style.css">
</head>
<body>
<script type="module" src="../summarize.js" defer></script><script src="../script.js" defer></script>

<div class="background"></div>

<main class="article">

<a class="back" href="../index.html">&larr; Back</a>

<div class="meta">July 2026 &middot; Tooling</div>

<h1>One Script, Every Benchmark</h1>

<p>Until today, every Bench Labs benchmark had its own eval files — a notebook here, an lm-eval config there, a custom scorer somewhere else. It worked, but reproducing a leaderboard entry meant figuring out which pipeline produced it. That's over.</p>

<h2>Download, run, done</h2>

<pre><code>curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py
pip install torch transformers
python script.py --model your/model</code></pre>

<p>That's the whole workflow. One <code>script.py</code>, hosted in the leaderboard Space itself, runs the full suite:</p>

<ul>
  <li><strong>bench-effortless-6-2026</strong> — normalized exact-match.</li>
  <li><strong>bench-easy-6-2026</strong> — the hybrid category-aware scorer (strict / flexible / semantic), same routing as the original evaluator. Plain exact-match is reported alongside.</li>
  <li><strong>bench-mid-6-2026</strong> — lm-eval-style loglikelihood over <code>target_scores</code>: <code>acc</code>, <code>acc_norm</code>, <code>soft_score</code>, <code>soft_score_norm</code>.</li>
  <li><strong>bench-AGI</strong> — auto-skipped while the scoring pipeline is reworked; it will slot in when ready.</li>
</ul>

<h2>Full category breakdown, not one number</h2>

<p>Every run reports overall, per-category, and per-subcategory scores, with macro averages and stderr:</p>

<pre><code>=== bench-mid-6-2026 (tier 3) ===
  headline [soft_score_norm]: 0.6103   (n=155, stderr=0.0388)
  also: acc=0.5806  acc_norm=0.6000  soft_score=0.5871
  category                       n  soft_score_norm
  Math                          28  0.675
    - arithmetic                 8  0.625
    - pattern                   10  0.800
    - reasoning                 10  0.500
  ...</code></pre>

<h2>Straight into the leaderboard</h2>

<p>Besides <code>results.json</code> and per-sample CSVs, the script writes <code>leaderboard.json</code> — an entry already in the exact shape of the leaderboard's <code>models.json</code>, category breakdown included. Submitting is now: run the script, paste the entry, open a PR on the Space. Same scoring for everyone, reproducible with one command.</p>

<p>Options worth knowing: <code>--benchmarks effortless,easy,mid</code> to pick tiers, <code>--limit 10</code> for a smoke test, <code>--device</code> / <code>--dtype</code> / <code>--batch-size</code> for hardware, <code>--no-chat-template</code> to force plain QA prompting. <code>--help</code> has the rest.</p>

<p><a href="https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard" target="_blank" rel="noopener">Leaderboard</a> &middot; <a href="https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/blob/main/script.py" target="_blank" rel="noopener">script.py</a></p>

<hr>

<p><em>Bench Labs &middot; Simple, Reliable, Open sourced</em></p>

</main>

</body>
</html>