Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>One Script, Every Benchmark</title> | |
| <link rel="stylesheet" href="../style.css"> | |
| </head> | |
| <body> | |
| <script type="module" src="../summarize.js" defer></script><script src="../script.js" defer></script> | |
| <div class="background"></div> | |
| <main class="article"> | |
| <a class="back" href="../index.html">← Back</a> | |
| <div class="meta">July 2026 · Tooling</div> | |
| <h1>One Script, Every Benchmark</h1> | |
| <p>Until today, every Bench Labs benchmark had its own eval files — a notebook here, an lm-eval config there, a custom scorer somewhere else. It worked, but reproducing a leaderboard entry meant figuring out which pipeline produced it. That's over.</p> | |
| <h2>Download, run, done</h2> | |
| <pre><code>curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py | |
| pip install torch transformers | |
| python script.py --model your/model</code></pre> | |
| <p>That's the whole workflow. One <code>script.py</code>, hosted in the leaderboard Space itself, runs the full suite:</p> | |
| <ul> | |
| <li><strong>bench-effortless-6-2026</strong> — normalized exact-match.</li> | |
| <li><strong>bench-easy-6-2026</strong> — the hybrid category-aware scorer (strict / flexible / semantic), same routing as the original evaluator. Plain exact-match is reported alongside.</li> | |
| <li><strong>bench-mid-6-2026</strong> — lm-eval-style loglikelihood over <code>target_scores</code>: <code>acc</code>, <code>acc_norm</code>, <code>soft_score</code>, <code>soft_score_norm</code>.</li> | |
| <li><strong>bench-AGI</strong> — auto-skipped while the scoring pipeline is reworked; it will slot in when ready.</li> | |
| </ul> | |
| <h2>Full category breakdown, not one number</h2> | |
| <p>Every run reports overall, per-category, and per-subcategory scores, with macro averages and stderr:</p> | |
| <pre><code>=== bench-mid-6-2026 (tier 3) === | |
| headline [soft_score_norm]: 0.6103 (n=155, stderr=0.0388) | |
| also: acc=0.5806 acc_norm=0.6000 soft_score=0.5871 | |
| category n soft_score_norm | |
| Math 28 0.675 | |
| - arithmetic 8 0.625 | |
| - pattern 10 0.800 | |
| - reasoning 10 0.500 | |
| ...</code></pre> | |
| <h2>Straight into the leaderboard</h2> | |
| <p>Besides <code>results.json</code> and per-sample CSVs, the script writes <code>leaderboard.json</code> — an entry already in the exact shape of the leaderboard's <code>models.json</code>, category breakdown included. Submitting is now: run the script, paste the entry, open a PR on the Space. Same scoring for everyone, reproducible with one command.</p> | |
| <p>Options worth knowing: <code>--benchmarks effortless,easy,mid</code> to pick tiers, <code>--limit 10</code> for a smoke test, <code>--device</code> / <code>--dtype</code> / <code>--batch-size</code> for hardware, <code>--no-chat-template</code> to force plain QA prompting. <code>--help</code> has the rest.</p> | |
| <p><a href="https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard" target="_blank" rel="noopener">Leaderboard</a> · <a href="https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/blob/main/script.py" target="_blank" rel="noopener">script.py</a></p> | |
| <hr> | |
| <p><em>Bench Labs · Simple, Reliable, Open sourced</em></p> | |
| </main> | |
| </body> | |
| </html> | |