Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>Evaluation · Switchboard</title> | |
| <meta name="description" content="How Switchboard was measured: routing quality on 10,000 held-out prompts, how each build compares to the full model, speed, and the raw outputs."> | |
| <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 24 24%22%3E%3Ccircle cx=%225%22 cy=%2212%22 r=%223%22 fill=%22%23ea580c%22/%3E%3Ccircle cx=%2219%22 cy=%225%22 r=%222.4%22 fill=%22%23ea580c%22/%3E%3Ccircle cx=%2219%22 cy=%2212%22 r=%222.4%22 fill=%22%23ea580c%22/%3E%3Ccircle cx=%2219%22 cy=%2219%22 r=%222.4%22 fill=%22%23ea580c%22/%3E%3C/svg%3E"> | |
| <link rel="preconnect" href="https://fonts.googleapis.com"> | |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> | |
| <link href="https://fonts.googleapis.com/css2?family=Fraunces:opsz,wght@9..144,500;9..144,600;9..144,700&family=Inter:wght@400;500;600&family=JetBrains+Mono:wght@400;600&display=swap" rel="stylesheet"> | |
| <link rel="stylesheet" href="style.css?v=9"> | |
| </head> | |
| <body> | |
| <header class="top"> | |
| <div class="top-inner"> | |
| <a class="brand" href="index.html"> | |
| <svg class="brand-mark" viewBox="0 0 24 24" aria-hidden="true"><circle cx="5" cy="12" r="3"/><circle cx="19" cy="5" r="2.4"/><circle cx="19" cy="12" r="2.4"/><circle cx="19" cy="19" r="2.4"/><path d="M8 12c4 0 5-7 8.6-7"/></svg> | |
| Switchboard | |
| </a> | |
| <nav class="nav"> | |
| <a href="index.html#demo">Demo</a> | |
| <a href="index.html#how">How it works</a> | |
| <a href="savings.html">Savings</a> | |
| <a href="evaluation.html" aria-current="page">Evaluation</a> | |
| <a href="index.html#faq">FAQ</a> | |
| <a href="https://huggingface.co/switchboardllm/switchboard-router-0.5b">Model card</a> | |
| </nav> | |
| <a class="btn btn-primary btn-sm" href="index.html#demo">Try the router</a> | |
| </div> | |
| </header> | |
| <main> | |
| <section class="hero hero-sm"> | |
| <div class="wrap narrow center"> | |
| <p class="badge"><span class="dot" aria-hidden="true"></span>Updated October 12, 2026</p> | |
| <h1>What we measured so far</h1> | |
| <p class="lede">Every version of Switchboard is tested on the same 10,000 prompts it never saw during training. We publish the numbers that look bad too, and you can download every answer below.</p> | |
| </div> | |
| </section> | |
| <section class="band white"> | |
| <div class="wrap"> | |
| <h2 class="center">Routing quality</h2> | |
| <p class="section-lede center">10,000 held-out prompts in their natural mix: 6,931 easy, 1,716 medium and 1,353 that need a frontier model. Full-precision models, greedy decoding.</p> | |
| <div class="metrics"> | |
| <article class="metric"><header><h3>Cost saved</h3><span class="better">Higher is better</span></header><p class="plain">How much cheaper the bill is than sending every prompt to Claude Opus 5.5.</p><div class="ebars"><div class="ebar ours"><span class="ebar-label">Switchboard v0.1</span><span class="ebar-track"><span class="ebar-fill" style="width:64.5%"></span></span><span class="ebar-value">64.5%</span></div> | |
| <div class="ebar"><span class="ebar-label">Switchboard v0</span><span class="ebar-track"><span class="ebar-fill" style="width:67.8%"></span></span><span class="ebar-value">67.8%</span></div> | |
| <div class="ebar"><span class="ebar-label">Perfect router</span><span class="ebar-track"><span class="ebar-fill" style="width:76.2%"></span></span><span class="ebar-value">76.2%</span></div> | |
| <div class="ebar"><span class="ebar-label">Always small</span><span class="ebar-track"><span class="ebar-fill" style="width:97.5%"></span></span><span class="ebar-value">97.5%</span></div></div></article> | |
| <article class="metric"><header><h3>Prompts sent too low</h3><span class="better">Lower is better</span></header><p class="plain">Prompts that needed the frontier model but went to a cheaper one. This is the mistake you'd notice.</p><div class="ebars"><div class="ebar ours"><span class="ebar-label">Switchboard v0.1</span><span class="ebar-track"><span class="ebar-fill" style="width:27.6%"></span></span><span class="ebar-value">5.5%</span></div> | |
| <div class="ebar"><span class="ebar-label">Switchboard v0</span><span class="ebar-track"><span class="ebar-fill" style="width:35.5%"></span></span><span class="ebar-value">7.1%</span></div> | |
| <div class="ebar"><span class="ebar-label">Random router, same budget</span><span class="ebar-track"><span class="ebar-fill" style="width:49.9%"></span></span><span class="ebar-value">10.0%</span></div> | |
| <div class="ebar"><span class="ebar-label">Always small</span><span class="ebar-track"><span class="ebar-fill" style="width:67.7%"></span></span><span class="ebar-value">13.5%</span></div></div></article> | |
| <article class="metric"><header><h3>Hard prompts caught</h3><span class="better">Higher is better</span></header><p class="plain">Of the prompts that needed the frontier model, the share Switchboard sent there.</p><div class="ebars"><div class="ebar ours"><span class="ebar-label">Switchboard v0.1</span><span class="ebar-track"><span class="ebar-fill" style="width:59.1%"></span></span><span class="ebar-value">59.1%</span></div> | |
| <div class="ebar"><span class="ebar-label">Switchboard v0</span><span class="ebar-track"><span class="ebar-fill" style="width:47.5%"></span></span><span class="ebar-value">47.5%</span></div> | |
| <div class="ebar"><span class="ebar-label">Random router, same budget</span><span class="ebar-track"><span class="ebar-fill" style="width:26.2%"></span></span><span class="ebar-value">26.2%</span></div></div></article> | |
| <article class="metric"><header><h3>Ranking hard above easy (AUC)</h3><span class="better">Higher is better</span></header><p class="plain">How well the model orders prompts by difficulty, whatever the cut-off. 0.5 is a coin flip, 1.0 is perfect.</p><div class="ebars"><div class="ebar ours"><span class="ebar-label">Switchboard v0.1</span><span class="ebar-track"><span class="ebar-fill" style="width:78.7%"></span></span><span class="ebar-value">0.787</span></div> | |
| <div class="ebar"><span class="ebar-label">Switchboard v0</span><span class="ebar-track"><span class="ebar-fill" style="width:78.5%"></span></span><span class="ebar-value">0.785</span></div> | |
| <div class="ebar"><span class="ebar-label">Coin flip</span><span class="ebar-track"><span class="ebar-fill" style="width:50.0%"></span></span><span class="ebar-value">0.500</span></div></div></article> | |
| <article class="metric"><header><h3>Unreadable answers</h3><span class="better">Lower is better</span></header><p class="plain">Answers that weren't valid JSON, out of 10,000. Those would fall back to frontier.</p><div class="ebars"><div class="ebar ours"><span class="ebar-label">Switchboard v0.1</span><span class="ebar-track"><span class="ebar-fill" style="width:0.0%"></span></span><span class="ebar-value">0</span></div> | |
| <div class="ebar"><span class="ebar-label">Switchboard v0</span><span class="ebar-track"><span class="ebar-fill" style="width:0.0%"></span></span><span class="ebar-value">0</span></div></div></article> | |
| </div> | |
| <p class="fineprint center">The random router sends the same share of prompts to frontier as Switchboard v0.1, picked at random. A perfect router knows every answer in advance.</p> | |
| </div> | |
| </section> | |
| <section class="band cream"> | |
| <div class="wrap narrow"> | |
| <h2 class="center">Does every build route the same way?</h2> | |
| <p class="section-lede center">Smaller files make the model faster to download, but they can change its decisions. We checked each build against the full model on 347 test prompts: 300 picked at random from the 10,000 above, plus 47 borderline ones that cover all three tiers.</p> | |
| <div class="table-wrap"> | |
| <table class="etable"> | |
| <thead><tr><th>Build</th><th class="num">Size</th><th class="num">Same decision</th><th></th></tr></thead> | |
| <tbody> | |
| <tr><td class="build"><b>GGUF F16</b><small>Ollama 0.40.3 via ollama run hf.co/..., temperature 0 from the repo params</small><small class="note">347 in order, 346 when the prompts were sent in random order (Ollama reuses cached prompt prefixes)</small></td><td class="num">994 MB</td><td class="num"><b>347 of 347</b><small>100.0%</small></td><td><span class="tag-yes">Shipped</span></td></tr> | |
| <tr><td class="build"><b>Browser ONNX, 8-bit</b><small>Transformers.js 4.2.0, WebGPU, desktop Chrome</small><small class="note">299 of 300 random, 45 of 47 borderline; domain 347 of 347</small></td><td class="num">660 MB</td><td class="num"><b>344 of 347</b><small>99.1%</small></td><td><span class="tag-yes">Shipped</span></td></tr> | |
| <tr><td class="build"><b>GGUF Q8_0</b><small>Ollama 0.40.3, GPU</small><small class="note">338 of 347 on CPU</small></td><td class="num">531 MB</td><td class="num"><b>333 of 347</b><small>96.0%</small></td><td><span class="tag-no">Not shipped</span></td></tr> | |
| <tr><td class="build"><b>GGUF Q4_K_M</b><small>Ollama 0.40.3, GPU</small></td><td class="num">398 MB</td><td class="num"><b>325 of 347</b><small>93.7%</small></td><td><span class="tag-no">Not shipped</span></td></tr> | |
| <tr><td class="build"><b>GGUF Q6_K</b><small>Ollama 0.40.3, GPU</small></td><td class="num">506 MB</td><td class="num"><b>320 of 347</b><small>92.2%</small></td><td><span class="tag-no">Not shipped</span></td></tr> | |
| <tr><td class="build"><b>Browser ONNX, 4-bit layers</b><small>onnxruntime 1.24, CPU</small><small class="note">measured on the 300 random prompts only</small></td><td class="num">463 MB</td><td class="num"><b>197 of 300</b><small>65.7%</small></td><td><span class="tag-no">Not shipped</span></td></tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <p class="fineprint">For a router, a changed decision is a wrong answer, so we only ship builds that match the full model. 4-bit builds of a 0.5B model lose too much precision on the borderline prompts.</p> | |
| </div> | |
| </section> | |
| <section class="band white"> | |
| <div class="wrap narrow"> | |
| <h2 class="center">Speed</h2> | |
| <p class="section-lede center">Time to get one routing decision, measured on one desktop PC.</p> | |
| <div class="table-wrap"> | |
| <table class="etable"> | |
| <thead><tr><th>Setup</th><th class="num">Median</th><th class="num">Slowest 10%</th></tr></thead> | |
| <tbody><tr><td>GGUF F16 in Ollama, desktop CPU</td><td class="num">281 ms</td><td class="num">336 ms</td></tr><tr><td>Browser ONNX, WebGPU in desktop Chrome<small>after the one-time 660 MB download</small></td><td class="num">400 ms</td><td class="num">460 ms</td></tr></tbody> | |
| </table> | |
| </div> | |
| </div> | |
| </section> | |
| <section class="band cream" id="raw"> | |
| <div class="wrap narrow"> | |
| <h2 class="center">Every answer, downloadable</h2> | |
| <p class="section-lede center">One JSON line per prompt: the prompt, its label, the Mixtral score behind the label, Switchboard's raw output and decision, and whether the prompt was served well.</p> | |
| <ul class="downloads"> | |
| <li><a href="data/outputs-v0.1.jsonl" download>Raw outputs v0.1</a> <small>6.0 MB, 10,000 lines</small></li> | |
| <li><a href="data/outputs-v0.jsonl" download>Raw outputs v0</a> <small>6.0 MB, 10,000 lines</small></li> | |
| </ul> | |
| </div> | |
| </section> | |
| <section class="band white" id="method"> | |
| <div class="wrap narrow"> | |
| <h2 class="center">How we tested</h2> | |
| <dl class="method"> | |
| <dt>Prompts</dt> | |
| <dd>The validation split of <a href="https://huggingface.co/datasets/routellm/gpt4_dataset">routellm/gpt4_dataset</a> (Apache-2.0, from LMSYS): 10,000 real prompts from chat logs, instruction sets and coding questions. None were used in training.</dd> | |
| <dt>What "needs frontier" means</dt> | |
| <dd>Every prompt comes with a 2023 Mixtral-8x7B answer that a GPT-4 judge scored from 1 to 5. A score of 5 means a small model is enough, 4 means mid, and 1 to 3 means frontier. A 2026 small model is stronger than Mixtral-8x7B, so this label is cautious: some prompts marked frontier would be fine on a smaller model.</dd> | |
| <dt>Served well</dt> | |
| <dd>A prompt counts as served well if Switchboard sent it to frontier, or sent it lower and the Mixtral answer scored 4 or 5.</dd> | |
| <dt>Cost</dt> | |
| <dd>Output price per 1M tokens on Surplus: Claude Haiku 5.5 ($0.50) for small, Claude Sonnet 5.5 ($10) for mid, Claude Opus 5.5 ($20) for frontier. Savings compare the routed bill with sending every prompt to Opus.</dd> | |
| <dt>Settings</dt> | |
| <dd>Greedy decoding (temperature 0), at most 24 new tokens, prompts cut to 512 tokens, the default chat template with no extra instructions.</dd> | |
| <dt>Builds</dt> | |
| <dd>The full model in fp32 is the reference. Each build answered the same prompts and we counted how often its tier matched. GGUF builds ran in Ollama 0.40.3, the browser build in Transformers.js 4.2.0 with WebGPU in desktop Chrome.</dd> | |
| <dt>Where it ran</dt> | |
| <dd>One desktop PC with an RTX 5070 Ti, for training, evaluation and the speed numbers.</dd> | |
| <dt>Limits</dt> | |
| <dd>The labels come from models and a judge, not from people. Most prompts are English. Each number is one run, so differences of a few tenths of a point don't mean much. On prompts that sit right between two tiers, tiny numeric differences can flip the decision.</dd> | |
| </dl> | |
| </div> | |
| </section> | |
| </main> | |
| </body> | |
| </html> | |