Compute Budget Evaluator
A transparent decision aid for the question "how much inference compute should I spend on this request?", built directly from measured cost–performance points.
What this is
Two operations, both of which are lookups and interpolations over measurements rather than learned functions:
- Evaluate a budget. Given a token budget and a task family (optionally a difficulty tier), return the accuracy measured at that budget, an interval, and the local marginal gain per additional 1,000 tokens.
- Meet a target. Given a target accuracy, return the cheapest measured budget level that reached it.
from router import ComputeBudgetRouter
r = ComputeBudgetRouter.from_pretrained(".")
print(r.recommend(domain="arithmetic", difficulty=3, token_budget=160))
print(r.recommend(domain="code_reasoning", target_accuracy=0.5))
What this is not
It is not a trained router, and the original study deliberately did not ship one. A router is a function from task features to budget. Fitting one is only meaningful if the routing decision can be evaluated on held-out data that the fit never saw. The pilot behind this table is far too small for that: it contains a handful of tasks per family, a single model checkpoint, and one hardware configuration. A fitted router would therefore be an untested policy presented as a tool.
What is shipped instead is the measured surface plus an explicit validity contract:
- outside the measured token range →
extrapolated: trueand a warning; - a task family or tier that was not measured → falls back to a coarser curve and
says which one it used (
curve_source); - a different model checkpoint → not valid. The table records the checkpoint it was measured on. Accuracy at a given budget is a property of (model, task family, prompt template), not of the budget alone.
Files
| File | Purpose |
|---|---|
routing_table.json |
Measured points per family and tier, with intervals and policy costs |
router.py |
ComputeBudgetRouter: interpolation + explicit extrapolation flags |
example.py |
Runnable demonstration |
requirements.txt |
No dependencies beyond the standard library |
routing_table.json structure
{
"schema_version": "1.0.0",
"model_checkpoint": "<the checkpoint this was measured on>",
"measured_token_range": [48, 160],
"families": {
"arithmetic": {
"all": [{"tokens": 48, "accuracy": 0.33, "ci_lo": 0.0, "ci_hi": 0.71, "n": 3}, ...],
"by_difficulty": {"1": [...], "3": [...], "5": [...]},
"policies": [{"config_id": "cot_ladder", "accuracy": ..., "tokens_per_correct": ...}]
}
},
"global": {"all": [...], "policies": [...]}
}
How to read the numbers
accuracyis verified accuracy on the released tasks, not a model self-report.ci_lo/ci_hiare 95% bootstrap intervals over tasks. With 3 tasks per family, these intervals are wide — that is the honest picture, and it is why the recommendation API prefers "cheapest measured level" over a fitted optimum.tokens_per_correctis total completion tokens divided by the number of verified-correct answers. It is the quantity to minimise if cost matters more than raw accuracy.- A negative
marginal_gain_per_1k_tokensmeans additional compute was, at that point, making the answer no better and the bill larger.
Intended use
Deciding whether a proposed budget is inside the range where measurement exists; comparing policies at matched token expenditure; teaching cost-aware evaluation.
Not intended for: production routing, deciding budgets for models other than the one recorded, or any use where an incorrect budget allocation has consequences. The tool reports what was measured and refuses (by flagging) to pretend otherwise.
Limitations
- One model checkpoint, one hardware configuration, a small task sample.
- Interpolation between three budget levels is piecewise linear by construction, not a fitted scaling law.
- No task-feature model:
domainanddifficultyare the only conditioning variables. Predicting difficulty from a prompt is a separate problem this artefact does not solve. - Confidence and calibration are reported in the companion dataset, not here.
Citation
@misc{ttcbench_eval2026,
title = {Compute Budget Evaluator: measured cost-performance for inference-time compute},
author = {Hafeez, Fahad},
year = {2026},
note = {Companion artefact to Test-Time Compute Bench},
url = {https://huggingface.co/fadhafeezofficial/compute-budget-evaluator}
}