| ---
|
| license: mit
|
| library_name: skops
|
| tags: [tabular-regression, xgboost, llm-inference, performance-prediction]
|
| ---
|
|
|
| # FitCheck speed predictor
|
|
|
| Predicts local-LLM **decode tokens/sec** from hardware + model features.
|
| Part of [FitCheck](https://huggingface.co/spaces/build-small-hackathon/FitCheck),
|
| the honest "what AI can your computer run" advisor.
|
|
|
| ## Method
|
|
|
| Gradient-boosted regression (XGBoost) following the methodology of
|
| **LLM-Pilot** (IBM, SC'24): [arXiv:2410.02425](https://arxiv.org/abs/2410.02425) —
|
| performance prediction for LLM inference on consumer hardware. Cross-validation
|
| groups by raw accelerator label (10-fold `GroupKFold`). **Caveat:** this is NOT
|
| a strict leave-one-hardware-out split — roughly 48% of held-out rows still have
|
| an equivalent hardware alias present in training, so the held-out error below is
|
| optimistic. Read it as grouped k-fold, not clean unseen-hardware generalization.
|
|
|
| Features: effective memory bandwidth, bytes read per token (weights + KV),
|
| weights size, KV size, MoE active fraction, offload fraction, and the
|
| analytical roofline prior (bandwidth / bytes). Decode is memory-bandwidth-bound;
|
| the roofline value is included as a **feature**, but the model predicts decode
|
| tok/s directly (it does not fit an explicit residual). Training rows all have
|
| active_fraction=1 and offload_fraction=0, so MoE / offload inputs are
|
| extrapolation.
|
|
|
| ## Training data
|
|
|
| 6,633 real measurements across 595 distinct
|
| accelerators (consumer CPUs, Apple Silicon, NVIDIA/AMD GPUs), from the
|
| [LocalScore](https://www.localscore.ai) community benchmark (Mozilla Builders /
|
| cjpais — thank you; data attributed, not owned, takedown requests honoured).
|
| Trained 2026-06-10.
|
|
|
| ## Holdout results (grouped k-fold by accelerator label)
|
|
|
| | metric | roofline baseline | this model |
|
| |---|---|---|
|
| | median APE (bandwidth-known hardware) | 28.1% | 17.5% |
|
| | median abs error (tok/s) | 11.63 | 9.55 |
|
| | all hardware incl. CPUs (no baseline possible) | — | 23.6% median APE |
|
|
|
| **Honest caveats on these numbers:**
|
| - The model wins on **median** error but, against the *production* baseline
|
| (roofline + the per-token overhead the engine actually uses), it **loses** on
|
| mean absolute error and RMSE. It improves ~105 of 191 accelerator labels; a
|
| bootstrap CI on the median per-accelerator gain includes zero.
|
| - The "likely" interval shown in the UI is a fixed ±20% heuristic, **not
|
| calibrated** (out-of-fold coverage ≈43%). Treat it as a rough range.
|
| - An independent llama.cpp benchmark set (Apple unified-memory) is used as an
|
| out-of-source check: the shipping roofline path is ~29% median APE on it
|
| (28.6% on the current set), systematically under-predicting Apple speed. See
|
| `scripts/eval_speed_independent.py` and `artifacts/speed_independent.json`.
|
|
|
| **Shipping rule:** the model ships because it beats the analytical baseline on
|
| median held-out error; if a retrain fails that, FitCheck falls back to the
|
| labelled roofline estimate.
|
|
|
| ## Limits (read this)
|
|
|
| - Trained on **dense LLMs running fully on-device** (LocalScore's fixed grid:
|
| 1B / 8B / 14B at Q4_K_M, varied context). The model axis generalises through
|
| the bytes-per-token feature, not data diversity.
|
| - MoE and GPU->RAM offload are corrected analytically upstream, then fed
|
| through — those corrections are engineering estimates, labelled as such.
|
| - Does NOT cover vision/diffusion models (compute-bound, different physics).
|
|
|
