model/README.md

---

license: mit
library_name: skops
tags: [tabular-regression, xgboost, llm-inference, performance-prediction]
---


# FitCheck speed predictor

Predicts local-LLM **decode tokens/sec** from hardware + model features.
Part of [FitCheck](https://huggingface.co/spaces/build-small-hackathon/FitCheck),
the honest "what AI can your computer run" advisor.

## Method

Gradient-boosted regression (XGBoost) following the methodology of
**LLM-Pilot** (IBM, SC'24): [arXiv:2410.02425](https://arxiv.org/abs/2410.02425) —
performance prediction for LLM inference on consumer hardware. Cross-validation
groups by raw accelerator label (10-fold `GroupKFold`). **Caveat:** this is NOT
a strict leave-one-hardware-out split — roughly 48% of held-out rows still have
an equivalent hardware alias present in training, so the held-out error below is
optimistic. Read it as grouped k-fold, not clean unseen-hardware generalization.

Features: effective memory bandwidth, bytes read per token (weights + KV),
weights size, KV size, MoE active fraction, offload fraction, and the
analytical roofline prior (bandwidth / bytes). Decode is memory-bandwidth-bound;
the roofline value is included as a **feature**, but the model predicts decode
tok/s directly (it does not fit an explicit residual). Training rows all have
active_fraction=1 and offload_fraction=0, so MoE / offload inputs are
extrapolation.

## Training data

6,633 real measurements across 595 distinct
accelerators (consumer CPUs, Apple Silicon, NVIDIA/AMD GPUs), from the
[LocalScore](https://www.localscore.ai) community benchmark (Mozilla Builders /
cjpais — thank you; data attributed, not owned, takedown requests honoured).
Trained 2026-06-10.

## Holdout results (grouped k-fold by accelerator label)

| metric | roofline baseline | this model |
|---|---|---|
| median APE (bandwidth-known hardware) | 28.1% | 17.5% |
| median abs error (tok/s) | 11.63 | 9.55 |
| all hardware incl. CPUs (no baseline possible) | — | 23.6% median APE |

**Honest caveats on these numbers:**
- The model wins on **median** error but, against the *production* baseline
  (roofline + the per-token overhead the engine actually uses), it **loses** on
  mean absolute error and RMSE. It improves ~105 of 191 accelerator labels; a
  bootstrap CI on the median per-accelerator gain includes zero.
- The "likely" interval shown in the UI is a fixed ±20% heuristic, **not

  calibrated** (out-of-fold coverage ≈43%). Treat it as a rough range.
- An independent llama.cpp benchmark set (Apple unified-memory) is used as an
  out-of-source check: the shipping roofline path is ~29% median APE on it
  (28.6% on the current set), systematically under-predicting Apple speed. See
  `scripts/eval_speed_independent.py` and `artifacts/speed_independent.json`.

**Shipping rule:** the model ships because it beats the analytical baseline on
median held-out error; if a retrain fails that, FitCheck falls back to the
labelled roofline estimate.

## Limits (read this)

- Trained on **dense LLMs running fully on-device** (LocalScore's fixed grid:
  1B / 8B / 14B at Q4_K_M, varied context). The model axis generalises through
  the bytes-per-token feature, not data diversity.
- MoE and GPU->RAM offload are corrected analytically upstream, then fed
  through — those corrections are engineering estimates, labelled as such.
- Does NOT cover vision/diffusion models (compute-bound, different physics).