model/README.md

metadata

license: mit
library_name: skops
tags:
  - tabular-regression
  - xgboost
  - llm-inference
  - performance-prediction

FitCheck speed predictor

Predicts local-LLM decode tokens/sec from hardware + model features. Part of FitCheck, the honest "what AI can your computer run" advisor.

Method

Gradient-boosted regression (XGBoost) following the methodology of LLM-Pilot (IBM, SC'24): arXiv:2410.02425 — performance prediction for LLM inference on consumer hardware. Cross-validation groups by raw accelerator label (10-fold GroupKFold). Caveat: this is NOT a strict leave-one-hardware-out split — roughly 48% of held-out rows still have an equivalent hardware alias present in training, so the held-out error below is optimistic. Read it as grouped k-fold, not clean unseen-hardware generalization.

Features: effective memory bandwidth, bytes read per token (weights + KV), weights size, KV size, MoE active fraction, offload fraction, and the analytical roofline prior (bandwidth / bytes). Decode is memory-bandwidth-bound; the roofline value is included as a feature, but the model predicts decode tok/s directly (it does not fit an explicit residual). Training rows all have active_fraction=1 and offload_fraction=0, so MoE / offload inputs are extrapolation.

Training data

6,633 real measurements across 595 distinct accelerators (consumer CPUs, Apple Silicon, NVIDIA/AMD GPUs), from the LocalScore community benchmark (Mozilla Builders / cjpais — thank you; data attributed, not owned, takedown requests honoured). Trained 2026-06-10.

Holdout results (grouped k-fold by accelerator label)

metric	roofline baseline	this model
median APE (bandwidth-known hardware)	28.1%	17.5%
median abs error (tok/s)	11.63	9.55
all hardware incl. CPUs (no baseline possible)	—	23.6% median APE

Honest caveats on these numbers:

• The model wins on median error but, against the production baseline (roofline + the per-token overhead the engine actually uses), it loses on mean absolute error and RMSE. It improves ~105 of 191 accelerator labels; a bootstrap CI on the median per-accelerator gain includes zero.
• The "likely" interval shown in the UI is a fixed ±20% heuristic, not calibrated (out-of-fold coverage ≈43%). Treat it as a rough range.
• An independent llama.cpp benchmark set (Apple unified-memory) is used as an out-of-source check: the shipping roofline path is ~29% median APE on it (28.6% on the current set), systematically under-predicting Apple speed. See scripts/eval_speed_independent.py and artifacts/speed_independent.json.

Shipping rule: the model ships because it beats the analytical baseline on median held-out error; if a retrain fails that, FitCheck falls back to the labelled roofline estimate.

Limits (read this)

• Trained on dense LLMs running fully on-device (LocalScore's fixed grid: 1B / 8B / 14B at Q4_K_M, varied context). The model axis generalises through the bytes-per-token feature, not data diversity.
• MoE and GPU->RAM offload are corrected analytically upstream, then fed through — those corrections are engineering estimates, labelled as such.
• Does NOT cover vision/diffusion models (compute-bound, different physics).