POSET length predictors

Trained length predictors for POSET (conformal poset scheduling for vLLM) and its baselines, one folder per served model. Produced by branch poset-unified of the POSET repository (PIPELINE.md there is the full recipe).

folder served model dtype max_tokens layer
r1q14b/ deepseek-ai/DeepSeek-R1-Distill-Qwen-14B float16 16,384 28

More models (Qwen3.5-27B, a 70B) will be added as folders.

What each folder holds

file what
model.json the model spec every stage reads (POSET_SPEC): model, layer, dtype, </think> id, max_tokens, ...
layer_sweep.json the layer sweep that chose layer
heads/{chat,base}_head.pt POSET's length head per traffic group: MLP 512 x 2 on [layer-L activation, 5 position features] -> sqrt(remaining output)
conformal/{chat,base}_head_a*.npz per-position split-conformal radii q(p) for alpha = 0.05, 0.1, ..., 0.9
eval_{chat,base}.json offline evaluation on held-out prompts at alpha = 0.1, per phase (prompt, last prompt token, reasoning, answer, all)
heads/{chat,base}_binned.pt trail baseline: binned decode-time head (20 equal bins over [0, max_tokens])
bert/{chat,base}_bert.pt trail baseline: BERT (bert-base-uncased) prompt predictor, 20 quantile bins of out_len

Groups: chat = WildChat + LMSYS-Chat prompts, base = ForeLen prompts.

Label. Every predictor here targets the remaining whole output, out_len - generated (reasoning, </think> and the answer), at temperature 0. Position p is 1-based from the start of the prompt.

Training data. POSET's heads, trail's heads and the BERTs train on the same prompts (chat_train + chat_train_wc2, and base_train of the POSET prompt split), with lengths from the served model's own greedy generations. Radii are calibrated on the separate calib roles; evaluation uses the eval roles.

r1q14b: evaluation at alpha = 0.1 (target coverage 0.90)

group phase coverage width MAE / token Spearman
chat all 0.889 2,436 705 0.897
chat last prompt token 0.908 1,322 439 0.797
base all 0.880 6,451 1,843 0.820
base last prompt token 0.890 3,711 1,234 0.830

Full per-phase numbers are in r1q14b/eval_{chat,base}.json. Note: r1q14b was collected at max_tokens 16,384 in float16; later models use 32,768 and the checkpoint's dtype.

Loading

from poset.predictor import load_head        # POSET repo, branch poset-unified
head = load_head("r1q14b/heads/chat_head.pt")

The checkpoints are torch.save dicts (load with weights_only=False); the radii are .npz with q_by_pos and alpha.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support