POSET length predictors
Trained length predictors for POSET (conformal poset scheduling for vLLM) and its baselines, one folder per
served model. Produced by branch poset-unified of the POSET repository (PIPELINE.md there is the full recipe).
| folder | served model | dtype | max_tokens | layer |
|---|---|---|---|---|
r1q14b/ |
deepseek-ai/DeepSeek-R1-Distill-Qwen-14B | float16 | 16,384 | 28 |
More models (Qwen3.5-27B, a 70B) will be added as folders.
What each folder holds
| file | what |
|---|---|
model.json |
the model spec every stage reads (POSET_SPEC): model, layer, dtype, </think> id, max_tokens, ... |
layer_sweep.json |
the layer sweep that chose layer |
heads/{chat,base}_head.pt |
POSET's length head per traffic group: MLP 512 x 2 on [layer-L activation, 5 position features] -> sqrt(remaining output) |
conformal/{chat,base}_head_a*.npz |
per-position split-conformal radii q(p) for alpha = 0.05, 0.1, ..., 0.9 |
eval_{chat,base}.json |
offline evaluation on held-out prompts at alpha = 0.1, per phase (prompt, last prompt token, reasoning, answer, all) |
heads/{chat,base}_binned.pt |
trail baseline: binned decode-time head (20 equal bins over [0, max_tokens]) |
bert/{chat,base}_bert.pt |
trail baseline: BERT (bert-base-uncased) prompt predictor, 20 quantile bins of out_len |
Groups: chat = WildChat + LMSYS-Chat prompts, base = ForeLen prompts.
Label. Every predictor here targets the remaining whole output, out_len - generated (reasoning,
</think> and the answer), at temperature 0. Position p is 1-based from the start of the prompt.
Training data. POSET's heads, trail's heads and the BERTs train on the same prompts (chat_train + chat_train_wc2, and base_train of the POSET prompt split), with lengths from the served model's own greedy generations. Radii are calibrated on the separate calib roles; evaluation uses the eval roles.
r1q14b: evaluation at alpha = 0.1 (target coverage 0.90)
| group | phase | coverage | width | MAE / token | Spearman |
|---|---|---|---|---|---|
| chat | all | 0.889 | 2,436 | 705 | 0.897 |
| chat | last prompt token | 0.908 | 1,322 | 439 | 0.797 |
| base | all | 0.880 | 6,451 | 1,843 | 0.820 |
| base | last prompt token | 0.890 | 3,711 | 1,234 | 0.830 |
Full per-phase numbers are in r1q14b/eval_{chat,base}.json. Note: r1q14b was collected at max_tokens 16,384 in
float16; later models use 32,768 and the checkpoint's dtype.
Loading
from poset.predictor import load_head # POSET repo, branch poset-unified
head = load_head("r1q14b/heads/chat_head.pt")
The checkpoints are torch.save dicts (load with weights_only=False); the radii are .npz with q_by_pos and
alpha.