Q4: add the MTP drafting-vocabulary ranking (mtp.draft_rank), weights unchanged

#3
Base Compute org

Adds the MTP drafting-vocabulary ranking to Qwen3.8-27B-Q4.base. No weight changes.

What changed

  • One tensor added to the MTP head section: mtp.draft_rank, 248,320 token ids as f32, most frequent first, 993 KB. It's the Qwen3.8-27B ranking built with tools/mtp_draft_ranking.py; its tokenizer hash d868d01502157b74 is this bundle's own.
  • draft_vocab_size: 65536 added to the speculator config.
  • It was grafted onto the published file with tools/internal/graft_mtp_draft_rank.py (baseRT-internal #750), not reconverted. All 17,773,920,256 weight bytes and every other header byte are identical to the current file (sha256 03978674…0526, matching the catalog pin).

Why

With --speculate mtp-head, the MTP head scored every drafted token against all 248,320 output rows, about 715 MB per draft. It now scores the top 65,536 plus every special token (189 MB). Verify is unchanged.

Measured on an M5 Pro

BaseRT 0.3.0 library through superfluid, one request at a time, 13 SPEED-Bench-style prompts, 256 tokens, greedy, decode tok/s (two runs each, in alternating order):

this file current file
MTP 32.8, 34.0 31.7, 31.4
Acceptance 71% 69%
No speculation unchanged (logits bitwise identical) 18.6

The gain is smaller than on Qwen3.6-35B-A3B (+6% here vs +13%). On a dense 27B, each verify pass reads the whole 17.8 GB model, so drafting is a smaller share of the time.

With --spec-bitexact, MTP output equals plain decode on 12/13 prompts. The current file is also 12/13 (on a different prompt): bit-exact verify is guaranteed on routed-MoE models, not on this dense one. So the graft doesn't change that.

⚠️ Merge together with the catalog

The baseRT hub catalog pins this file's size and sha256 (base-convert/crates/base-hub/catalog.json, in the public basecompute/baseRT repo and bundled in each release). basert pull refuses a mismatch. Update that entry when this merges:

  • size: 17841094656 → 17842087936
  • sha256: 03978674a487c567f0e5378b725b8ee1226bf554e8ecafa72943a4e7b99a0526 → 527c522a441900029b133e76316fd1f58e1bcbd86c5341ccc5315ee4d9ed8de6
Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment