Q4: add the MTP drafting-vocabulary ranking (mtp.draft_rank), weights unchanged

#2
Base Compute org

Adds the MTP drafting-vocabulary ranking to Qwen3.6-35B-A3B-Q4.base. No weight changes.

What changed

  • One tensor added to the MTP head section: mtp.draft_rank, 248,320 token ids as f32, most frequent first, 993 KB. It's the ranking tools/mtp_draft_ranking.py built for Qwen3.6-35B-A3B. Its sources are chat, reasoning, code and web text plus the model's own greedy outputs; SPEED-Bench is not a source.
  • draft_vocab_size: 65536 added to the speculator config.
  • It was grafted onto the published file rather than reconverted. The ranking is appended past the end of the weights and spliced into the header. All 20,448,493,568 weight bytes and every other header byte are identical to the current file (sha256 7171b37f…881b). Same tensor name, layout and checksum (xxh64) as base-convert --mtp-draft-ranking writes.

Why

With --speculate mtp-head, BaseRT's MTP head scored every drafted token against all 248,320 rows of the output matrix (0.32 GB per draft). With the ranking it scores the top 65,536 plus every special token (75.5 MB). Verify is unchanged, so the committed output is too.

Measured on an M5 Pro

One request at a time, 13 SPEED-Bench-style prompts, 256 tokens, greedy, decode tok/s:

this file current file
MTP, BaseRT 0.3.0 library 150.6 131–134
MTP, with baseRT-internal PR #748 169.0 132.5
No speculation unchanged (logits bitwise identical)

Acceptance is about 75% either way. With --spec-bitexact, MTP output is identical to plain decode on 13/13 prompts.

⚠️ Merge together with the catalog

The baseRT hub catalog pins this file's size and sha256 (base-convert/crates/base-hub/catalog.json, in the public basecompute/baseRT repo and bundled in each release). basert pull refuses a mismatch. Update that entry when this merges:

  • size: 20515667968 → 20516661248
  • sha256: 7171b37ffccdf16dfd920f55be766e6faffa31f64e04ca9cd0064943bf72881b → c57db6a481ed630859040cf65405bb591c8de8a4dc3e62c3f70979e91669b09b
isuramanchi changed pull request status to closed

Sign up or log in to comment