AREX-2-oQ5e-mtp
oMLX oQ5e quant of BAAI/AREX-2 (Qwen3.8-27B family, Qwen3_5ForConditionalGeneration)
with a trained MTP (multi-token prediction) draft head grafted on, for Lightning MTP
speculative decoding.
The backbone shards are byte-identical hardlinks to ../AREX-2-oQ5e; the only additions are
model-mtp.safetensors (0.36 GB) and text_config.mtp_num_hidden_layers = 1.
Provenance
| Piece | Source |
|---|---|
| Target backbone | BAAI/AREX-2 โ oQ5e (this dir's model-0000*.safetensors) |
| MTP head | numsu/AREX-2-27B-INT8-W8A16-MTP, model_mtp.safetensors (BF16, 15 tensors) |
Donor file model_mtp.safetensors SHA-256 (1,268,830,944 bytes):
38892896461c1a2c30609aebfba9b99ab0eb57a8cb49830ab98b817a398e924a โ this equals the
"trained MTP tensor SHA-256" published on that model card, i.e. the grafted head is
bit-exactly the trained artifact it documents.
That MTP module was adapted (target frozen, MTP only) on AREX-2 W8A16's own hidden states, initialised from the Qwen3.8 BF16 MTP head โ i.e. it is trained for this target, not a generic Qwen3.8 head.
What was transformed
- Layout: PyTorch/compressed-tensors โ MLX (
Linear[out,in], no conv1d in the head). - RMSNorm gammas: raw-HF stores the qwen3_5/qwen3-next head norms zero-centered; shifted
+1to MLX form (input_layernorm,post_attention_layernorm,q_norm,k_norm,pre_fc_norm_hidden,pre_fc_norm_embedding,mtp.norm). Verified against the backbone:AREX-2-oQ5enorm == raw HF norm + 1 exactly. - Quantization: the MTP layer's 7 linears (
q/k/v/o_proj,mlp.{gate,up,down}_proj) at 5-bit / group 64 / affine, matching the oQ5e global scheme.mtp.fcstays BF16 (oMLXquant_predicatekeeps the fusion projection full precision). - Draft head dropped:
numsu's checkpoint scores drafts with a 40,960-token shortlist head (mtp.draft_lm_head.weight,mtp_draft_vocab_ids.pt) that oMLX has no support for. It was measurably a no-op: every shortlist row is cosine โฅ 0.9999 / relL2 โค 1.2% versus the corresponding row of this model's 8-bitlm_head(the residual is quantization noise), so oMLX scoring drafts with the targetlm_headloses nothing.
Measured (M5 Pro, greedy, 400 tokens, code prompt)
- acceptance
260/315 (82.5%)and241/287 (84.0%); depth-1 84โ87%, depth-3 81โ83% - 3.1โ3.5 tokens/cycle, 30.4โ30.8 tok/s vs 13.5 tok/s for the same request on
AREX-2-oQ5e(โ2.25ร) - output token-identical to the non-MTP model at temperature 0, i.e. speculation is lossless
Enable with the per-model mtp_enabled model setting (oMLX auto-enabled it on discovery).
- Downloads last month
- 50