AREX-2-oQ5e-mtp

oMLX oQ5e quant of BAAI/AREX-2 (Qwen3.8-27B family, Qwen3_5ForConditionalGeneration) with a trained MTP (multi-token prediction) draft head grafted on, for Lightning MTP speculative decoding.

The backbone shards are byte-identical hardlinks to ../AREX-2-oQ5e; the only additions are model-mtp.safetensors (0.36 GB) and text_config.mtp_num_hidden_layers = 1.

Provenance

Piece Source
Target backbone BAAI/AREX-2 โ†’ oQ5e (this dir's model-0000*.safetensors)
MTP head numsu/AREX-2-27B-INT8-W8A16-MTP, model_mtp.safetensors (BF16, 15 tensors)

Donor file model_mtp.safetensors SHA-256 (1,268,830,944 bytes): 38892896461c1a2c30609aebfba9b99ab0eb57a8cb49830ab98b817a398e924a โ€” this equals the "trained MTP tensor SHA-256" published on that model card, i.e. the grafted head is bit-exactly the trained artifact it documents.

That MTP module was adapted (target frozen, MTP only) on AREX-2 W8A16's own hidden states, initialised from the Qwen3.8 BF16 MTP head โ€” i.e. it is trained for this target, not a generic Qwen3.8 head.

What was transformed

  • Layout: PyTorch/compressed-tensors โ†’ MLX (Linear [out,in], no conv1d in the head).
  • RMSNorm gammas: raw-HF stores the qwen3_5/qwen3-next head norms zero-centered; shifted +1 to MLX form (input_layernorm, post_attention_layernorm, q_norm, k_norm, pre_fc_norm_hidden, pre_fc_norm_embedding, mtp.norm). Verified against the backbone: AREX-2-oQ5e norm == raw HF norm + 1 exactly.
  • Quantization: the MTP layer's 7 linears (q/k/v/o_proj, mlp.{gate,up,down}_proj) at 5-bit / group 64 / affine, matching the oQ5e global scheme. mtp.fc stays BF16 (oMLX quant_predicate keeps the fusion projection full precision).
  • Draft head dropped: numsu's checkpoint scores drafts with a 40,960-token shortlist head (mtp.draft_lm_head.weight, mtp_draft_vocab_ids.pt) that oMLX has no support for. It was measurably a no-op: every shortlist row is cosine โ‰ฅ 0.9999 / relL2 โ‰ค 1.2% versus the corresponding row of this model's 8-bit lm_head (the residual is quantization noise), so oMLX scoring drafts with the target lm_head loses nothing.

Measured (M5 Pro, greedy, 400 tokens, code prompt)

  • acceptance 260/315 (82.5%) and 241/287 (84.0%); depth-1 84โ€“87%, depth-3 81โ€“83%
  • 3.1โ€“3.5 tokens/cycle, 30.4โ€“30.8 tok/s vs 13.5 tok/s for the same request on AREX-2-oQ5e (โ‰ˆ2.25ร—)
  • output token-identical to the non-MTP model at temperature 0, i.e. speculation is lossless

Enable with the per-model mtp_enabled model setting (oMLX auto-enabled it on discovery).

Downloads last month
50
Safetensors
Model size
28B params
Tensor type
U32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Estirius/AREX-2-oQ5e-mtp

Base model

Qwen/Qwen3.8-27B
Finetuned
BAAI/AREX-2
Quantized
(21)
this model