ToPo-ToPo/Ornith-1.5-35B-A3B-MTP-bf16

MTP (multi-token prediction) drafter for speculative decoding with ornith-ai/Ornith-1.5-35B-A3B on Apple Silicon (mlx-vlm).

This is not a standalone chat model โ€” it only works bound to an Ornith-1.5-35B-A3B target model. The MLX conversions (ToPo-ToPo/Ornith-1.5-35B-A3B-mlx-4bit, ornith-ai/Ornith-1.5-35B-A3B-MLX-4bit, โ€ฆ) do not include the MTP weights.

Provenance

  • Split from the official checkpoint's built-in mtp.* weights (785 tensors in model-00016-of-00016.safetensors, revision 10fbf86f) with mlx-vlm 0.7.10 (mlx 0.32.4): python -m mlx_vlm.speculative.drafters.qwen3_5_mtp.split --model ornith-ai/Ornith-1.5-35B-A3B --output . (the 256 per-expert MLPs are stacked into switch_mlp, giving 20 tensors)
  • Precision: bf16 (unquantized, split as-is)
  • Size: 1.7 GB, block_size: 3, model_type: qwen3_5_mtp

Usage

mlx_vlm.generate --model ornith-ai/Ornith-1.5-35B-A3B \
  --draft-model ToPo-ToPo/Ornith-1.5-35B-A3B-MTP-bf16 --draft-kind mtp \
  --prompt "..." --max-tokens 400

Measured (Apple M4 Max, 128 GB, mlx-vlm 0.7.10, greedy, 200 tokens)

Target Workload No MTP MTP (block 3) Acceptance
bf16 (ornith-ai/Ornith-1.5-35B-A3B) code 59.4 tok/s 79.7 tok/s (1.34x) 75.9%
bf16 Japanese prose 61.4 tok/s 69.1 tok/s (1.13x) 39.6%
4bit (ToPo-ToPo/Ornith-1.5-35B-A3B-mlx-4bit) code 114.2 tok/s 103.6 tok/s (0.91x) 69.5%
4bit Japanese prose 110.2 tok/s 95.4 tok/s (0.87x) 36.4%

Use it with the bf16 target. With a 4bit target the drafter is still accepted well (87.7% on code at block size 2), but the A3B MoE target is already so fast that drafting and batched verification cost more than they save โ€” MTP is slower than plain decoding there. Speedup depends on hardware, target quantization, workload and the mlx-vlm version โ€” measure it yourself.

Downloads last month
88
Safetensors
Model size
0.8B params
Tensor type
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ToPo-ToPo/Ornith-1.5-35B-A3B-MTP-bf16

Finetuned
(40)
this model