Instructions to use ToPo-ToPo/Ornith-1.5-35B-A3B-MTP-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use ToPo-ToPo/Ornith-1.5-35B-A3B-MTP-bf16 with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download ToPo-ToPo/Ornith-1.5-35B-A3B-MTP-bf16 --local-dir Ornith-1.5-35B-A3B-MTP-bf16
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
ToPo-ToPo/Ornith-1.5-35B-A3B-MTP-bf16
MTP (multi-token prediction) drafter for speculative decoding with
ornith-ai/Ornith-1.5-35B-A3B on Apple Silicon (mlx-vlm).
This is not a standalone chat model โ it only works bound to an Ornith-1.5-35B-A3B target model.
The MLX conversions (ToPo-ToPo/Ornith-1.5-35B-A3B-mlx-4bit,
ornith-ai/Ornith-1.5-35B-A3B-MLX-4bit, โฆ) do not include the MTP weights.
Provenance
- Split from the official checkpoint's built-in
mtp.*weights (785 tensors inmodel-00016-of-00016.safetensors, revision10fbf86f) with mlx-vlm 0.7.10 (mlx 0.32.4):python -m mlx_vlm.speculative.drafters.qwen3_5_mtp.split --model ornith-ai/Ornith-1.5-35B-A3B --output .(the 256 per-expert MLPs are stacked intoswitch_mlp, giving 20 tensors) - Precision: bf16 (unquantized, split as-is)
- Size: 1.7 GB,
block_size: 3,model_type: qwen3_5_mtp
Usage
mlx_vlm.generate --model ornith-ai/Ornith-1.5-35B-A3B \
--draft-model ToPo-ToPo/Ornith-1.5-35B-A3B-MTP-bf16 --draft-kind mtp \
--prompt "..." --max-tokens 400
Measured (Apple M4 Max, 128 GB, mlx-vlm 0.7.10, greedy, 200 tokens)
| Target | Workload | No MTP | MTP (block 3) | Acceptance |
|---|---|---|---|---|
bf16 (ornith-ai/Ornith-1.5-35B-A3B) |
code | 59.4 tok/s | 79.7 tok/s (1.34x) | 75.9% |
| bf16 | Japanese prose | 61.4 tok/s | 69.1 tok/s (1.13x) | 39.6% |
4bit (ToPo-ToPo/Ornith-1.5-35B-A3B-mlx-4bit) |
code | 114.2 tok/s | 103.6 tok/s (0.91x) | 69.5% |
| 4bit | Japanese prose | 110.2 tok/s | 95.4 tok/s (0.87x) | 36.4% |
Use it with the bf16 target. With a 4bit target the drafter is still accepted well (87.7% on code at block size 2), but the A3B MoE target is already so fast that drafting and batched verification cost more than they save โ MTP is slower than plain decoding there. Speedup depends on hardware, target quantization, workload and the mlx-vlm version โ measure it yourself.
- Downloads last month
- 88
Quantized
Model tree for ToPo-ToPo/Ornith-1.5-35B-A3B-MTP-bf16
Base model
ornith-ai/Ornith-1.5-35B-A3B