File size: 2,611 Bytes
3489f1d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | ---
license: apache-2.0
base_model: Kwaipilot/KAT-Coder-V2.5-Dev
tags:
- qwen3_5_moe
- code
- mxfp4
- compressed-tensors
- vllm
- xpu
library_name: vllm
---
# KAT-Coder-V2.5-Dev-MXFP4
MXFP4 (`compressed-tensors` `mxfp4-pack-quantized`: E2M1 group-32 symmetric,
E8M0 scales) quantization of [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev)
(35B-A3B MoE, 40 layers / 256 experts / 8 active), built for the
**vLLM XPU backend** (`CompressedTensorsW4A4Mxfp4` + `XPUExpertsMxFp4` on Intel Arc / Xe2).
## Contents
- **Trunk**: all 30,720 routed-expert projections (per-expert gate/up/down) +
150 linear-attention projections, quantized with the inference kernel's own
quantizer (`torch.ops.vllm.xpu_mxfp4_quantize` — format match by construction).
Everything else (self-attn, norms, shared experts, embed, lm_head) is unchanged BF16.
- **MTP draft block** (`mtp.*`, from
[SpectreSystems/KAT-Coder-V2.5-Dev-MTP](https://huggingface.co/SpectreSystems/KAT-Coder-V2.5-Dev-MTP)):
768 experts unfused and MXFP4-quantized, 17 support tensors BF16 — ready for
`--speculative-config '{"method":"mtp","num_speculative_tokens":2}'`.
- `text_config.mtp_num_hidden_layers` is set to 1 (the base checkpoint ships 0,
which would build a zero-layer draft).
63,756 tensors / 29 shards / **19.60 GiB**. `SHA256SUMS` included.
## Measured quality of the quantization
- In-conversion spot checks (kernel dequantizer, 1-in-64): max mean rel-err 0.134.
- Independent post-hoc CPU audit of 2,075 pairs vs fresh BF16: mean 0.116,
max 0.157 on a single pair (outlier-group concentrated, median-normal).
## Measured serving (Intel Arc 140V, vLLM 0.30.0+xpu, eager)
| Config | 1024/128/1 decode |
|---|---|
| Eager | ~13–14 tok/s (E-cores; ~19 expected on P-cores) |
| MTP K=1 | 23.22 tok/s |
| MTP K=2 | 24.82 tok/s |
10/10 golden-suite prompts clean, 0 errors. Example launch:
```bash
vllm serve KAT-Coder-V2.5-Dev-MXFP4 --served-model-name KAT-Coder-V2.5-Dev \
--max-model-len 4096 --max-num-seqs 1 --gpu-memory-utilization 0.68 \
--language-model-only --enforce-eager \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--kv-cache-memory-bytes 500000000
```
## Provenance
Built with the `quantize_kat_mxfp4.py` / `quantize_kat_mtp_mxfp4.py` /
`finalize_kat_mxfp4.py` / `audit_kat_mxfp4.py` pipeline
(see [alpha2beta/vllm.xpu](https://github.com/alpha2beta/vllm.xpu)).
Base weights: `Kwaipilot/KAT-Coder-V2.5-Dev` (Apache-2.0).
MTP weights: `SpectreSystems/KAT-Coder-V2.5-Dev-MTP`.
Reference schema: `pahajokiconsulting/Qwen3.6-35B-A3B-MXFP4`.
|