KAT-Coder-V2.5-Dev-oQ2

This model was quantized using oQ (oMLX v0.5.4) mixed-precision quantization.

• Base model: Kwaipilot/KAT-Coder-V2.5-Dev
• Chat template: Qwen Fixed Chat Template (v21) — fixes KV cache optimization, thinking mode control, tool calling, and agent loop issues. Original backed up as chat_template.jinja.bak.

Quantization details

• Model type: qwen3_5_moe
• Bits: 2
• Group size: 64
• Format: MLX safetensors
• Calibration: oQ2 (standard sensitivity-based)

Environment

• Hardware: M5 MacBook Air 32GB
• Inference Framework: oMLX v0.5.4
• Max Concurrent Requests: 4
• Settings:
  • Thinking: Disabled
  • TurboQuant KV Cache: Enabled (4-bit)

Performance Benchmarks

Note: Results are for reference only and may vary depending on hardware, software configuration, and workload.

Single Request Results

Test	TTFT(ms)	TPOT(ms)	pp TPS	tg TPS	E2E(s)	Throughput	Peak Mem
pp1024/tg128	1097.4	20.22	933.1 tok/s	49.8 tok/s	3.678	313.2 tok/s	12.62 GB
pp4096/tg128	3777.7	21.17	1084.2 tok/s	47.6 tok/s	6.485	651.3 tok/s	13.34 GB

Continuous Batching (pp1024 / tg128)

Batch	tg TPS	Speedup	pp TPS	pp TPS/req	TTFT(ms)	E2E(s)
1x	49.8 tok/s	1.00x	933.1 tok/s	933.1 tok/s	1097.4	3.678
2x	67.5 tok/s	1.36x	833.0 tok/s	416.5 tok/s	2458.2	6.252
4x	98.0 tok/s	1.97x	825.7 tok/s	206.4 tok/s	4812.4	10.184

Intelligence Benchmark

Note: Each benchmark round tests only 30 questions. Results are for reference only.

Benchmark	Accuracy	Correct	Total	Time(s)	Think
MMLU	66.7%	20	30	36.2	No
TRUTHFULQA	83.3%	25	30	15.3	No
GSM8K	90.0%	27	30	78.1	No
MATHQA	46.7%	14	30	46.2	No
HUMANEVAL	83.3%	25	30	113.7	No