Post
3,843
Published TR Quantisation versions of Qwen3.6-27B, Qwen3.6-35B-A3B and Qwen3.8-Flash-Next: weights packed into trits (2.25-3 per weight), no retraining.
- Mac: Qwen3.6-35B-A3B in a 16.5 GB GGUF that fits a 24 GB Mac, ~120 tok/s sustained on an M5 Max. Needs our llama.cpp build (ready-built for macOS and Linux arm64).
- vLLM on one DGX Spark: 35B at 180.9 tok/s for 1 user vs 128.0 for NVIDIA's NVFP4, 17.4 vs 21.1 GiB, no significant accuracy difference on the core benchmarks.
- Trade-offs on each card: slower prompt processing, lower multilingual maths, and 4-bit GGUF stays closer to the original text.
https://huggingface.co/collections/Ternii/tr-quantisation
- Mac: Qwen3.6-35B-A3B in a 16.5 GB GGUF that fits a 24 GB Mac, ~120 tok/s sustained on an M5 Max. Needs our llama.cpp build (ready-built for macOS and Linux arm64).
- vLLM on one DGX Spark: 35B at 180.9 tok/s for 1 user vs 128.0 for NVIDIA's NVFP4, 17.4 vs 21.1 GiB, no significant accuracy difference on the core benchmarks.
- Trade-offs on each card: slower prompt processing, lower multilingual maths, and 4-bit GGUF stays closer to the original text.
https://huggingface.co/collections/Ternii/tr-quantisation
- 2 replies