Mirai's Qwen3.8-27B Small Quantization

A small, high-quality quantization of Qwen3.8-27B prepared for efficient local inference on Apple silicon.

Quickstart

If you are on macOS, the easiest way is to install the mirai Homebrew package and then run the CLI:

brew install mirai
mirai --model trymirai/Qwen3.8-27B-S

Currently only Apple silicon inference is supported. If you want to build things from source, read this overview.

Method

Mirai Small uses QTIP-style trellis-coded quantization with the bits of each matrix chosen from measured KL. Kronecker-structured random rotations are used to reduce weight outliers. The input embedding uses a 2-bit D4 lattice code, and the output head uses 4-bit integers with group size 64 and block-32 Hadamard transforms. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation, including a round on the model's own agent turns.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}