philipjohnbasile/GLM-5.2-colibri-int4-with-int8-mtp

GLM-5.2 — int4 container for iliria (full 744B, SSD-streamed)

The full GLM-5.2 (744B total / 39B active) quantized to int4 with an int8 MTP head, packaged as the container that iliria serves directly.

iliria runs this model on a single 128 GB Apple Silicon machine by streaming experts from the SSD rather than holding all 744B parameters in memory. This repo exists so you don't have to run that conversion yourself — otherwise the on-ramp is downloading ~372 GB of FP8 weights and converting them locally for hours.

This is not the pruned build

If you came from GLM-5.2-Demolition-q4a4-soul-MLX, note these are different artifacts and not interchangeable:

	this repo	the demolition build
model	full 744B, unpruned	expert-pruned + LoRA-healed
size	~384 GB (int4)	~98 GB
format	iliria container (out-*.safetensors)	MLX
runs on	iliria	MLX / mlx-lm

Requirements

• Apple Silicon M-Max / M-Ultra class with 128 GB unified memory
• ~384 GB free SSD (the weights are streamed from disk, not loaded into RAM)
• iliria

Use

hf download philipjohnbasile/GLM-5.2-colibri-int4-with-int8-mtp --local-dir ./glm52-int4
# then, from iliria:
./illi serve --model ./glm52-int4 --ram 60

That exposes an OpenAI-compatible endpoint. See the iliria README for the engine's flags, context/RAM tuning, and measured throughput — expect deep-tier speeds, not chat speeds: this is a 744B model reading experts off an SSD for every token.

Contents

164 out-*.safetensors shards (int4 experts + int8 MTP head) plus config.json, generation_config.json, and the tokenizer. Nothing else — no runtime state, no caches.

Provenance

Quantized from zai-org/GLM-5.2-FP8 (MIT). Quantization only — no pruning, no retraining, no distillation; the full expert set is intact. Produced by iliria's tools/convert_fp8_to_int4.py. Released under MIT, matching the base model.