--- license: apache-2.0 base_model: vllm-sr/d3-mini base_model_relation: quantized library_name: mlx tags: - mlx - quantized - ternary - zero-shot-classification --- # d3-mini mixed-bit MLX Unofficial MLX post-training quantization of [vllm-sr/d3-mini](https://huggingface.co/vllm-sr/d3-mini) (bf16, Apache-2.0). The source is a 32-layer Qwen3.5 text stack plus a vision tower and a 255-way decision readout. This package is not an official vLLM Semantic Router release. Suggested repository id: `Jakevin/d3-mini-ternary-mlx`. ## Method The original bfloat16 checkpoint is the reference. For each language-model projection (attention Q/K/V/O, DeltaNet `in_proj_qkv`, `in_proj_z`, `in_proj_a`, `in_proj_b`, `out_proj`, and MLP gate/up/down) one candidate replaces that projection alone and the KL of the decision readout's option distribution is measured. Candidates are ternary (one scale per group of 64, values `-s, 0, +s`), and MLX affine 2-bit, 3-bit and 4-bit with group size 64. Ternary is stored in MLX's built-in 2-bit affine tensors so `mlx-lm` / `mlx-vlm` quantized matmul can run it. No custom Metal kernel is included. Layer kinds are read from `text_config.layer_types`: 24 DeltaNet layers and 8 full-attention layers. Full attention is non-causal, as in the source checkpoint. DeltaNet stays causal. Bits start from the assignment that minimizes the sum of those one-at-a-time KL values with the average storage width of the language projections about 3 bits. Ternary counts as 2 storage bits because that is the container it uses; it is selected only when its KL is lower than affine 2-bit. That additive minimum sums to 1.5411. 4 later swaps each raise one projection by one bit and lower another by one bit. A swap is kept only when the joint KL of the fully quantized model falls on 20 calibration questions and the average stays inside the same budget. Joint KL on those questions went from 0.2733 to 0.2351. The shipped assignment's one-at-a-time sum is 1.5414. The decision readout is a separate matrix, `last_hidden @ readout.T`. It does not use the tied `lm_head`, so the token embedding is input-only and stays 8-bit affine (group 64). The decision readout, RMSNorms, DeltaNet convolution / `A_log` / `dt_bias`, and the entire vision tower stay at the original precision. Calibration: 64 questions from the author's `decision-v7` development split (seed 20261011, one question from each drawn record) plus the three text questions in the source model card. 67 questions total. hard-v1 was not used. ## Size | | bytes | |---|---:| | `model.safetensors` (quantized text + full-precision vision tower) | 2,928,233,188 | | `readout.safetensors` (fp32 decision head, unchanged) | 2,611,280 | | this folder | 2,953,926,604 | | source bf16 shards + readout | 9,081,227,360 | | tied embedding tensors, 8-bit (used) | 675,430,400 | Language-projection storage mix (ternary counted as 2 bits). Average **3.050** bits over 3,569,090,560 projection parameters. The tied embedding is 8-bit and is not in that average. The vision tower inside `model.safetensors` is uncompressed bf16. | assignment | parameters | share | storage bits | |---|---:|---:|---:| | ternary | 186,286,080 | 5.22% | 2 | | 2 | 1,048,821,760 | 29.39% | 2 | | 3 | 921,108,480 | 25.81% | 3 | | 4 | 1,412,874,240 | 39.59% | 4 | ## Evaluation Single run, no confidence interval. 150 questions: 50 holdout from `decision-v7` (seed 20261013, disjoint from calibration and the 8 smoke questions) plus 50 `decision-v7` and 50 `transfer-v9` (same seed, disjoint from calibration). Agreement is argmax vs the source bf16 model on the same MLX decision path. Accuracy is argmax vs the dataset label. Overall agreement **140/150** (93.3%). Mean KL(bf16 || quantized) **0.0261**, median **0.0077**. Compressed accuracy **117/150**, bf16 accuracy **119/150**. | suite | compressed correct | bf16 correct | agree | mean KL | median KL | |---|---:|---:|---:|---:|---:| | holdout | 40/50 | 41/50 | 47/50 | 0.0167 | 0.0068 | | decision-v7 | 41/50 | 42/50 | 47/50 | 0.0162 | 0.0063 | | transfer-v9 | 36/50 | 36/50 | 46/50 | 0.0455 | 0.0099 | Reload check: eight other questions scored from the in-memory quantized model and again after loading `out/release/` from disk. Agreement with memory **8/8**. **The vision tower was not quantized and was not part of this evaluation.** Official source-card scores, quoted from [vllm-sr/d3-mini](https://huggingface.co/vllm-sr/d3-mini) (Jev Decision Index 0.3.1, source evaluation, not reproduced here): Decision Index **54.2**, public suite **54.9**, same-skill tests **52.7**, new-domain tasks **48.6**. Vision Index **65.5**. Perception Test (internal) **70.3**. ## Use ```python from pathlib import Path from transformers import AutoTokenizer import decide import d3_mlx root = Path(".") model, config, decision, readout = d3_mlx.load_model(root) tokenizer = AutoTokenizer.from_pretrained(root) answers = decide.predict(model, readout, tokenizer, decision["codes"], state, questions) ``` From a shell, in this directory: ```bash python decide.py ``` That runs the three text questions from the source model card. `python decide.py request.json` reads `{"state": ..., "questions": ...}`. `mlx_lm` can load the language tower because `config.json` sets `model_file` to `d3_mlx.py`, which builds layers from `layer_types` and uses stock MLX quantized linears. The vision weights use mlx-vlm's `vision_tower.*` names and stay bf16. Text decisions use `decide.py`. ## License Apache-2.0. Source: [vllm-sr/d3-mini](https://huggingface.co/vllm-sr/d3-mini), itself built on [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B). See `LICENSE` and `NOTICE`. ## 繁體中文摘要 這是 [vllm-sr/d3-mini](https://huggingface.co/vllm-sr/d3-mini) 的非官方 MLX 量化包。 語言層的每個投影各自試過三值、2、3、4 bit,先用一次換一層的 KL 在平均約 3.05 bit 的預算內分配, 再做 4 次對調。三值借 MLX 內建 2-bit affine 儲存。決策頭是獨立 readout,不經過 tied 的輸出頭, 所以嵌入維持 8 bit。決策頭、norm 與視覺塔維持原精度,視覺部分未壓縮。 `model.safetensors` 為 2928.2 MB,整個資料夾 2953.9 MB。 150 題(單次、無信賴區間)與 bf16 一致 140/150, 壓縮版正確 117/150,平均 KL 0.0261。