Instructions to use Jakevin/d3-mini-ternary-mlx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use Jakevin/d3-mini-ternary-mlx with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download Jakevin/d3-mini-ternary-mlx --local-dir d3-mini-ternary-mlx
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
|
Download README.md from Jakevin/d3-mini-ternary-mlx: direct link, hf CLI and curl.
- Browser
- Download file 6.34 kB
-
https://huggingface.co/Jakevin/d3-mini-ternary-mlx/resolve/main/README.md
- Command line
-
hf download hf://Jakevin/d3-mini-ternary-mlx/README.md
-
curl -L -o README.md https://huggingface.co/Jakevin/d3-mini-ternary-mlx/resolve/main/README.md
6.34 kB
| license: apache-2.0 | |
| base_model: vllm-sr/d3-mini | |
| base_model_relation: quantized | |
| library_name: mlx | |
| tags: | |
| - mlx | |
| - quantized | |
| - ternary | |
| - zero-shot-classification | |
| # d3-mini mixed-bit MLX | |
| Unofficial MLX post-training quantization of [vllm-sr/d3-mini](https://huggingface.co/vllm-sr/d3-mini) | |
| (bf16, Apache-2.0). The source is a 32-layer Qwen3.5 text stack plus a vision tower and a | |
| 255-way decision readout. This package is not an official vLLM Semantic Router release. | |
| Suggested repository id: `Jakevin/d3-mini-ternary-mlx`. | |
| ## Method | |
| The original bfloat16 checkpoint is the reference. For each language-model projection | |
| (attention Q/K/V/O, DeltaNet `in_proj_qkv`, `in_proj_z`, `in_proj_a`, `in_proj_b`, `out_proj`, | |
| and MLP gate/up/down) one candidate replaces that projection alone and the KL of the decision | |
| readout's option distribution is measured. Candidates are ternary (one scale per group of 64, | |
| values `-s, 0, +s`), and MLX affine 2-bit, 3-bit and 4-bit with group size 64. Ternary is stored | |
| in MLX's built-in 2-bit affine tensors so `mlx-lm` / `mlx-vlm` quantized matmul can run it. | |
| No custom Metal kernel is included. | |
| Layer kinds are read from `text_config.layer_types`: 24 DeltaNet layers and 8 | |
| full-attention layers. Full attention is non-causal, as in the source checkpoint. DeltaNet stays causal. | |
| Bits start from the assignment that minimizes the sum of those one-at-a-time KL values with the | |
| average storage width of the language projections about 3 bits. Ternary counts as 2 storage bits | |
| because that is the container it uses; it is selected only when its KL is lower than affine 2-bit. | |
| That additive minimum sums to 1.5411. | |
| 4 later swaps each raise one projection by one bit and lower another by one bit. | |
| A swap is kept only when the joint KL of the fully quantized model falls on 20 | |
| calibration questions and the average stays inside the same budget. Joint KL on those questions went from | |
| 0.2733 to 0.2351. | |
| The shipped assignment's one-at-a-time sum is 1.5414. | |
| The decision readout is a separate matrix, `last_hidden @ readout.T`. It does not use the tied | |
| `lm_head`, so the token embedding is input-only and stays 8-bit affine (group 64). | |
| The decision readout, RMSNorms, DeltaNet convolution / `A_log` / `dt_bias`, and the entire vision | |
| tower stay at the original precision. | |
| Calibration: 64 questions from the author's `decision-v7` development split | |
| (seed 20261011, one question from each drawn record) plus the three text questions in the | |
| source model card. 67 questions total. hard-v1 was not used. | |
| ## Size | |
| | | bytes | | |
| |---|---:| | |
| | `model.safetensors` (quantized text + full-precision vision tower) | 2,928,233,188 | | |
| | `readout.safetensors` (fp32 decision head, unchanged) | 2,611,280 | | |
| | this folder | 2,953,926,604 | | |
| | source bf16 shards + readout | 9,081,227,360 | | |
| | tied embedding tensors, 8-bit (used) | 675,430,400 | | |
| Language-projection storage mix (ternary counted as 2 bits). Average **3.050** bits | |
| over 3,569,090,560 projection parameters. The tied embedding is 8-bit and is not in that average. | |
| The vision tower inside `model.safetensors` is uncompressed bf16. | |
| | assignment | parameters | share | storage bits | | |
| |---|---:|---:|---:| | |
| | ternary | 186,286,080 | 5.22% | 2 | | |
| | 2 | 1,048,821,760 | 29.39% | 2 | | |
| | 3 | 921,108,480 | 25.81% | 3 | | |
| | 4 | 1,412,874,240 | 39.59% | 4 | | |
| ## Evaluation | |
| Single run, no confidence interval. 150 questions: 50 holdout from `decision-v7` (seed 20261013, | |
| disjoint from calibration and the 8 smoke questions) plus 50 `decision-v7` and 50 `transfer-v9` | |
| (same seed, disjoint from calibration). Agreement is argmax vs the source bf16 model on the same | |
| MLX decision path. Accuracy is argmax vs the dataset label. | |
| Overall agreement **140/150** (93.3%). | |
| Mean KL(bf16 || quantized) **0.0261**, median **0.0077**. | |
| Compressed accuracy **117/150**, bf16 accuracy | |
| **119/150**. | |
| | suite | compressed correct | bf16 correct | agree | mean KL | median KL | | |
| |---|---:|---:|---:|---:|---:| | |
| | holdout | 40/50 | 41/50 | 47/50 | 0.0167 | 0.0068 | | |
| | decision-v7 | 41/50 | 42/50 | 47/50 | 0.0162 | 0.0063 | | |
| | transfer-v9 | 36/50 | 36/50 | 46/50 | 0.0455 | 0.0099 | | |
| Reload check: eight other questions scored from the in-memory quantized model and again after | |
| loading `out/release/` from disk. Agreement with memory **8/8**. | |
| **The vision tower was not quantized and was not part of this evaluation.** | |
| Official source-card scores, quoted from [vllm-sr/d3-mini](https://huggingface.co/vllm-sr/d3-mini) | |
| (Jev Decision Index 0.3.1, source evaluation, not reproduced here): Decision Index **54.2**, | |
| public suite **54.9**, same-skill tests **52.7**, new-domain tasks **48.6**. Vision Index **65.5**. | |
| Perception Test (internal) **70.3**. | |
| ## Use | |
| ```python | |
| from pathlib import Path | |
| from transformers import AutoTokenizer | |
| import decide | |
| import d3_mlx | |
| root = Path(".") | |
| model, config, decision, readout = d3_mlx.load_model(root) | |
| tokenizer = AutoTokenizer.from_pretrained(root) | |
| answers = decide.predict(model, readout, tokenizer, decision["codes"], state, questions) | |
| ``` | |
| From a shell, in this directory: | |
| ```bash | |
| python decide.py | |
| ``` | |
| That runs the three text questions from the source model card. `python decide.py request.json` | |
| reads `{"state": ..., "questions": ...}`. | |
| `mlx_lm` can load the language tower because `config.json` sets `model_file` to `d3_mlx.py`, | |
| which builds layers from `layer_types` and uses stock MLX quantized linears. The vision weights | |
| use mlx-vlm's `vision_tower.*` names and stay bf16. Text decisions use `decide.py`. | |
| ## License | |
| Apache-2.0. Source: [vllm-sr/d3-mini](https://huggingface.co/vllm-sr/d3-mini), | |
| itself built on [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B). | |
| See `LICENSE` and `NOTICE`. | |
| ## 繁體中文摘要 | |
| 這是 [vllm-sr/d3-mini](https://huggingface.co/vllm-sr/d3-mini) 的非官方 MLX 量化包。 | |
| 語言層的每個投影各自試過三值、2、3、4 bit,先用一次換一層的 KL 在平均約 3.05 bit 的預算內分配, | |
| 再做 4 次對調。三值借 MLX 內建 2-bit affine 儲存。決策頭是獨立 readout,不經過 tied 的輸出頭, | |
| 所以嵌入維持 8 bit。決策頭、norm 與視覺塔維持原精度,視覺部分未壓縮。 | |
| `model.safetensors` 為 2928.2 MB,整個資料夾 2953.9 MB。 | |
| 150 題(單次、無信賴區間)與 bf16 一致 140/150, | |
| 壓縮版正確 117/150,平均 KL 0.0261。 | |