kev-4b-mlx-4bit: model card

README.md

@@ -12,21 +12,39 @@ pipeline_tag: text-classification
 
 # Kev-4B for MLX, 4-bit, group 32, with the Gated DeltaNet projections (in_proj_qkv / in_proj_z / in_proj_a / in_proj_b / out_proj) kept at 8-bit
 
-A decision model: it reads a document (the *state*) and answers typed questions about it — `choice`
-over named options, `noul` (the probability a statement is true), `score` over ordered levels — with
-calibrated probabilities, in one forward pass, without generating text.
+**What it is.** A decision model: it reads a document (the *state*) and answers typed questions about it
+— `choice` over named options, `noul` (the probability that a statement is true), `score` over ordered
+levels — with calibrated probabilities, in one forward pass, without generating text. Use it to classify,
+route, triage, score or check documents and threshold the probabilities (send the uncertain cases to a human).
 
-This is [jaredpalmer/kev-4b](https://huggingface.co/jaredpalmer/kev-4b) (Jared Palmer, Apache-2.0: a rank-16 LoRA plus a
-pointer head on a Qwen3.5 base) with the adapter folded into the base and the backbone quantized for
-[mlx-lm](https://github.com/ml-explore/mlx-lm) at 4-bit, group 32, with the Gated DeltaNet projections (in_proj_qkv / in_proj_z / in_proj_a / in_proj_b / out_proj) kept at 8-bit (3.0 GB). The pointer head
+**Why this checkpoint.** The published [jaredpalmer/kev-4b](https://huggingface.co/jaredpalmer/kev-4b) (Jared Palmer,
+Apache-2.0) is a rank-16 LoRA plus a pointer head on a Qwen3.5 base: running it means downloading the
+base model, folding the adapter in at every load and keeping bf16 weights in memory. This repository is the
+merged result quantized for [mlx-lm](https://github.com/ml-explore/mlx-lm) at 4-bit, group 32, with the Gated DeltaNet projections (in_proj_qkv / in_proj_z / in_proj_a / in_proj_b / out_proj) kept at 8-bit (3.0 GB):
+one download, loads in a few seconds, same answers within the measured tolerance below. The pointer head
 (`head.safetensors`) stays in float32; `decision.json` records the recipe and the source checkpoint.
 
+**How accuracy is protected.** A decision model's value is its calibration, and a plain 4-bit
+quantization damages it: every linear layer at 4 bits (group 64) costs Kev-4B a point of accuracy and
+moves a tenth of its answers by more than 0.10 probability. The recipes here were chosen by measuring
+every candidate against the bf16 weights on the model author's public development set (1,328 records,
+1,592 questions: argmax agreement, mean and maximum probability shift, accuracy, Brier, ECE), per module
+group and group size. Quantization sensitivity sits in the Gated DeltaNet projections and the group size;
+keeping those projections at 8 bits with 32-element groups keeps the accuracy, and the 8-bit checkpoint is
+indistinguishable from bf16. The companion checkpoints are `AbstractFramework/kev-4b-mlx-8bit` and
+`AbstractFramework/kev-4b-mlx-4bit`; prefer 8-bit when you threshold probabilities, 4-bit when size matters.
+
 ## Measured
 
 public-pool-v6 development set (1,328 records): accuracy 0.847 (bf16 0.846), Brier 0.226, ECE 0.024; argmax agreement with bf16 0.973, mean |Δp| 0.023, 3.1 % of questions moved by more than 0.10; 113 ms per record on an M5 Max. A plain 4-bit conversion (every linear at 4 bits, group 64) loses a point of accuracy and moves 10.6 % of the answers; this recipe is the measured compromise. Use the 8-bit checkpoint when you threshold probabilities.
 
 ## Use with AbstractCore
 
+```bash
+pip install "abstractcore[apple]"
+abstractcore models download mlx AbstractFramework/kev-4b-mlx-4bit
+```
+
 ```python
 from abstractcore import create_llm
 
@@ -39,18 +57,44 @@ result = llm.generate(
                        "criteria": {"returns": "Exchanges, refunds, wrong or damaged items",
                                     "shipping": "Delivery status, delays, lost packages",
                                     "billing": "Charges, invoices, payment problems"}},
-        "escalate": {"type": "noul", "instructions": "Does this need urgent human attention?"},
+        "escalate": "Does this need urgent human attention?",   # shorthand for a noul question
         "frustration": {"type": "score", "instructions": "How frustrated is the customer?",
                         "criteria": ["Calm", "Frustrated", "Very angry"]},
     },
 )
-result.decisions["department"].choice, result.decisions["escalate"].noul
+result.decisions["department"].choice          # "returns"
+result.decisions["department"].probabilities   # {"returns": 0.50, "shipping": 0.37, "billing": 0.13}
+result.decisions["escalate"].noul              # 0.73
+result.decisions["frustration"].score          # expected level, 0..2
 ```
 
-Or over HTTP on an AbstractCore server: `POST /v1/decisions` with `{"model": "mlx/AbstractFramework/kev-4b-mlx-4bit", "state": ..., "questions": ...}`.
-Docs: https://github.com/lpalbou/AbstractCore/blob/main/docs/decision-models.md
+Over HTTP (`abstractcore serve`), the same request is `POST /v1/decisions` with
+`{"model": "mlx/AbstractFramework/kev-4b-mlx-4bit", "state": ..., "questions": ...}`.
+
+## The state cache (KV cache) and what to expect
+
+The document is read once per request and reused for every question in it; AbstractCore also keeps
+the computed state of the last four documents, so a second request about the same document only pays
+for its questions. Measured on an M5 Max with this family's 8-bit Kev-4B:
+
+| Request | First time | Same document again |
+|---|---|---|
+| a 100-token support ticket, three questions | ~95 ms | ~45 ms |
+| a 6,300-token contract, three questions | ~1.4 s | ~0.1 s |
+
+Reading the document is the cost (about 3,000 tokens per second for Kev-4B, four to five times faster
+for Kev-0.8B); the questions themselves cost tens of milliseconds. Ask everything you need about a
+document in one call, or in consecutive calls while it is still cached. To disable the cache (every
+request re-reads its document): `create_llm("mlx", model=..., decision_prefix_cache_size=0)`; to keep
+more documents: `decision_prefix_cache_size=16`. The cache lives in memory for the life of the provider.
+
+## Docs
+
+- Decision models in AbstractCore: https://github.com/lpalbou/AbstractCore/blob/main/docs/decision-models.md
+  (engines, every supported model, conversion recipes, the full measurements)
+- AbstractFramework: https://github.com/lpalbou/abstractframework
 
 ## Attribution
 
 Weights derived from jaredpalmer/kev-4b (Jared Palmer) on Qwen/Qwen3.5 base models (Alibaba), both Apache-2.0.
-Converted with `python -m abstractcore.decision.convert`.
+Converted and measured with `python -m abstractcore.decision.convert`.