| --- |
| license: apache-2.0 |
| base_model: jaredpalmer/kev-0.8b |
| tags: |
| - decision-model |
| - mlx |
| - abstractcore |
| - kev |
| library_name: mlx |
| pipeline_tag: text-classification |
| --- |
| |
| # Kev-0.8B for MLX, 8-bit (group 64) |
|
|
| **What it is.** A decision model: it reads a document (the *state*) and answers typed questions about it |
| — `choice` over named options, `noul` (the probability that a statement is true), `score` over ordered |
| levels — with calibrated probabilities, in one forward pass, without generating text. Use it to classify, |
| route, triage, score or check documents and threshold the probabilities (send the uncertain cases to a human). |
|
|
| **Why this checkpoint.** The published [jaredpalmer/kev-0.8b](https://huggingface.co/jaredpalmer/kev-0.8b) (Jared Palmer, |
| Apache-2.0) is a rank-16 LoRA plus a pointer head on a Qwen3.5 base: running it means downloading the |
| base model, folding the adapter in at every load and keeping bf16 weights in memory. This repository is the |
| merged result quantized for [mlx-lm](https://github.com/ml-explore/mlx-lm) at 8-bit (group 64) (0.8 GB): |
| one download, loads in a few seconds, same answers within the measured tolerance below. The pointer head |
| (`head.safetensors`) stays in float32; `decision.json` records the recipe and the source checkpoint. |
|
|
| **How accuracy is protected.** A decision model's value is its calibration, and a plain 4-bit |
| quantization damages it: every linear layer at 4 bits (group 64) costs Kev-4B a point of accuracy and |
| moves a tenth of its answers by more than 0.10 probability. The recipes here were chosen by measuring |
| every candidate against the bf16 weights on the model author's public development set (1,328 records, |
| 1,592 questions: argmax agreement, mean and maximum probability shift, accuracy, Brier, ECE), per module |
| group and group size. Quantization sensitivity sits in the Gated DeltaNet projections and the group size; |
| keeping those projections at 8 bits with 32-element groups keeps the accuracy, and the 8-bit checkpoint is |
| indistinguishable from bf16. The companion checkpoints are `AbstractFramework/kev-0.8b-mlx-8bit` and |
| `AbstractFramework/kev-0.8b-mlx-4bit`; prefer 8-bit when you threshold probabilities, 4-bit when size matters. |
|
|
| ## Measured |
|
|
| public-pool-v6 development set (1,328 records): accuracy 0.761 (bf16 0.763), Brier 0.321, ECE 0.025; argmax agreement with bf16 0.990, mean |Δp| 0.006; 24 ms per record on an M5 Max. |
|
|
| ## Use with AbstractCore |
|
|
| ```bash |
| pip install "abstractcore[apple]" |
| abstractcore models download mlx AbstractFramework/kev-0.8b-mlx-8bit |
| ``` |
|
|
| ```python |
| from abstractcore import create_llm |
| |
| llm = create_llm("mlx", model="AbstractFramework/kev-0.8b-mlx-8bit") |
| result = llm.generate( |
| "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.", |
| output="decision", |
| questions={ |
| "department": {"type": "choice", "instructions": "Which team should handle this?", |
| "criteria": {"returns": "Exchanges, refunds, wrong or damaged items", |
| "shipping": "Delivery status, delays, lost packages", |
| "billing": "Charges, invoices, payment problems"}}, |
| "escalate": "Does this need urgent human attention?", # shorthand for a noul question |
| "frustration": {"type": "score", "instructions": "How frustrated is the customer?", |
| "criteria": ["Calm", "Frustrated", "Very angry"]}, |
| }, |
| ) |
| result.decisions["department"].choice # "returns" |
| result.decisions["department"].probabilities # {"returns": 0.50, "shipping": 0.37, "billing": 0.13} |
| result.decisions["escalate"].noul # 0.73 |
| result.decisions["frustration"].score # expected level, 0..2 |
| ``` |
|
|
| Over HTTP (`abstractcore serve`), the same request is `POST /v1/decisions` with |
| `{"model": "mlx/AbstractFramework/kev-0.8b-mlx-8bit", "state": ..., "questions": ...}`. |
|
|
| ## The state cache (KV cache) and what to expect |
|
|
| The document is read once per request and reused for every question in it; AbstractCore also keeps |
| the computed state of the last four documents, so a second request about the same document only pays |
| for its questions. Measured on an M5 Max with this family's 8-bit Kev-4B: |
|
|
| | Request | First time | Same document again | |
| |---|---|---| |
| | a 100-token support ticket, three questions | ~95 ms | ~45 ms | |
| | a 6,300-token contract, three questions | ~1.4 s | ~0.1 s | |
|
|
| Reading the document is the cost (about 3,000 tokens per second for Kev-4B, four to five times faster |
| for Kev-0.8B); the questions themselves cost tens of milliseconds. Ask everything you need about a |
| document in one call, or in consecutive calls while it is still cached. To disable the cache (every |
| request re-reads its document): `create_llm("mlx", model=..., decision_prefix_cache_size=0)`; to keep |
| more documents: `decision_prefix_cache_size=16`. The cache lives in memory for the life of the provider. |
|
|
| ## Docs |
|
|
| - Decision models in AbstractCore: https://github.com/lpalbou/AbstractCore/blob/main/docs/decision-models.md |
| (engines, every supported model, conversion recipes, the full measurements) |
| - AbstractFramework: https://github.com/lpalbou/abstractframework |
|
|
| ## Attribution |
|
|
| Weights derived from jaredpalmer/kev-0.8b (Jared Palmer) on Qwen/Qwen3.5 base models (Alibaba), both Apache-2.0. |
| Converted and measured with `python -m abstractcore.decision.convert`. |
|
|
