model card

README.md

@@ -0,0 +1,153 @@
+---
+license: mit
+library_name: transformers
+pipeline_tag: text-classification
+base_model: Qwen/Qwen3-4B-Base
+tags: [tinyjev, jev, decision-model, system-one, typed-decisions, mlx, qwen3]
+language: [en]
+datasets: [jaredpalmer/kev-suites]
+---
+
+<div align="center">
+
+<img alt="TinyJev" src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/tinyjev_header.png" width="620">
+
+<p>Typed decisions, on your laptop, in one forward pass. The bigger one.</p>
+
+<p>
+  <a href="https://pypi.org/project/tinyjev/"><img alt="PyPI" src="https://img.shields.io/pypi/v/tinyjev?label=pypi&color=E46412"></a>
+  <a href="https://pypi.org/project/tinyjev/"><img alt="Python" src="https://img.shields.io/badge/python-3.9%2B-E46412"></a>
+  <a href="https://github.com/ankit-aglawe/tinyjev"><img alt="GitHub" src="https://img.shields.io/badge/code-github-E46412?logo=github&logoColor=white"></a>
+  <a href="https://github.com/ankit-aglawe/tinyjev/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/license-MIT-E46412"></a>
+</p>
+
+<p>
+  <a href="https://github.com/ankit-aglawe/tinyjev">GitHub</a> ·
+  <a href="https://pypi.org/project/tinyjev/">PyPI</a> ·
+  <a href="https://huggingface.co/AnkitAI/tinyjev-0.6b">TinyJev 0.6B</a> ·
+  <a href="https://github.com/ankit-aglawe/tinyjev/tree/main/examples">Examples</a>
+</p>
+
+</div>
+
+Send this model some state, a ticket or a record or a log line, plus questions with the answers you
+will accept. It returns a probability for every option you offered. It cannot answer with anything
+else, because it never generates text; it scores the options you gave it and stops.
+
+TinyJev 4B is the same head and the same training data as TinyJev 0.6B on a Qwen3-4B-Base backbone.
+It is the accurate one: 474 of 500 never-seen decisions against 440, and at confidence ≥ 0.85 it
+answers 87% of the queue on its own at 99.1%. It costs 8 GB (4.5 GB at 8 bits) and about 630 ms per
+decision on a base M1, seven times the 0.6B's.
+
+- `Choice` picks one option from a list, with a probability for each.
+- `Noul` measures whether a statement is true.
+- `Score` places state on an ordered scale.
+- Calibrated confidence (ECE 0.022), so a threshold means something and you can decide what to automate.
+- A Python API, a local HTTP server, and a System One compatible endpoint.
+
+## Models
+
+Two models so far, same head, same training data, scored on the same 500 never-seen cases from 25
+domains ([`benchmarks/opendecision`](https://github.com/ankit-aglawe/tinyjev/tree/main/benchmarks/opendecision), every case and probability logged).
+Latency is a base M1 (16 GB) via MLX, one forward pass per case.
+
+| Model | Params | OD-500 | Gate 0.85 | ms / case | Weights |
+|---|---:|---:|---|---:|---|
+| <img src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/logos/tinyjev.png" width="18"> **TinyJev&nbsp;0.6B** | 596M, 1.2 GB | 440 (88.0%) | 59% @ 98.0% | 85 | 🤗 [AnkitAI/tinyjev-0.6b](https://huggingface.co/AnkitAI/tinyjev-0.6b) |
+| <img src="https://raw.githubusercontent.com/ankit-aglawe/tinyjev/main/assets/logos/tinyjev.png" width="18"> **TinyJev&nbsp;4B** | 4.0B, 8.0 GB | 474 (94.8%) | 87% @ 99.1% | 628 | 🤗 [AnkitAI/tinyjev-4b](https://huggingface.co/AnkitAI/tinyjev-4b) |
+
+OD-500 is correct answers out of 500. Gate 0.85 is the share of decisions answered on its own at
+confidence ≥ 0.85, and how often those were right. Calibration (ECE 0.071 vs 0.022), coverage at 2%
+error (63% vs 92%) and transfer-v4 dev (0.625 vs 0.762) are on the benchmark page. Load either with
+`tinyjev.load("tinyjev-0.6b")` or `tinyjev.load("tinyjev-4b")`.
+
+Both rows are fp16. Loading with `quantize=8` keeps the same weights in half the memory and changes
+almost nothing: the 0.6B scores 440 at 90 ms, the 4B 473 at 845 ms, one answer in 500 different from
+fp16. The gate is the number that matters in production; the rest of the queue goes to a person or a
+bigger model. On the same 500: Kev-0.8B 463, Claude Opus 5.5 496, the same Qwen3-0.6B weights read
+through letter logits with no head 354.
+
+## Measured
+
+On OpenDecision's Original Choice 500, a suite of 25 domains that was not in the training data, with the same 500 inputs for every model:
+
+| Model | Correct / 500 | Handled alone at confidence ≥ 0.85 |
+|---|---:|---:|
+| Claude Opus 5.5 (cloud, self-reported probabilities) | 496 | 477 at 100.0% |
+| **tinyjev-4b** (fp16) | **474** | **437 at 99.1%** |
+| tinyjev-4b at `quantize=8` | 473 | 437 at 99.1% |
+| Kev-0.8B (raw logits) | 463 | 186 at 100.0% |
+| tinyjev-0.6b | 440 | 296 at 98.0% |
+
+353/375 on dev, 121/125 on holdout, 95% CI 0.928–0.966, ECE 0.022, Brier 0.071, coverage at 2% error 92%.
+On Kev's transfer-v4 dev it scores 0.762 against 0.625 for the 0.6B (Kev-4B, a full fine-tune, 0.790).
+Every case, every probability, the coverage curves and all the baseline rows:
+[benchmarks/opendecision](https://github.com/ankit-aglawe/tinyjev/tree/main/benchmarks/opendecision).
+
+Known weakness, measured: statement-form yes/no checks. On a 21-item check list about one support
+email it scores 18 (the 0.6B: 14, it says yes to everything); on hard negatives, statements about a
+topic the text mentions but does not support, 81%. Training data for that is the next experiment.
+
+## Use it
+
+```bash
+pip install 'tinyjev[mlx]'     # Apple Silicon
+pip install 'tinyjev[torch]'   # everything else
+```
+
+```python
+import tinyjev
+agent = tinyjev.load("tinyjev-4b", quantize=8)   # 4.5 GB in memory; drop quantize for fp16, 8 GB
+
+agent.predict({
+    "state": "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.",
+    "questions": {
+        "team":     {"type": "choice", "instructions": "Which team should handle this?",
+                     "criteria": {"returns": "Exchanges, refunds, wrong or damaged items",
+                                  "shipping": "Delivery status, delays, lost packages",
+                                  "billing":  "Charges, invoices, payment problems"}},
+        "escalate": {"type": "noul",   "instructions": "Does this need urgent human attention?"},
+        "anger":    {"type": "score",  "instructions": "How angry is the customer?",
+                     "criteria": ["calm", "frustrated", "very angry"]},
+    }})
+```
+
+Eight bits changed one answer in 500 on the held-out suite. Serve it over HTTP, speaking the System
+One request shape:
+
+```bash
+tinyjev serve tinyjev-4b --quantize 8     # POST /v1/systemone on 127.0.0.1:8077
+```
+
+## What is in this repo
+
+`AutoModel.from_pretrained("AnkitAI/tinyjev-4b")` loads the backbone on its own, a standard
+`Qwen3Model` in fp16 with the LoRA already merged. The decision head lives in `head.safetensors`, and
+`tinyjev` is what turns hidden states into calibrated answers.
+
+## How it was built, and how it scores
+
+Qwen3-4B-Base with a pointer head, LoRA r16 at lr 5e-5, seed 2, two epochs on the public
+`jaredpalmer/kev-suites` decision-v7 training split (12,576 records), Kev's study runner on one H100
+for 45 minutes, the adapter merged back into the base. No held-out transfer source was used in
+training. Temperature 1.0 at inference; the calibration figures above are at raw logits.
+
+| | transfer-v4 dev | decision-v7 dev | OpenDecision 500 |
+| --- | --- | --- | --- |
+| tinyjev-4b | 0.762 | 0.859 | 474 / 500 |
+| tinyjev-0.6b | 0.625 | | 440 / 500 |
+
+## Support the Project
+
+If this model is useful in your work, you can support independent research:
+
+<p align="left">
+  <a href="https://www.buymeacoffee.com/AnkitAI" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me a Coffee" height="60" width="217" /></a>
+</p>
+
+## Credits
+
+Built on [Qwen3-4B-Base](https://huggingface.co/Qwen/Qwen3-4B-Base) (Apache-2.0). The training data,
+evaluation suites and the pointer-head design come from [Kev](https://github.com/jaredpalmer/kev) by
+Jared Palmer (Apache-2.0). The typed-decision interface follows
+[TypeSafe's Jev](https://docs.typesafe.ai/introduction). MIT licensed.