plain-language card: public-only example, sub-200M table

README.md

@@ -16,60 +16,77 @@ base_model: lightonai/LateOn
 
 # Tasksource-JEV-Nano-v0
 
-Decoupled multi-vector late-interaction decision model: **LateOn (~149M, ModernBERT backbone) fine-tuned on 512k natural heterogeneous Tasksource typed decisions** (choice + noul + score, natural mix, no distillation).
+A small decision model that picks the best option from a list — fraud routings, intents, topics, sentiments — using token-level late interaction instead of a classifier head.
 
-Factorization `sq_o = E(S+Q) ∥ E(O_i)` with PyLate native ColBERT MaxSim scoring. Context representations `E(S+Q)` are computed once and reused across decisions (state caching); candidate options are encoded independently (bit-exact permutation equivariance, verified max diff 0.0).
+It is LateOn (~149M parameters) fine-tuned on 512k real decisions covering three judgment types: picking one option (*choice*), yes/no questions (*noul*), and graded scores (*score*). No teacher model was used.
+
+**Why this architecture.** The situation (state + question) is encoded once into token vectors and reused for every candidate set, while each option is encoded independently. That gives two properties classifier heads don't have: caching the situation across decisions, and outputs that don't depend on option order (verified exactly permutation-equivariant).
 
 - **Base model:** [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn)
-- **Training data:** [`tasksource/tasksource-jev-typed-decisions`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) @ `071f0cf2201f06b2dea3e55323aa3e264284e139`
-- **Stream:** 512,000 decisions, manifest SHA-256 `997d241e34682f2b8b354103d1bbdc7e873fb45333e09637231f513176969331`, kind mix choice 82.7% / noul 11.4% / score 6.0%, 519 train tasks
-- **Recipe:** 8,000 steps × 64, LR 2e-5, kind-aware loss (choice: soft-target CE, noul: 2-way CE, score: CE + RPS), fail-closed `--canonical_run`
-- **Selection:** Tasksource unseen-dev NLL only → step 4000 of 8000 (best NLL 1.2073 internal / 1.0002 canonical probe). Learned temperature T=0.5347.
-- **Code:** training/eval harness in the `decision-models` repo (`scripts/train_multivector.py`, `scripts/eval_multivector.py`)
-
-## Benchmarks (canonical probes: deterministic 1k slices, seed 13 + full Fast Decisions)
-
-| Benchmark | Accuracy | NLL | Brier | ECE |
-|---|---|---|---|---|
-| Tasksource unseen-dev | 58.9% | 1.0002 | 0.4822 | 0.0435 |
-| Tasksource unseen-test | 53.0% | 0.9676 | 0.5367 | 0.0590 |
-| Tasksource seen-dev | 56.0% | 0.9573 | 0.5317 | 0.0586 |
-| Typed Decisions | 42.6% | 1.3144 | 0.2935 | 0.1294 |
-| AG News | 73.8% | 0.7510 | 0.4017 | 0.0840 |
-| Emotion | 49.0% | 1.4226 | 0.7033 | 0.1309 |
-| Banking77 (K=77) | 44.7% | 2.2661 | 0.7536 | 0.1857 |
-| Fast Decisions (2,900, 17 domains) | 41.31% EM (macro 40.59) | — | — | — |
-| classifier-benchmark v2 (866 cases) | micro 57.51 / macro 57.17 | — | — | — |
-
-Context: same-slice sub-400M peers score AG News 65–78%, Banking77 23–70%, Typed 34–49%, cb-v2 micro 35–60% — v0 is competitive while adding state caching and permutation equivariance, which classifier heads lack. Relative to its choice+KD sibling (50.8% unseen-dev), v0 gains +8.1pp in-domain and +6.9pp Typed at some AG/Banking77 transfer cost (documented specialization tradeoff).
-
-## Usage (minimal inference example)
+- **Training data:** [`tasksource/tasksource-jev-typed-decisions`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (pinned revision `071f0cf2`), 512k-decision stream, natural mix of the three judgment types, 519 tasks
+- **Checkpoint selection:** Tasksource unseen tasks only (step 4000 of 8000)
+- **Training code:** `decision-models` repo (`scripts/train_multivector.py`, canonical run)
+
+## How it scores
+
+| Benchmark (1k examples each, same samples for every model) | v0 |
+|---|---|
+| Tasksource, unseen tasks | 58.9% |
+| Tasksource, unseen test tasks | 53.0% |
+| Typed Decisions | 42.6% |
+| AG News | 73.8% |
+| Emotion | 49.0% |
+| Banking77 (77 options) | 44.7% |
+| Fast Decisions (2,900 cases, 17 domains) | 41.3% exact match |
+| classifier-benchmark v2 (866 cases) | 57.5 micro / 57.2 macro |
+
+## Compact peers (all under 200M, same 1k samples)
+
+| Model | Size | AG News | Emotion | Banking77 | Typed | cb-v2 micro |
+|---|---|---|---|---|---|---|
+| **Tasksource-JEV-Nano-v0** | 149M | 73.8 | 49.0 | 44.7 | 42.6 | 57.5 |
+| GLiClass Base v3 | 187M | 76.9 | 51.8 | 52.4 | 47.3 | 51.2 |
+| GLiClass Modern-Base v3 | 151M | 75.7 | 56.1 | 38.7 | 48.6 | 43.9 |
+| GLiNER2.5 Base | 194M | 76.2 | 56.6 | 69.7 | 43.9 | 60.0 |
+| GLiNER2.5 Small | 74M | 70.4 | 54.2 | 66.5 | 34.0 | 53.5 |
+| GLiClass Edge v3 | 33M | 65.5 | 50.1 | 23.2 | 40.0 | 35.3 |
+
+Larger peers (GLiClass Modern-Large 399M, GLiNER2.5-Multi 287M, Laya-Multilingual 322M) are omitted from this table; Laya's 90.0 AG News is training overlap (AG News is in its training mix). v0 trades some raw high-cardinality accuracy for state caching and order-independence, which none of the classifier heads offer.
+
+## Use it (only public packages)
+
+```bash
+pip install -U pylate torch
+```
 
 ```python
-from modernjev.multivector.lateon import LateOnDecisionScorer
+from pylate import models
+import torch
 
-scorer = LateOnDecisionScorer("tasksource/tasksource-jev-nano-v0", device="cuda")
+model = models.ColBERT("tasksource/tasksource-jev-nano-v0")
 
-state = "Customer reports unauthorized international wire transfer of $4,500 on account #4092."
+state = "Customer reports unauthorized international wire transfer of $4,500."
 question = "Select the appropriate fraud mitigation routing:"
 options = [
-    "approve_silent_monitor",
-    "two_factor_push_challenge",
-    "temporary_freeze_outreach",
-    "hard_decline_and_report",
+    "approve and monitor silently",
+    "challenge with a push notification",
+    "freeze the account and call the customer",
+    "decline and file a report",
 ]
 
-probs = scorer.predict_batch([state], [question], [options], kinds=["choice"])[0]
-for opt, p in zip(options, probs):
-    print(f"  {opt:30s} -> {p:.4f}")
+# Situation encoded once; each option scored against it (MaxSim)
+context = [state + "\nQuestion: " + question]
+ctx = model.encode(context, is_query=False, convert_to_tensor=True)[0]
+opts = model.encode(options, is_query=True, convert_to_tensor=True)
+scores = torch.stack([(o @ ctx.T).max(dim=1).values.sum() for o in opts])
 
-# State caching: encode once, score many candidate sets with zero re-encoding
-cache = scorer.encode_state([state], questions=[question])
-out = scorer.decide_cached(cache, questions=[question], options=[options], kinds=["choice"])
-print("Logits:", out.logits[0, :len(options)])
+temperature = 0.53  # learned temperature of this release
+probs = torch.softmax(scores / temperature, dim=0)
+for opt, p in zip(options, probs):
+    print(f"  {opt:45s} -> {p:.4f}")
 ```
 
-Heterogeneous kinds are supported per decision: `choice` (softmax), `noul` (2-way), `score` (softmax + ordinal), `multilabel` (independent sigmoids).
+Yes/no and graded decisions work the same way with two options or an ordered scale; encode the situation once and reuse it across as many candidate sets as you like.
 
 ## Citation
 
@@ -82,4 +99,4 @@ Heterogeneous kinds are supported per decision: `choice` (softmax), `noul` (2-wa
 }
 ```
 
-Base model: LateOn by LightOn (`lightonai/LateOn`, Apache 2.0). Training data: `tasksource/tasksource-jev-typed-decisions`.
+Base model LateOn by LightOn (Apache 2.0). If you use the training data, please also cite `tasksource/tasksource-jev-typed-decisions`.