Feature Extraction
Transformers
Safetensors
modernbert
typed-decisions
decision-index
decision-model
cross-encoder
text-embeddings-inference
sileod commited on
Commit
6501eaa
·
verified ·
1 Parent(s): 5ee8d27

docs: consolidate v0 benchmark results

Browse files
Files changed (1) hide show
  1. README.md +6 -13
README.md CHANGED
@@ -23,24 +23,15 @@ It is LateOn (~149M parameters) fine-tuned on 512k real decisions covering three
23
  **Why this architecture.** The situation (state + question) is encoded once into token vectors and reused for every candidate set, while each option is encoded independently. That gives two properties classifier heads don't have: caching the situation across decisions, and outputs that don't depend on option order (verified exactly permutation-equivariant).
24
 
25
  - **Base model:** [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn)
26
- - **Training data:** [`tasksource/tasksource-jev-typed-decisions`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (pinned revision `071f0cf2`), 512k-decision stream, natural mix of the three judgment types, 519 tasks
27
  - **Checkpoint selection:** Tasksource unseen tasks only (step 4000 of 8000)
28
  - **Training code:** `decision-models` repo (`scripts/train_multivector.py`, canonical run)
29
 
30
- ## How it scores
31
 
32
- | Benchmark (1k examples each, same samples for every model) | v0 |
33
- |---|---|
34
- | Tasksource, unseen tasks (all three judgment types) | 46.8% |
35
- | Tasksource, unseen test tasks (all three judgment types) | 49.7% |
36
- | Typed Decisions | 42.6% |
37
- | AG News | 73.8% |
38
- | Emotion | 49.0% |
39
- | Banking77 (77 options) | 44.7% |
40
- | Fast Decisions (2,900 cases, 17 domains) | 41.3% exact match |
41
- | classifier-benchmark v2 (866 cases) | 57.5 micro / 57.2 macro |
42
 
43
- ## Compact peers (all under 200M, same 1k samples)
44
 
45
  | Model | Size | AG News | Emotion | Banking77 | Typed | cb-v2 micro |
46
  |---|---|---|---|---|---|---|
@@ -51,6 +42,8 @@ It is LateOn (~149M parameters) fine-tuned on 512k real decisions covering three
51
  | GLiNER2.5 Small | 74M | 70.4 | 54.2 | 66.5 | 34.0 | 53.5 |
52
  | GLiClass Edge v3 | 33M | 65.5 | 50.1 | 23.2 | 40.0 | 35.3 |
53
 
 
 
54
  Larger peers (GLiClass Modern-Large 399M, GLiNER2.5-Multi 287M, Laya-Multilingual 322M) are omitted from this table; Laya's 90.0 AG News is training overlap (AG News is in its training mix). v0 trades some raw high-cardinality accuracy for state caching and order-independence, which none of the classifier heads offer.
55
 
56
  ## Use it (only public packages)
 
23
  **Why this architecture.** The situation (state + question) is encoded once into token vectors and reused for every candidate set, while each option is encoded independently. That gives two properties classifier heads don't have: caching the situation across decisions, and outputs that don't depend on option order (verified exactly permutation-equivariant).
24
 
25
  - **Base model:** [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn)
26
+ - **Training data:** [`tasksource/tasksource-jev-typed-decisions`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (pinned revision [`071f0cf2201f06b2dea3e55323aa3e264284e139`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions/tree/071f0cf2201f06b2dea3e55323aa3e264284e139)), 512k-decision stream, natural mix of the three judgment types, 519 tasks
27
  - **Checkpoint selection:** Tasksource unseen tasks only (step 4000 of 8000)
28
  - **Training code:** `decision-models` repo (`scripts/train_multivector.py`, canonical run)
29
 
30
+ ## Evaluation
31
 
32
+ On Tasksource's held-out tasks across all three judgment types, v0 scores **46.8%** on unseen-dev and **49.7%** on unseen-test (1k examples each). It reaches **41.3% exact match** on Fast Decisions (2,900 cases across 17 domains). The table compares models on the same 1k examples per public dataset; classifier-benchmark v2 has 866 cases.
 
 
 
 
 
 
 
 
 
33
 
34
+ ### Compact peers (all under 200M)
35
 
36
  | Model | Size | AG News | Emotion | Banking77 | Typed | cb-v2 micro |
37
  |---|---|---|---|---|---|---|
 
42
  | GLiNER2.5 Small | 74M | 70.4 | 54.2 | 66.5 | 34.0 | 53.5 |
43
  | GLiClass Edge v3 | 33M | 65.5 | 50.1 | 23.2 | 40.0 | 35.3 |
44
 
45
+ v0's classifier-benchmark v2 macro accuracy is **57.2%**.
46
+
47
  Larger peers (GLiClass Modern-Large 399M, GLiNER2.5-Multi 287M, Laya-Multilingual 322M) are omitted from this table; Laya's 90.0 AG News is training overlap (AG News is in its training mix). v0 trades some raw high-cardinality accuracy for state caching and order-independence, which none of the classifier heads offer.
48
 
49
  ## Use it (only public packages)