Feature Extraction
Transformers
Safetensors
modernbert
typed-decisions
decision-index
decision-model
cross-encoder
text-embeddings-inference
Instructions to use tasksource/tasksource-decider-nano with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use tasksource/tasksource-decider-nano with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="tasksource/tasksource-decider-nano")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("tasksource/tasksource-decider-nano") model = AutoModel.from_pretrained("tasksource/tasksource-decider-nano", device_map="auto") - Notebooks
- Google Colab
- Kaggle
docs: consolidate v0 benchmark results
Browse files
README.md
CHANGED
|
@@ -23,24 +23,15 @@ It is LateOn (~149M parameters) fine-tuned on 512k real decisions covering three
|
|
| 23 |
**Why this architecture.** The situation (state + question) is encoded once into token vectors and reused for every candidate set, while each option is encoded independently. That gives two properties classifier heads don't have: caching the situation across decisions, and outputs that don't depend on option order (verified exactly permutation-equivariant).
|
| 24 |
|
| 25 |
- **Base model:** [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn)
|
| 26 |
-
- **Training data:** [`tasksource/tasksource-jev-typed-decisions`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (pinned revision `
|
| 27 |
- **Checkpoint selection:** Tasksource unseen tasks only (step 4000 of 8000)
|
| 28 |
- **Training code:** `decision-models` repo (`scripts/train_multivector.py`, canonical run)
|
| 29 |
|
| 30 |
-
##
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|---|---|
|
| 34 |
-
| Tasksource, unseen tasks (all three judgment types) | 46.8% |
|
| 35 |
-
| Tasksource, unseen test tasks (all three judgment types) | 49.7% |
|
| 36 |
-
| Typed Decisions | 42.6% |
|
| 37 |
-
| AG News | 73.8% |
|
| 38 |
-
| Emotion | 49.0% |
|
| 39 |
-
| Banking77 (77 options) | 44.7% |
|
| 40 |
-
| Fast Decisions (2,900 cases, 17 domains) | 41.3% exact match |
|
| 41 |
-
| classifier-benchmark v2 (866 cases) | 57.5 micro / 57.2 macro |
|
| 42 |
|
| 43 |
-
## Compact peers (all under 200M
|
| 44 |
|
| 45 |
| Model | Size | AG News | Emotion | Banking77 | Typed | cb-v2 micro |
|
| 46 |
|---|---|---|---|---|---|---|
|
|
@@ -51,6 +42,8 @@ It is LateOn (~149M parameters) fine-tuned on 512k real decisions covering three
|
|
| 51 |
| GLiNER2.5 Small | 74M | 70.4 | 54.2 | 66.5 | 34.0 | 53.5 |
|
| 52 |
| GLiClass Edge v3 | 33M | 65.5 | 50.1 | 23.2 | 40.0 | 35.3 |
|
| 53 |
|
|
|
|
|
|
|
| 54 |
Larger peers (GLiClass Modern-Large 399M, GLiNER2.5-Multi 287M, Laya-Multilingual 322M) are omitted from this table; Laya's 90.0 AG News is training overlap (AG News is in its training mix). v0 trades some raw high-cardinality accuracy for state caching and order-independence, which none of the classifier heads offer.
|
| 55 |
|
| 56 |
## Use it (only public packages)
|
|
|
|
| 23 |
**Why this architecture.** The situation (state + question) is encoded once into token vectors and reused for every candidate set, while each option is encoded independently. That gives two properties classifier heads don't have: caching the situation across decisions, and outputs that don't depend on option order (verified exactly permutation-equivariant).
|
| 24 |
|
| 25 |
- **Base model:** [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn)
|
| 26 |
+
- **Training data:** [`tasksource/tasksource-jev-typed-decisions`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (pinned revision [`071f0cf2201f06b2dea3e55323aa3e264284e139`](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions/tree/071f0cf2201f06b2dea3e55323aa3e264284e139)), 512k-decision stream, natural mix of the three judgment types, 519 tasks
|
| 27 |
- **Checkpoint selection:** Tasksource unseen tasks only (step 4000 of 8000)
|
| 28 |
- **Training code:** `decision-models` repo (`scripts/train_multivector.py`, canonical run)
|
| 29 |
|
| 30 |
+
## Evaluation
|
| 31 |
|
| 32 |
+
On Tasksource's held-out tasks across all three judgment types, v0 scores **46.8%** on unseen-dev and **49.7%** on unseen-test (1k examples each). It reaches **41.3% exact match** on Fast Decisions (2,900 cases across 17 domains). The table compares models on the same 1k examples per public dataset; classifier-benchmark v2 has 866 cases.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
+
### Compact peers (all under 200M)
|
| 35 |
|
| 36 |
| Model | Size | AG News | Emotion | Banking77 | Typed | cb-v2 micro |
|
| 37 |
|---|---|---|---|---|---|---|
|
|
|
|
| 42 |
| GLiNER2.5 Small | 74M | 70.4 | 54.2 | 66.5 | 34.0 | 53.5 |
|
| 43 |
| GLiClass Edge v3 | 33M | 65.5 | 50.1 | 23.2 | 40.0 | 35.3 |
|
| 44 |
|
| 45 |
+
v0's classifier-benchmark v2 macro accuracy is **57.2%**.
|
| 46 |
+
|
| 47 |
Larger peers (GLiClass Modern-Large 399M, GLiNER2.5-Multi 287M, Laya-Multilingual 322M) are omitted from this table; Laya's 90.0 AG News is training overlap (AG News is in its training mix). v0 trades some raw high-cardinality accuracy for state caching and order-independence, which none of the classifier heads offer.
|
| 48 |
|
| 49 |
## Use it (only public packages)
|