v5: 64,000 steps on an updated mix (Decision Index 0.3 public 33.51); v4 kept at tag v4

README.md

@@ -5,6 +5,7 @@ base_model: cross-encoder/ettin-reranker-150m-v1
 datasets:
 - tasksource/tasksource-jev-typed-decisions
 - tasksource/synthetic-typed-decisions
+- LocalLLaMA/typed-decisions
 tags:
 - typed-decisions
 - decision-index
@@ -14,10 +15,10 @@ tags:
 
 # tasksource-decider-nano
 
-This repository was `tasksource/tasksource-jev-nano-v1`. Its `main` (tag `v4`) is the joint cross-encoder trained for
-48,000 steps. The same recipe at 24,000 steps (public index 29.05) stays at tag
-[`v3`](https://huggingface.co/tasksource/tasksource-decider-nano/tree/v3), at 12,000 steps (26.77) at tag
-[`v2`](https://huggingface.co/tasksource/tasksource-decider-nano/tree/v2), and the previous multi-vector model at tag
+This repository was `tasksource/tasksource-jev-nano-v1`. Its `main` (tag `v5`) is the joint cross-encoder trained for
+64,000 steps on an updated mix. Earlier versions stay at their tags: [`v4`](https://huggingface.co/tasksource/tasksource-decider-nano/tree/v4)
+(48,000 steps, public index 30.90), [`v3`](https://huggingface.co/tasksource/tasksource-decider-nano/tree/v3) (24,000 steps, 29.05),
+[`v2`](https://huggingface.co/tasksource/tasksource-decider-nano/tree/v2) (12,000 steps, 26.77) and the previous multi-vector model at
 [`v1`](https://huggingface.co/tasksource/tasksource-decider-nano/tree/v1).
 
 A 150M-parameter typed-decision model for the [Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index).
@@ -26,9 +27,9 @@ generate text.
 
 | | |
 |---|---|
-| Decision Index 0.3, public index | **30.90** |
+| Decision Index 0.3, public index | **33.51** |
 | Requests | 140,171 `ok`, 7 `unsupported` (pairs above 8,192 tokens) |
-| Latency, 1× NVIDIA A10 23 GB, one request at a time | median 52 ms · mean 65 ms · p80 63 ms |
+| Latency, 1× NVIDIA A30 24 GB (shared), one request at a time | median 23 ms · mean 33 ms · p80 37 ms |
 | Complete results | [runs/tasksource-decider-nano](https://huggingface.co/datasets/tasksource/decision-index-results/tree/main/runs/tasksource-decider-nano) |
 
 ## Architecture
@@ -68,16 +69,18 @@ python -m decision_index pipeline --engine decision_index_engine:DeciderEngine \
 
 ## Training
 
-48,000 steps, batches of 8 requests, learning rate 2e-5, inputs up to 2,048 tokens, seed 7. The loss is soft
-cross-entropy over each question's options. The training mix has 480,000 requests:
+64,000 steps, batches of 8 requests, learning rate 2e-5, inputs up to 2,048 tokens, seed 1. The loss is soft
+cross-entropy over each question's options. 20% of the questions are reworded with task-specific prompt styles, and 5% of
+the questions get a "none of the above" option. The training mix has 480,000 requests:
 
 | Share | Source |
 |---:|---|
-| 42% | [tasksource/tasksource-jev-typed-decisions](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (658 tasks, at most 600 questions per task) |
+| 42% | [tasksource/tasksource-jev-typed-decisions](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) (release `9fe3dd87`, balanced across tasks) |
 | 30% | *train* splits of public datasets behind Decision Index benchmarks, e.g. MMLU, ANLI, WinoGrande, HellaSwag, ARC, CLINC150, BANKING77, Amazon ESCI, HoVer, RAGTruth, New Yorker caption contest, When2Call, ACOS, iSarcasmEval, NLI4CT, ContractNLI, GSM8K (at most 10,000 requests each) |
-| 12% | tool use: ToolRet train, Glaive function calling, Hermes function calling |
+| 12% | tool use: Glaive, xLAM, ToolACE and Hermes function calling, ToolRet train, Home Assistant tool calls (at most 60,000 questions per source) |
 | 8% | LLM routing: xRouteBench and LLMRouterBench train data |
 | 8% | [tasksource/synthetic-typed-decisions](https://huggingface.co/datasets/tasksource/synthetic-typed-decisions) |
+| 0.5% | [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) *train* (2 passes) |
 
 No evaluation split is used, and benchmarks without a train split (for example RouterBench) are evaluation only.
 Every training request goes through a firewall against the Decision Index 0.3 suite: an exact match of the
@@ -88,10 +91,10 @@ request.
 
 | Area | Skill | Raw |
 |---|---:|---:|
-| Knowledge & Reasoning | 7.8 | 30.8 |
-| Language Understanding | 37.7 | 54.7 |
-| Retrieval & Classification | 49.4 | 63.3 |
-| Tools & Automation | 43.6 | 48.3 |
-| Arts & Human Taste | 12.8 | 42.7 |
+| Knowledge & Reasoning | 9.7 | 31.9 |
+| Language Understanding | 38.6 | 57.6 |
+| Retrieval & Classification | 50.1 | 64.0 |
+| Tools & Automation | 50.9 | 54.4 |
+| Arts & Human Taste | 16.6 | 44.7 |
 
-Raw index 47.86; kit `9eb2dbe`, edition 0.3, 140,171 requests scored.
+Raw index 50.38; kit `9eb2dbe`, edition 0.3, 140,171 requests scored.

decider_config.json

@@ -4,19 +4,21 @@
   "base": "cross-encoder/ettin-reranker-150m-v1",
   "train_max_length": 2048,
   "inference_max_length": 8192,
-  "step": 48000,
+  "step": 64000,
   "train": {
     "arch": "joint",
     "opt_pool": "marker",
     "shuffle_options": false,
-    "manifest": "runs/manifests/mix_all_480k.jsonl",
+    "nota": 0.05,
+    "long": null,
+    "manifest": "runs/manifests/mix_480k_9fe3dd87_tool3_td05.jsonl",
     "extra_manifest": null,
     "extra_frac": 0.0,
     "lr": 2e-05,
     "batch_requests": 8,
     "max_k": 32,
-    "seed": 7,
-    "total_steps": 48000,
+    "seed": 1,
+    "total_steps": 64000,
     "max_length": 2048,
     "pack": 0,
     "init": null,
@@ -25,7 +27,8 @@
     "layout": "shared",
     "pool": "opt",
     "row_tokens": 8192,
-    "tok_budget": 12288
+    "tok_budget": 12288,
+    "attn": "cached"
   },
   "layout": "shared",
   "pool": "opt",

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e26e29f12339163b473f985270ebb4b9b3b90ee3b7ffd488715d45550842bef4
+oid sha256:ef09f0a1f2490512e69e56af9870fecfbfe321bccf18fb46427e511193ec1c1b
 size 298044578