v3: same recipe as v2 trained for 24,000 steps (Decision Index 0.3 public 29.05); v2 kept at tag v2

README.md

@@ -13,8 +13,10 @@ tags:
 
 # tasksource-decider-nano
 
-This repository was `tasksource/tasksource-jev-nano-v1`. Its `main` (tag `v2`) is now a new model; the previous
-multi-vector model (v1) stays at tag [`v1`](https://huggingface.co/tasksource/tasksource-decider-nano/tree/v1).
+This repository was `tasksource/tasksource-jev-nano-v1`. Its `main` (tag `v3`) is the joint cross-encoder trained for
+24,000 steps. The same model trained for 12,000 steps (public index 26.77) stays at tag
+[`v2`](https://huggingface.co/tasksource/tasksource-decider-nano/tree/v2), and the previous multi-vector model at tag
+[`v1`](https://huggingface.co/tasksource/tasksource-decider-nano/tree/v1).
 
 A 150M-parameter typed-decision model for the [Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index).
 It takes a `state` and typed questions (`choice`, `noul`) and returns a probability for every option. It does not
@@ -22,9 +24,9 @@ generate text.
 
 | | |
 |---|---|
-| Decision Index 0.3, public index | **26.77** |
+| Decision Index 0.3, public index | **29.05** |
 | Requests | 140,171 `ok`, 7 `unsupported` (pairs above 8,192 tokens) |
-| Latency, 1× NVIDIA A10 23 GB, one request at a time | median 45 ms · mean 71 ms · p80 71 ms |
+| Latency, 1× NVIDIA A10 23 GB, one request at a time | median 73 ms · mean 96 ms · p80 95 ms |
 | Complete results | [runs/tasksource-decider-nano](https://huggingface.co/datasets/tasksource/decision-index-results/tree/main/runs/tasksource-decider-nano) |
 
 ## Architecture
@@ -64,7 +66,7 @@ python -m decision_index pipeline --engine decision_index_engine:DeciderEngine \
 
 ## Training
 
-12,000 steps, batches of 8 requests, learning rate 2e-5, inputs up to 2,048 tokens, seed 7. The loss is soft
+24,000 steps, batches of 8 requests, learning rate 2e-5, inputs up to 2,048 tokens, seed 7. The loss is soft
 cross-entropy over each question's options. The training mix has 240,000 requests:
 
 | Share | Source |
@@ -84,10 +86,10 @@ request.
 
 | Area | Skill | Raw |
 |---|---:|---:|
-| Knowledge & Reasoning | 6.0 | 29.5 |
-| Language Understanding | 28.2 | 47.9 |
-| Retrieval & Classification | 42.0 | 58.2 |
-| Tools & Automation | 44.6 | 49.8 |
-| Arts & Human Taste | 13.9 | 43.5 |
+| Knowledge & Reasoning | 6.6 | 29.8 |
+| Language Understanding | 33.2 | 51.5 |
+| Retrieval & Classification | 47.1 | 61.3 |
+| Tools & Automation | 43.3 | 48.2 |
+| Arts & Human Taste | 14.1 | 43.2 |
 
-Raw index 45.1; kit `9eb2dbe`, edition 0.3, 140,171 requests scored.
+Raw index 46.41; kit `9eb2dbe`, edition 0.3, 140,171 requests scored.

decider_config.json

@@ -4,7 +4,7 @@
   "base": "cross-encoder/ettin-reranker-150m-v1",
   "train_max_length": 2048,
   "inference_max_length": 8192,
-  "step": 12000,
+  "step": 24000,
   "train": {
     "arch": "joint",
     "opt_pool": "marker",
@@ -16,7 +16,7 @@
     "batch_requests": 8,
     "max_k": 32,
     "seed": 7,
-    "total_steps": 12000,
+    "total_steps": 24000,
     "max_length": 2048,
     "pack": 0,
     "init": null,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a96dcfdf2d4e7da62429ba9f836535adf3a328f30cb51366095ad2c38fb13659
+oid sha256:ca302526bc031a820677508630f72806c7f757fe25191d5eb1e154f658191717
 size 596074556