tasksource-decider-nano

This repository was tasksource/tasksource-jev-nano-v1. Its main (tag v3) is the joint cross-encoder trained for 24,000 steps. The same model trained for 12,000 steps (public index 26.77) stays at tag v2, and the previous multi-vector model at tag v1.

A 150M-parameter typed-decision model for the Decision Index. It takes a state and typed questions (choice, noul) and returns a probability for every option. It does not generate text.

Decision Index 0.3, public index	29.05
Requests	140,171 ok, 7 unsupported (pairs above 8,192 tokens)
Latency, 1× NVIDIA A10 23 GB, one request at a time	median 73 ms · mean 96 ms · p80 95 ms
Complete results	runs/tasksource-decider-nano

Architecture

A joint cross-encoder on cross-encoder/ettin-reranker-150m-v1 (ModernBERT-style encoder, 8,192-token context). Every option is scored against the full state, its question and the option text, like the reranker pair (state, "Question: <instructions>\nOption: <option>"), by a linear head on the mean of the option's hidden states. The scores are softmaxed over each question's options.

The state is encoded once per request. All questions and options of a request share it through a tree attention mask: the state attends to itself, a question to the state and itself, and an option to the state, its question and itself. Rotary positions are those of the single pair, so every option gets the score it would get on its own (up to float rounding), while a request with 50 options costs one state encoding instead of 50.

Inference reads every input in full. There is no truncation: a pair above 8,192 tokens raises Unsupported.

Usage

from decider import Decider
model = Decider.from_pretrained("tasksource/tasksource-decider-nano")
model.answer(state={"review": "The battery died after a week."},
             questions={"q": {"type": "choice", "instructions": "What is the sentiment?",
                              "criteria": {"neg": "negative", "pos": "positive"}}})

Decision Index run (kit 0.3, 9eb2dbe), from a local copy of this repository:

hf download tasksource/tasksource-decider-nano --local-dir tasksource-decider-nano
cd tasksource-decider-nano
python -m decision_index pipeline --engine decision_index_engine:DeciderEngine \
    --option repo=. --edition 0.3 --out runs/tasksource-decider-nano

decider.py and decision_index_engine.py are self-contained (PyTorch and transformers only).

Training

24,000 steps, batches of 8 requests, learning rate 2e-5, inputs up to 2,048 tokens, seed 7. The loss is soft cross-entropy over each question's options. The training mix has 240,000 requests:

Share	Source
42%	tasksource/tasksource-jev-typed-decisions (658 tasks, at most 600 questions per task)
30%	train splits of public datasets behind Decision Index benchmarks, e.g. MMLU, ANLI, WinoGrande, HellaSwag, ARC, CLINC150, BANKING77, Amazon ESCI, HoVer, RAGTruth, New Yorker caption contest, When2Call, ACOS, iSarcasmEval, NLI4CT, ContractNLI, GSM8K (at most 10,000 requests each)
12%	tool use: ToolRet train, Glaive function calling, Hermes function calling
8%	LLM routing: xRouteBench and LLMRouterBench train data
8%	tasksource/synthetic-typed-decisions

No evaluation split is used, and benchmarks without a train split (for example RouterBench) are evaluation only. Every training request goes through a firewall against the Decision Index 0.3 suite: an exact match of the normalized state or of the (state, question) pair, and a template-insensitive word 6-gram content check, drop the request.

Evaluation

Area	Skill	Raw
Knowledge & Reasoning	6.6	29.8
Language Understanding	33.2	51.5
Retrieval & Classification	47.1	61.3
Tools & Automation	43.3	48.2
Arts & Human Taste	14.1	43.2

Raw index 46.41; kit 9eb2dbe, edition 0.3, 140,171 requests scored.