roberta-large-mnli-dynamic-labels

A dynamic-label zero-shot text classifier: roberta-large-mnli fine-tuned to judge "This text is about ." as entailment vs contradiction across 16 datasets, so the label set is supplied at inference time and nothing fixes a vocabulary. Independent fine-tune; NOT affiliated with or endorsed by Meta, the Jev project, openjev, or the authors of any benchmark or model that appears below as a cited comparison.

How it works

• Input: the (premise, hypothesis) pair (text, "This text is about ."), encoded the way the MNLI head expects (truncation='longest_first', max 512 tokens).
• Training target: gold label -> entailment; 2 sampled wrong labels per example -> contradiction (15% of negatives drawn from other datasets' label sets).
• Readout: renormalized P(entailment) per candidate label.
  • single-label: softmax across the per-label log-odds (quickstart below)
  • multi-label: threshold the raw P(entailment) per label

Training data

16 datasets (caps on positive examples; negatives on top). Profile: full.

dataset	task	positives
SNLI	Stanford NLI	25000
MNLI	MultiNLI	25000
WANLI	Worker-and-AI NLI	30000
ANLI	Adversarial NLI	10000
AGNEWS	news topics	6000
DBPEDIA	ontology types	6000
YAHOO	Q&A topics	6000
IMDB	movie reviews	5000
SST2	sentiment	5000
YELP	review polarity	5000
BANKING77	banking intents	8000
CLINC	assistant intents	8000
HWU64	assistant intents	5000
GOEMOTIONS	emotions	5000
EMOTION	emotions	5000
MASSIVE	assistant intents	5000

Training procedure

• Base: roberta-large-mnli (355M), full fine-tune, fp32 on Apple MPS
• 3 epochs, cosine LR schedule at 1e-5, effective batch 64 (batch 8 x 8 accumulation)
• weight decay 0.01, seed 42, max sequence length 512
• Wall-clock training time: 7.0 h (step-rate estimate from checkpoint timestamps, 7218 steps)

Training-data licenses

Model weights and this card: MIT. Base model roberta-large-mnli: MIT. The 16 training datasets carry their own licenses (checked on the Hub 2026-10-15):

dataset (Hub repo)	license
SNLI (stanfordnlp/snli)	CC BY-SA 4.0
MNLI (nyu-mll/multi_nli)	CC BY 3.0 / CC BY-SA 3.0 / MIT / other (multi-part)
WANLI (alisawuffles/WANLI)	CC BY 4.0
ANLI (facebook/anli)	CC BY-NC 4.0 (non-commercial)
AGNEWS (fancyzhx/ag_news)	not stated
DBPEDIA (fancyzhx/dbpedia_14)	CC BY-SA 3.0
YAHOO (yahoo_answers_topics)	not stated
IMDB (stanfordnlp/imdb)	other
SST2 (stanfordnlp/sst2)	not stated
YELP (Yelp/yelp_review_full)	other
BANKING77 (PolyAI/banking77)	CC BY 4.0
CLINC (clinc_oos)	CC BY 3.0
HWU64 (PolyAI/hwu64)	not stated
GOEMOTIONS (google-research-datasets/go_emotions)	Apache-2.0
EMOTION (dair-ai/emotion)	other
MASSIVE (AmazonScience/massive)	CC BY 4.0

Note: ANLI is CC BY-NC 4.0 (non-commercial), and several datasets state no license - review this table before commercial use of the fine-tuned weights. The weights and card are MIT; the upstream dataset terms still govern the data itself.

Evaluation

tweet_eval sentiment (3 labels) + emotion (4 labels) test rows were NOT in the training mixture. 400 rows each, fixed seed. One temperature fitted on ag_news + sst2 + banking77 (in-mixture, calibration only). T=1.0 and the fitted T are both reported.

dataset / readout	accuracy	macro-F1	NLL	Brier	ECE
tweet_eval/sentiment (T=1.0)	0.440	0.414	4.324	0.997	0.473
tweet_eval/sentiment (T=fitted)	0.440	0.414	1.218	0.733	0.239
tweet_eval/emotion (T=1.0)	0.590	0.514	1.697	0.647	0.254
tweet_eval/emotion (T=fitted)	0.590	0.514	1.033	0.550	0.094
overall (T=1.0)	0.515	0.480	3.011	0.822	0.363
overall (T=fitted)	0.515	0.480	1.126	0.642	0.110

S1MB - shared benchmark, measured here

hotchpotch/S1MB-english-v1 (106 subsets, 26,269 typed decisions) evaluated with the shipped scorer: premise = rendered state + documents + instructions, candidates = the decision's criteria, prediction = argmax renormalized P(entailment), gold = the target distribution's mode. RAW accuracy (the published leaderboard columns are baseline- adjusted scores, which is why they are kept separate below):

decision type	n	accuracy
choice	8764	0.313
noul	12357	0.446
score	5148	0.149
overall	26269	0.343

Subsets whose upstream dataset was in the training mixture (banking77, clinc, dbpedia, go_emotions, hwu64, laya__ag_news, massive, snli): accuracy 0.484 on n=3651 of judgments; the other subsets: accuracy 0.3207 on n=22618. Same distribution, other splits - S1MB itself does not establish training-data non-overlap.

Multi-label readout

go_emotions test rows (200 rows, 28 labels; in-mixture dataset, checking the thresholding readout, not unseen accuracy):

• sample-F1 at threshold 0.5: 0.1506
• best threshold 0.2 -> sample-F1 0.1754
• precision@3: 0.55

Latency

On Apple M2 Pro (mps, 355.4):

sweep	mean ms	p50 ms	p95 ms
batch=1, 2 labels	30.2	30.1	31.9
batch=1, 4 labels	41.4	41.1	43.1
batch=1, 10 labels	79.2	77.7	81.0
batch=1, 27 labels	178.9	177.4	185.5
batch=1, 4 labels	38.6	38.3	41.2
batch=8, 4 labels	214.3	210.2	231.2
batch=32, 4 labels	840.5	840.8	850.7

Comparison with open Jev-style decision models

MEASURED rows ran on this machine in this notebook; every other row is published, or computed from published measurements, and carries its source.

Same benchmark, same metric: S1MB raw accuracy

One unit everywhere: raw accuracy (percent) of the argmax prediction on the same hotchpotch/S1MB-english-v1 benchmarks and the same public targets. Our row was measured here with the shipped scorer; the other rows are computed from the published per-benchmark predictions of those models in hotchpotch/s1mb-result (noul rows use the published accuracy field of each file; tie order on their rounded probabilities can shift a noul row by ~0-2pp).

model	params	Choice	Noul	Score	overall
roberta-large-mnli-dynamic-labels (this model) (MEASURED)	355.4M	31.3%	44.6%	14.9%	34.3%
Jev 1.13 (TypeSafe AI)	-	76.8%	83.1%	63.1%	77.1%
OpenJev 27B (openjev)	27B	75.5%	83.4%	67.2%	77.6%
Open-Jev-27B-v1.1 (ZefanCai)	25.6B	72.9%	82.5%	66.5%	76.2%
Open-Jev-9B (ZefanCai)	7.9B	65.6%	82.7%	62.1%	73.0%
Tev1-4B-experimental	4.5B	65.5%	78.8%	63.1%	71.3%
ModernBERT-large-zeroshot-v2.0	396M	36.4%	49.9%	16.2%	38.8%
open-jev-deberta-v3-large (kotobalabs)	304M	39.8%	65.8%	19.4%	48.0%
Qwen3-Reranker-4B	4B	44.9%	71.2%	16.9%	51.8%
laya	421M	34.2%	65.1%	34.3%	48.8%
decider-4b (Mapika)	4B	63.7%	81.3%	60.0%	71.2%
chance-level baseline (uniform for Choice, majority for Noul)	-	22.9%	72.9%	-	-

Source: https://huggingface.co/datasets/hotchpotch/s1mb-result (accessed 2026-10-15).

S1MB leaderboard ranking (their unit: baseline-ADJUSTED scores)

The leaderboard ranks models with baseline-adjusted scores and a Borda count - a DIFFERENT unit from the raw accuracy above, kept only for ranking context. These are not percentages of anything; do not compare them to the table above.

model	params	Borda	TaskAvg	Noul	Choice	Score
Jev 1.13 (TypeSafe AI)	-	86.51	59.59	64.63	67.22	46.92
bekko-system-one-v0-400m	395M	76.21	50.6	51.24	61.32	39.25
Open-Jev-9B (ZefanCai)	7.9B	72.54	52.1	55.76	55.51	45.04
Open-Jev-27B-v1.1 (ZefanCai)	25.6B	72.35	56.88	51.48	66.91	52.26
Tev1-4B-experimental	4.5B	71.67	47.07	50.73	53.36	37.11
kev-9b (jaredpalmer)	7.9B	70.52	42.4	50.21	56.36	20.64
kev-4b (jaredpalmer)	4.2B	68.81	42.33	48.48	52.03	26.47
JevK5	4.2B	67.49	41.14	48.93	52.84	21.65
OpenJev-4B (Alex)	4.5B	63.12	36.46	42.77	47.49	19.11
bekko-system-one-v0-68m	68M	62.23	40.46	42.91	51.62	26.85
Open-Jev-2B (ZefanCai)	1.9B	50.56	34.64	38.83	39.82	25.27
bekko-system-one-v0-17m	17M	46.57	27.57	31.43	35.57	15.7
OpenJev-2B (Alex)	2.2B	44.38	25.21	29.72	37.8	8.11
decider-0.8b (Mapika)	752M	42.65	22.57	28.97	32.36	6.39
Tev1-0.8B-experimental	853M	38.59	20.43	25.97	27.15	8.16
kev-0.8b (jaredpalmer)	753M	35.2	18.68	23.61	27.36	5.07
von	395M	32.4	16.21	20.15	23.99	4.48
OpenJev-0.8B (Alex)	853M	31.4	16.02	18.9	23.79	5.36
laya-typed-decisions	421M	31.22	15.0	20.06	18.93	5.99
laya	421M	29.31	13.36	20.19	16.31	3.58
laya-multilingual	322M	24.14	9.28	14.01	13.03	0.79
minojev	1.7B	19.99	7.93	8.57	14.56	0.67
JevForge-0.8B	854M	12.14	1.92	0.0	5.75	0.0

openjev's own suite (different benchmark, CITED)

model	accuracy on their 10,000-question suite
Jev (hosted API)	85.4%
OpenJev (27B)	84.0%
the same base model before tuning	80.4%
Nimble 9B (open)	75.7%

Latency cited there: ~80 ms short-text / ~210 ms web decisions on one H100 (FP8). License: CC BY-NC 4.0 (non-commercial).

JevBench leaderboard (different benchmark; latency and cost, CITED)

model	params	capability	p50 latency s	USD / 1000 decisions
Quyet-1.0-Large (chinhnc)	31B	81.69	0.116	0.0445
wity-1 (Wity)	-	79.12	1.575	0.0236
Sage 1.3.0 (Levanto Labs)	-	78.6	0.152	0.0247
deck-31B (krishna765)	31B	77.59	0.127	0.0475
Jev 1.13.0 (TypeSafe AI)	-	77.13	0.239	0.0323
torchcast-decision-12b	12B	71.74	0.037	0.0285
Jev-Omni (akhilaaa3)	12B	71.29	0.155	0.0292
Winnow-12B (EldanRing)	12B	71.25	0.117	0.0281
decider-4b v2 (Mapika)	4B	64.89	0.025	0.0152
lev (Interfaze AI)	4B	61.7	0.102	0.0188
jev-local (Qwen3.5-9B)	9B	57.46	0.365	0.0235

Our latency on this Mac is in the latency table above (4 labels, batch 1: 41.4 ms mean); it is not comparable to their server hardware.

Sources: https://huggingface.co/blog/hotchpotch/system-one-mosaic-benchmark; https://huggingface.co/openjev/openjev; https://huggingface.co/datasets/benchmarkheaven/jevbench. Licenses: openjev/openjev is CC BY-NC 4.0 (non-commercial); S1MB subsets carry their original upstream licenses (see its dataset card).

Artificial Analysis Decision Index benchmarks

The AA Decision Index (https://artificialanalysis.ai/articles/artificial-analysis-decision-index, 2026-10-13) is a weighted average of four benchmarks. This model was evaluated on the three public ones with the shipped readout; the fourth (AA-Omniscience-Decision) is a private eval set and cannot be run, and the composite index is NOT computed (the weights are unpublished). Our numbers are MEASURED; the others are CITED from the article.

benchmark	ours	chance	majority	cited leaders (AA article)
HLE-MultiChoice	19.8% (n=500)	22.9%	-	Jev 1.13: 21
When2Call	33.5% (n=1000)	25.0%	-	Jev 1.13: 81; Celeris-1 Decision: 84
WildGuardTest	56.9% (n=1699)	50.0%	55.6%	Jev 1.13: 94; GLiDE: 95; Decider v1.1 27B: 94
AA-Omniscience-Decision	not runnable (private)	-	-	Jev 1.13: 35 (accuracy); d1 (Liquid AI): 41

WildGuardTest per-label recall: harmful 30.6%, unharmful 77.8%. This run classifies the dataset's own prompt-harm labels; the AA article measures response safety - same public data, different protocol.

Caveats: HLE and WildGuardTest ran on public mirrors (the official repos are Hub-gated), so those two are approximate; When2Call is the same public MCQ split the article uses. Per-benchmark artifacts with full protocols live in eval_aa/.

Quickstart

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model = AutoModelForSequenceClassification.from_pretrained('<your-org>/roberta-large-mnli-dynamic-labels')
tokenizer = AutoTokenizer.from_pretrained('<your-org>/roberta-large-mnli-dynamic-labels')
id2label = model.config.id2label
ENTAIL_ID = next(k for k, v in id2label.items() if str(v).upper() == 'ENTAILMENT')
CONTRA_ID = next(k for k, v in id2label.items() if str(v).upper() == 'CONTRADICTION')

def build_prompt(text, hypothesis):
    return (
        'Text: ' + text.strip().replace('\n', ' ') + '\n'
        'Hypothesis: ' + str(hypothesis).strip() + '\n'
        'Verdict:'
    )

def label_to_hypothesis(label):
    return 'This text is about ' + str(label).strip().rstrip('.') + '.'
labels = ['sports', 'politics', 'finance']
text = 'The senate passed the budget after a late-night session.'
probs = []
for lab in labels:
    enc = tokenizer(text, label_to_hypothesis(lab), return_tensors='pt',
                    truncation='longest_first', max_length=512)
    with torch.no_grad():
        logits = model(**enc).logits[0]
    p = torch.softmax(torch.stack([logits[ENTAIL_ID], logits[CONTRA_ID]]), dim=0)[0]
    probs.append(float(p))
print('multi-label raw P(entail):', dict(zip(labels, probs)))
import math
z = [math.log(p / (1 - p)) for p in probs]   # per-label log-odds
q = torch.softmax(torch.tensor(z), dim=0)
print('single-label (calibrated):', dict(zip(labels, q.tolist())))

Limitations

• 512-token ceiling on the (text, hypothesis) pair; longer inputs are truncated.
• Softmax over the raw P(entailment) values is flat by construction (each P is bounded in [0,1]); use the log-odds softmax for single-label probabilities or raw P plus a threshold for multi-label, as in the quickstart.
• Calibration was fitted on ag_news / sst2 / banking77; expect worse calibration off-domain.
• Trained on 16 public datasets with varied licenses (see the table above); ANLI is CC BY-NC 4.0, so review before commercial use. Weights and card: MIT.
• English-centric training data.
• Comparison numbers for other models are CITED from published tables (the S1MB leaderboard snapshot, the openjev model card, the JevBench leaderboard dataset) or computed from their published s1mb-result predictions; our S1MB numbers are raw accuracy while the S1MB leaderboard columns are baseline-adjusted scores.
• The S1MB readout is the shipped NLI hypothesis ("This text is about ."), not the option-letter logit readout the S1MB models use.
• 8 S1MB subsets derive from datasets in the training mixture (train split there, evaluation split here); see the overlap split in the S1MB section.