Add Noah 2.5 (Beinex, standard export): en
Model: beinexai/noah-v2.5-research-std, a 0.4B ModernBertForSequenceClassification with no custom code. Weights: open (open: true). They are downloadable after a one-click access form with automatic approval, under licence CC BY-NC 4.0 (non-commercial).
Command: the unmodified score.py, default en sample (n = 5,000), no --limit.
python score.py --model beinexai/noah-v2.5-research-std --config en --entailment-index 0 --name 'Noah 2.5 (Beinex, standard export)' --open true --out results/noah-2.5-beinex-std-en.json
| ROC-AUC | number (headline) | overall | date | string | table | prose |
|---|---|---|---|---|---|---|
| Noah 2.5 (standard export) | 0.7041 | 0.8145 | 0.9653 | 0.8459 | 0.8030 | 0.9085 |
Run on NVIDIA A100-SXM4-40GB, batch 64, max length 1024, 28.6 pairs/s. Repeated runs gave identical results. Our bootstrap 95% intervals are [0.678, 0.730] for number and [0.802, 0.826] for overall.
How the model was made.
- The model is initialised from the encoder of our Noah 2.5 research model. That model also has a numeric-verification channel and chunked inference; neither is in this submission.
- The whole model was then fine-tuned for one epoch on 60,000 items sampled from Noah 2.5's training pool, using Noah 2.5's probabilities as soft targets, in
score.py's input format. - Noah 2.5's training pool, in items: synthetic table claims (Beinex) 130,933, TabFact (train split) 91,675, MultiNLI 80,000, WANLI 60,000, e-SNLI 40,000, ContractNLI (train) 23,124, TAT-QA (train) 21,851, FinQA (train) 18,508, MiniCheck C2D/D2C 14,395. No ANLI.
Overlap with the benchmark.
- TabFact and FinQA are sources of the
enset. We don't know which upstream split your rows come from. - Before training, we removed every training item whose premise or claim matched a benchmark row by exact hash or a shared 20-word shingle: 1,184 items (TabFact 910, FinQA 30, synthetic 244). That check cannot catch paraphrases or reformatted tables.
- On the 3,551 rows from the other source families (sdk_grounding ± synth, semtabfacts, fever), the model scores 0.819 overall and 0.699 on numbers.
- On the same rows, DeBERTa-v3-large zero-shot scores 0.840 overall in our re-run of your scorer. So on these rows it is ahead of us. On all rows we reproduce its published 0.7856 exactly.
Evaluation history.
- Nutrient
enwas our main evaluation target across versions, and we used it to decide whether to submit. The bar was overall ≥ 0.786, set before this export was trained. - Five model variants have been scored on it: Noah v2, two Noah 2.5 training arms, and this export.
- Our synthetic numeric negatives (scale, sign, transposed digits, adjacent cell) are similar in kind to the benchmark's perturbations.
Happy to answer questions, or to re-run on your hardware.
We pulled beinexai/noah-v2.5-research-std and reran it ourselves with the benchmark/score.py against our public 5,000-row EN benchmark to verify before merging.
We can verify the result of Overall AUC 0.8145, Number AUC 0.7041
Since the license is CC BY-NC 4.0, we'll tag it as non-commercial-use on the board rather than lumping it in with the permissively-licensed "open" models.
We have added the entry to our leaderboard. Thank you for submitting the model and let us know if you want anything changed.
Thank you for verifying and merging, and for the quick turnaround. The non-commercial tag is right for these weights.
Best regards,
Mebin