Upcoming submission (Noah 2.5, Beinex): one question about the TabFact/FinQA rows
Hi Nutrient team, thanks for the benchmark, and for making the scorer so easy to reproduce.
We're Beinex and we'll shortly open a PR for Noah 2.5 (standard export), a 0.4B ModernBertForSequenceClassification with no custom code. It scores 0.704 number / 0.814 overall ROC-AUC on en with your unmodified score.py (default sample, no --limit).
The weights are open on the Hub under CC BY-NC 4.0, with a one-click access form (automatic approval), so you can re-run it directly: beinexai/noah-v2.5-research-std.
One question before we submit. Our training pool includes the train splits of TabFact and FinQA, and both are sources of your en set. Which upstream split do your TabFact and FinQA rows come from?
We removed every training item that matched a benchmark premise or claim (exact hash or a shared 20-word shingle; 1,184 items).
We want the disclosure in our PR to be accurate.
The PR will also report our score on the other source families.
Thanks!
Mebin Joy,
Beinex
Hi Mebin,
For both TabFact and FinQA, the benchmark pull from the test split:
- TabFact: https://github.com/wenhuchen/Table-Fact-Checking/blob/master/data/test_id.json
- FinQA: https://github.com/czyssrs/FinQA/blob/main/dataset/test.json
Best regards,
Austin Nguyen
Nutrient
Thanks Austin - that’s what we needed, and thank you for re-running the model and adding the entry so quickly.
For the record: Noah trains on the TabFact and FinQA train splits only, never the test splits. Every training item built from tables or financial documents (TabFact, FinQA, TAT-QA, our synthetic cell claims, MiniCheck) is additionally checked against the benchmark’s en rows - exact matches on the hypothesis and near-duplicates on the premise - and 1,184 items were removed. The general NLI sources (MNLI, WANLI, e-SNLI, ContractNLI) share no source with the benchmark and were not filtered.
We’ll keep reporting the AUC excluding the TabFact/FinQA source families alongside the overall number in any future entry. Closing this one.
Best regards,
Mebin