scienceverse/bibr-statement-tagger-v1

statement-tagger

statement-tagger is a model for the bibr stage post_parse/integrity statements (proposed). Input: tokenizer pair (previous block's first 200 characters, block text), max 192 tokens; a block is a layout region on pages 1-2 or a text-layer line group elsewhere (as bibr extracts them).

Files

File	Bytes	sha256 (first 16)
bibr_model.json	4,442	191a8babda3e3026
onnx/bibr_onnx.json	2,550	37eb7225f10156c4
onnx/model.onnx	90,411,056	24e0fdb7b0841701
onnx/tokenizer.json	527,292	325e6539e0908116

The full checksums are in checksums.json.

Architecture

• encoder: sentence-transformers/all-MiniLM-L6-v2
• head: masked mean pool + linear, 7 sigmoid outputs (coi, funding, ethics, data_availability, author_contributions, acknowledgment, supplementary); per-type thresholds fit on PDF-view dev
• input: tokenizer pair (previous block's first 200 characters, block text), max 192 tokens; a block is a layout region on pages 1-2 or a text-layer line group elsewhere (as bibr extracts them)
• output: adjacent blocks of a type are joined into one statement (statements/join.py)
• format: ONNX fp32 opset 17 (90 MB), parity with torch 6e-8

Training data

• rows: 922468 train rows (400000 JATS-view sample + 174156 PDF-view rows x3)
• jats view: 44869 PMC papers (train 39580, dev 2154, test 2244, test_journal 891), 1.71M block rows; positives from JATS statement markup (sec-type, fn-type, ack, funding-statement, headings in 5 languages), sampled and hard negatives, ambiguous blocks never trained; at most 300 train / 25 eval papers per journal
• pdf view: 5165 papers extracted by bibr (train 4518, dev 243, test 256, test_journal 148); JATS statements projected onto PDF blocks by word 3-gram containment (recovery 94-98 percent per type, supplementary 46 percent)
• languages: English plus 448 non-English train papers (de, fr, zh, pt, es, ru, ko, pl)
• licences: CC BY, CC BY-SA, CC0 only, per row; registry check 0 of 44870 built papers excluded

Every evaluation set in the exclusion registry was kept out of training. Registry: exclusions, build manifest sha256 d8d427de3ea7b954. Split: corpus-split-v1, sha256 c6fc246571f435d7.

Evaluation

Set	Metric	Value	Shipped	Difference	Interval	Notes
silver paired PDF view (398)	text_f1_macro	0.908				vs: JATS-trained arm 0.835, rules 0.741
silver paired JATS view (398)	text_f1_macro	0.953
silver PDF test (256)	block_f1_macro	0.861
silver PDF test_journal (148)	block_f1_macro	0.825
statements gold (97 papers, 262 direct statements)	found_pooled	0.908			95% [0.868, 0.943]	spurious statements not judged (no text layer); vs_bibr_0.8.0_full: 0.744; vs_rules: 0.844
Metacheck psychsci COI (247)	found	0.984			95% [0.968, 0.996]	vs_rtransparent: 0.729; vs_bibr_full_12.2: 0.891
Metacheck psychsci funding (247, 192 with funding)	found	0.833			95% [0.781, 0.883]	vs_rtransparent: 0.542; vs_bibr_full_12.2: 0.677
Metacheck MDPI COI (249, GROBID-view rows)	found	0.855			95% [0.807, 0.9]	vs_rtransparent: 0.876
Metacheck economics COI (249, 68 with COI, GROBID-view rows)	found	0.75			95% [0.643, 0.842]	worse than the rules here; vs_rtransparent: 0.853
gate192, whole pipeline (65 papers with acknowledgment gold)	sec.type:acknowledgment:presence	0.857	0.696	+0.161	difference 95% [0.071, 0.259]	9 papers better, none worse
gate192, whole pipeline (54 papers with author-contribution gold)	sec.type:author_contributions:presence	0.9	0.7	+0.2	difference 95% [0.098, 0.314]	10 papers better, none worse
gate192, whole pipeline (99 papers with discussion gold)	sec.type:discussion:unigram_precision	0.747	0.737	+0.010	difference 95% [0.003, 0.019]	17 papers better, none worse
gate192, whole pipeline (60 papers with heading gold)	header.detect	0.851	0.856	-0.005	difference 95% [-0.008, -0.003]	12 papers one false heading worse, none better: the synthetic headings of new statement sections
gate192 and audience_eval200, whole pipeline (190 and 198 papers)	core fields (title, authors, abstract, doi, abstained)	identical	identical			no front-matter field differs on any paper

Known weaknesses

• the validation scorer judges no spurious statement without a text layer, so the gold numbers are recall-like; precision comes from the silver sets and the Metacheck cohorts only
• silver labels are JATS markup; ethics paragraphs inside methods and untagged statements are ambiguous and left out of training and of precision
• supplementary material statements are rarely printed as such (projection recovery 46 percent); weakest type
• running heads and repeated footers make up 9-13 percent of PDF blocks; JATS-only training typed some as acknowledgment
• CPU cost 2.9-3.6 ms per block at batch 1 and 4.1-4.3 ms at batch 32 on 4 pinned threads (512 PDF test blocks, tokenisation included; host load 9-11, so not an idle machine); about 300 blocks per paper, about 1 s per paper; 7-17 ms per block under heavier shared load
• the non-English test slice is small (31 paired papers)
• on Metacheck's economics COI cohort the rtransparent rules find more statements (0.853 vs 0.750); the cohort is GROBID text, a view the tagger never trained on
• psychsci and the statements gold may have been seen while bibr's own rules were developed; the export heading filter was fixed after a first look at the gold results
• in the pipeline a statement with no heading of its own gets a new section with a heading bibr supplies ("Author contributions", "Acknowledgments"). Since export schema 12.4 the export marks such a heading (extraction.diagnostics.section_classification[].header_supplied), but it does not carry the printed run-in label instead (most such papers print one, e.g. "Author Contributions:"). Heading detection reads 0.5 points lower on gate192 for this reason (12 of 60 papers, one heading each), measured with a scorer that does not yet leave the marked headings out

How it was made

• trainer script: scripts/train_statement_classifier.py
• data builder: scripts/build_statements_dataset.py (jats, pdf, manifest, ids)
• export script: scripts/export_statement_classifier_onnx.py
• recipe: lr 5e-5 encoder / 1e-3 head, batch 64, bf16, BCE, 2 epochs (best = epoch 1 by PDF-view dev macro block F1), length-grouped batches; PDF-view rows repeated 3x
• commit: 8521e69f6bb540e13e87fa27f785fa518477ed2e
• seed: 13
• run record sha256: 8befc900d89ce0d2db94675e16815847ba008f8985c3217c5349e13e45646f50

Licence and attribution

• model licence: apache-2.0
• training data licences: CC BY, CC BY-SA, CC0 only, per row; registry check 0 of 44870 built papers excluded
• starts from: sentence-transformers/all-MiniLM-L6-v2; that model's own licence and card apply