| --- |
| license: apache-2.0 |
| library_name: onnx |
| tags: |
| - bibr |
| - statement-block-classifier |
| --- |
| |
| # statement-tagger |
|
|
| `statement-tagger` is a model for the bibr stage `post_parse/integrity statements (proposed)`. Input: tokenizer pair (previous block's first 200 characters, block text), max 192 tokens; a block is a layout region on pages 1-2 or a text-layer line group elsewhere (as bibr extracts them). |
|
|
| ## Files |
|
|
| | File | Bytes | sha256 (first 16) | |
| |---|---|---| |
| | bibr_model.json | 4,442 | 191a8babda3e3026 | |
| | onnx/bibr_onnx.json | 2,550 | 37eb7225f10156c4 | |
| | onnx/model.onnx | 90,411,056 | 24e0fdb7b0841701 | |
| | onnx/tokenizer.json | 527,292 | 325e6539e0908116 | |
|
|
| The full checksums are in `checksums.json`. |
|
|
| ## Architecture |
|
|
| - encoder: sentence-transformers/all-MiniLM-L6-v2 |
| - head: masked mean pool + linear, 7 sigmoid outputs (coi, funding, ethics, data_availability, author_contributions, acknowledgment, supplementary); per-type thresholds fit on PDF-view dev |
| - input: tokenizer pair (previous block's first 200 characters, block text), max 192 tokens; a block is a layout region on pages 1-2 or a text-layer line group elsewhere (as bibr extracts them) |
| - output: adjacent blocks of a type are joined into one statement (statements/join.py) |
| - format: ONNX fp32 opset 17 (90 MB), parity with torch 6e-8 |
|
|
| ## Training data |
|
|
| - rows: 922468 train rows (400000 JATS-view sample + 174156 PDF-view rows x3) |
| - jats view: 44869 PMC papers (train 39580, dev 2154, test 2244, test_journal 891), 1.71M block rows; positives from JATS statement markup (sec-type, fn-type, ack, funding-statement, headings in 5 languages), sampled and hard negatives, ambiguous blocks never trained; at most 300 train / 25 eval papers per journal |
| - pdf view: 5165 papers extracted by bibr (train 4518, dev 243, test 256, test_journal 148); JATS statements projected onto PDF blocks by word 3-gram containment (recovery 94-98 percent per type, supplementary 46 percent) |
| - languages: English plus 448 non-English train papers (de, fr, zh, pt, es, ru, ko, pl) |
| - licences: CC BY, CC BY-SA, CC0 only, per row; registry check 0 of 44870 built papers excluded |
|
|
| Every evaluation set in the exclusion registry was kept out of training. Registry: exclusions, build manifest sha256 d8d427de3ea7b954. Split: corpus-split-v1, sha256 c6fc246571f435d7. |
|
|
| ## Evaluation |
|
|
| | Set | Metric | Value | Shipped | Difference | Interval | Notes | |
| |---|---|---|---|---|---|---| |
| | silver paired PDF view (398) | text_f1_macro | 0.908 | | | | vs: JATS-trained arm 0.835, rules 0.741 | |
| | silver paired JATS view (398) | text_f1_macro | 0.953 | | | | | |
| | silver PDF test (256) | block_f1_macro | 0.861 | | | | | |
| | silver PDF test_journal (148) | block_f1_macro | 0.825 | | | | | |
| | statements gold (97 papers, 262 direct statements) | found_pooled | 0.908 | | | 95% [0.868, 0.943] | spurious statements not judged (no text layer); vs_bibr_0.8.0_full: 0.744; vs_rules: 0.844 | |
| | Metacheck psychsci COI (247) | found | 0.984 | | | 95% [0.968, 0.996] | vs_rtransparent: 0.729; vs_bibr_full_12.2: 0.891 | |
| | Metacheck psychsci funding (247, 192 with funding) | found | 0.833 | | | 95% [0.781, 0.883] | vs_rtransparent: 0.542; vs_bibr_full_12.2: 0.677 | |
| | Metacheck MDPI COI (249, GROBID-view rows) | found | 0.855 | | | 95% [0.807, 0.9] | vs_rtransparent: 0.876 | |
| | Metacheck economics COI (249, 68 with COI, GROBID-view rows) | found | 0.75 | | | 95% [0.643, 0.842] | worse than the rules here; vs_rtransparent: 0.853 | |
| | gate192, whole pipeline (65 papers with acknowledgment gold) | sec.type:acknowledgment:presence | 0.857 | 0.696 | +0.161 | difference 95% [0.071, 0.259] | 9 papers better, none worse | |
| | gate192, whole pipeline (54 papers with author-contribution gold) | sec.type:author_contributions:presence | 0.9 | 0.7 | +0.2 | difference 95% [0.098, 0.314] | 10 papers better, none worse | |
| | gate192, whole pipeline (99 papers with discussion gold) | sec.type:discussion:unigram_precision | 0.747 | 0.737 | +0.010 | difference 95% [0.003, 0.019] | 17 papers better, none worse | |
| | gate192, whole pipeline (60 papers with heading gold) | header.detect | 0.851 | 0.856 | -0.005 | difference 95% [-0.008, -0.003] | 12 papers one false heading worse, none better: the synthetic headings of new statement sections | |
| | gate192 and audience_eval200, whole pipeline (190 and 198 papers) | core fields (title, authors, abstract, doi, abstained) | identical | identical | | | no front-matter field differs on any paper | |
| |
| ## Known weaknesses |
| |
| - the validation scorer judges no spurious statement without a text layer, so the gold numbers are recall-like; precision comes from the silver sets and the Metacheck cohorts only |
| - silver labels are JATS markup; ethics paragraphs inside methods and untagged statements are ambiguous and left out of training and of precision |
| - supplementary material statements are rarely printed as such (projection recovery 46 percent); weakest type |
| - running heads and repeated footers make up 9-13 percent of PDF blocks; JATS-only training typed some as acknowledgment |
| - CPU cost 2.9-3.6 ms per block at batch 1 and 4.1-4.3 ms at batch 32 on 4 pinned threads (512 PDF test blocks, tokenisation included; host load 9-11, so not an idle machine); about 300 blocks per paper, about 1 s per paper; 7-17 ms per block under heavier shared load |
| - the non-English test slice is small (31 paired papers) |
| - on Metacheck's economics COI cohort the rtransparent rules find more statements (0.853 vs 0.750); the cohort is GROBID text, a view the tagger never trained on |
| - psychsci and the statements gold may have been seen while bibr's own rules were developed; the export heading filter was fixed after a first look at the gold results |
| - in the pipeline a statement with no heading of its own gets a new section with a heading bibr supplies ("Author contributions", "Acknowledgments"). Since export schema 12.4 the export marks such a heading (extraction.diagnostics.section_classification[].header_supplied), but it does not carry the printed run-in label instead (most such papers print one, e.g. "Author Contributions:"). Heading detection reads 0.5 points lower on gate192 for this reason (12 of 60 papers, one heading each), measured with a scorer that does not yet leave the marked headings out |
| |
| ## How it was made |
| |
| - trainer script: scripts/train_statement_classifier.py |
| - data builder: scripts/build_statements_dataset.py (jats, pdf, manifest, ids) |
| - export script: scripts/export_statement_classifier_onnx.py |
| - recipe: lr 5e-5 encoder / 1e-3 head, batch 64, bf16, BCE, 2 epochs (best = epoch 1 by PDF-view dev macro block F1), length-grouped batches; PDF-view rows repeated 3x |
| - commit: 8521e69f6bb540e13e87fa27f785fa518477ed2e |
| - seed: 13 |
| - run record sha256: 8befc900d89ce0d2db94675e16815847ba008f8985c3217c5349e13e45646f50 |
|
|
| ## Licence and attribution |
|
|
| - model licence: apache-2.0 |
| - training data licences: CC BY, CC BY-SA, CC0 only, per row; registry check 0 of 44870 built papers excluded |
| - starts from: sentence-transformers/all-MiniLM-L6-v2; that model's own licence and card apply |
|
|
