--- license: apache-2.0 library_name: onnx tags: - bibr - statement-block-classifier --- # statement-tagger `statement-tagger` is a model for the bibr stage `post_parse/integrity statements (proposed)`. Input: tokenizer pair (previous block's first 200 characters, block text), max 192 tokens; a block is a layout region on pages 1-2 or a text-layer line group elsewhere (as bibr extracts them). ## Files | File | Bytes | sha256 (first 16) | |---|---|---| | bibr_model.json | 4,442 | 191a8babda3e3026 | | onnx/bibr_onnx.json | 2,550 | 37eb7225f10156c4 | | onnx/model.onnx | 90,411,056 | 24e0fdb7b0841701 | | onnx/tokenizer.json | 527,292 | 325e6539e0908116 | The full checksums are in `checksums.json`. ## Architecture - encoder: sentence-transformers/all-MiniLM-L6-v2 - head: masked mean pool + linear, 7 sigmoid outputs (coi, funding, ethics, data_availability, author_contributions, acknowledgment, supplementary); per-type thresholds fit on PDF-view dev - input: tokenizer pair (previous block's first 200 characters, block text), max 192 tokens; a block is a layout region on pages 1-2 or a text-layer line group elsewhere (as bibr extracts them) - output: adjacent blocks of a type are joined into one statement (statements/join.py) - format: ONNX fp32 opset 17 (90 MB), parity with torch 6e-8 ## Training data - rows: 922468 train rows (400000 JATS-view sample + 174156 PDF-view rows x3) - jats view: 44869 PMC papers (train 39580, dev 2154, test 2244, test_journal 891), 1.71M block rows; positives from JATS statement markup (sec-type, fn-type, ack, funding-statement, headings in 5 languages), sampled and hard negatives, ambiguous blocks never trained; at most 300 train / 25 eval papers per journal - pdf view: 5165 papers extracted by bibr (train 4518, dev 243, test 256, test_journal 148); JATS statements projected onto PDF blocks by word 3-gram containment (recovery 94-98 percent per type, supplementary 46 percent) - languages: English plus 448 non-English train papers (de, fr, zh, pt, es, ru, ko, pl) - licences: CC BY, CC BY-SA, CC0 only, per row; registry check 0 of 44870 built papers excluded Every evaluation set in the exclusion registry was kept out of training. Registry: exclusions, build manifest sha256 d8d427de3ea7b954. Split: corpus-split-v1, sha256 c6fc246571f435d7. ## Evaluation | Set | Metric | Value | Shipped | Difference | Interval | Notes | |---|---|---|---|---|---|---| | silver paired PDF view (398) | text_f1_macro | 0.908 | | | | vs: JATS-trained arm 0.835, rules 0.741 | | silver paired JATS view (398) | text_f1_macro | 0.953 | | | | | | silver PDF test (256) | block_f1_macro | 0.861 | | | | | | silver PDF test_journal (148) | block_f1_macro | 0.825 | | | | | | statements gold (97 papers, 262 direct statements) | found_pooled | 0.908 | | | 95% [0.868, 0.943] | spurious statements not judged (no text layer); vs_bibr_0.8.0_full: 0.744; vs_rules: 0.844 | | Metacheck psychsci COI (247) | found | 0.984 | | | 95% [0.968, 0.996] | vs_rtransparent: 0.729; vs_bibr_full_12.2: 0.891 | | Metacheck psychsci funding (247, 192 with funding) | found | 0.833 | | | 95% [0.781, 0.883] | vs_rtransparent: 0.542; vs_bibr_full_12.2: 0.677 | | Metacheck MDPI COI (249, GROBID-view rows) | found | 0.855 | | | 95% [0.807, 0.9] | vs_rtransparent: 0.876 | | Metacheck economics COI (249, 68 with COI, GROBID-view rows) | found | 0.75 | | | 95% [0.643, 0.842] | worse than the rules here; vs_rtransparent: 0.853 | | gate192, whole pipeline (65 papers with acknowledgment gold) | sec.type:acknowledgment:presence | 0.857 | 0.696 | +0.161 | difference 95% [0.071, 0.259] | 9 papers better, none worse | | gate192, whole pipeline (54 papers with author-contribution gold) | sec.type:author_contributions:presence | 0.9 | 0.7 | +0.2 | difference 95% [0.098, 0.314] | 10 papers better, none worse | | gate192, whole pipeline (99 papers with discussion gold) | sec.type:discussion:unigram_precision | 0.747 | 0.737 | +0.010 | difference 95% [0.003, 0.019] | 17 papers better, none worse | | gate192, whole pipeline (60 papers with heading gold) | header.detect | 0.851 | 0.856 | -0.005 | difference 95% [-0.008, -0.003] | 12 papers one false heading worse, none better: the synthetic headings of new statement sections | | gate192 and audience_eval200, whole pipeline (190 and 198 papers) | core fields (title, authors, abstract, doi, abstained) | identical | identical | | | no front-matter field differs on any paper | ## Known weaknesses - the validation scorer judges no spurious statement without a text layer, so the gold numbers are recall-like; precision comes from the silver sets and the Metacheck cohorts only - silver labels are JATS markup; ethics paragraphs inside methods and untagged statements are ambiguous and left out of training and of precision - supplementary material statements are rarely printed as such (projection recovery 46 percent); weakest type - running heads and repeated footers make up 9-13 percent of PDF blocks; JATS-only training typed some as acknowledgment - CPU cost 2.9-3.6 ms per block at batch 1 and 4.1-4.3 ms at batch 32 on 4 pinned threads (512 PDF test blocks, tokenisation included; host load 9-11, so not an idle machine); about 300 blocks per paper, about 1 s per paper; 7-17 ms per block under heavier shared load - the non-English test slice is small (31 paired papers) - on Metacheck's economics COI cohort the rtransparent rules find more statements (0.853 vs 0.750); the cohort is GROBID text, a view the tagger never trained on - psychsci and the statements gold may have been seen while bibr's own rules were developed; the export heading filter was fixed after a first look at the gold results - in the pipeline a statement with no heading of its own gets a new section with a heading bibr supplies ("Author contributions", "Acknowledgments"). Since export schema 12.4 the export marks such a heading (extraction.diagnostics.section_classification[].header_supplied), but it does not carry the printed run-in label instead (most such papers print one, e.g. "Author Contributions:"). Heading detection reads 0.5 points lower on gate192 for this reason (12 of 60 papers, one heading each), measured with a scorer that does not yet leave the marked headings out ## How it was made - trainer script: scripts/train_statement_classifier.py - data builder: scripts/build_statements_dataset.py (jats, pdf, manifest, ids) - export script: scripts/export_statement_classifier_onnx.py - recipe: lr 5e-5 encoder / 1e-3 head, batch 64, bf16, BCE, 2 epochs (best = epoch 1 by PDF-view dev macro block F1), length-grouped batches; PDF-view rows repeated 3x - commit: 8521e69f6bb540e13e87fa27f785fa518477ed2e - seed: 13 - run record sha256: 8befc900d89ce0d2db94675e16815847ba008f8985c3217c5349e13e45646f50 ## Licence and attribution - model licence: apache-2.0 - training data licences: CC BY, CC BY-SA, CC0 only, per row; registry check 0 of 44870 built papers excluded - starts from: sentence-transformers/all-MiniLM-L6-v2; that model's own licence and card apply