Hugging Face's logo

scienceverse
/
bibr-statement-tagger-v1

ONNX
bibr
statement-block-classifier
Model card Files
xet
 Community
bibr-statement-tagger-v1 / README.md
thesanogoeffect's picture
thesanogoeffect
Correct the heading weakness: export schema 12.4 marks supplied headings
b446f94 verified
preview code
|
Raw History Blame Contribute Delete
7.08 kB
metadata 
license: apache-2.0
library_name: onnx
tags:
  - bibr
  - statement-block-classifier

statement-tagger

statement-tagger is a model for the bibr stage post_parse/integrity statements (proposed). Input: tokenizer pair (previous block's first 200 characters, block text), max 192 tokens; a block is a layout region on pages 1-2 or a text-layer line group elsewhere (as bibr extracts them).

Files

File Bytes sha256 (first 16)
bibr_model.json 4,442 191a8babda3e3026
onnx/bibr_onnx.json 2,550 37eb7225f10156c4
onnx/model.onnx 90,411,056 24e0fdb7b0841701
onnx/tokenizer.json 527,292 325e6539e0908116

The full checksums are in checksums.json.

Architecture

  • encoder: sentence-transformers/all-MiniLM-L6-v2
  • head: masked mean pool + linear, 7 sigmoid outputs (coi, funding, ethics, data_availability, author_contributions, acknowledgment, supplementary); per-type thresholds fit on PDF-view dev
  • input: tokenizer pair (previous block's first 200 characters, block text), max 192 tokens; a block is a layout region on pages 1-2 or a text-layer line group elsewhere (as bibr extracts them)
  • output: adjacent blocks of a type are joined into one statement (statements/join.py)
  • format: ONNX fp32 opset 17 (90 MB), parity with torch 6e-8

Training data

  • rows: 922468 train rows (400000 JATS-view sample + 174156 PDF-view rows x3)
  • jats view: 44869 PMC papers (train 39580, dev 2154, test 2244, test_journal 891), 1.71M block rows; positives from JATS statement markup (sec-type, fn-type, ack, funding-statement, headings in 5 languages), sampled and hard negatives, ambiguous blocks never trained; at most 300 train / 25 eval papers per journal
  • pdf view: 5165 papers extracted by bibr (train 4518, dev 243, test 256, test_journal 148); JATS statements projected onto PDF blocks by word 3-gram containment (recovery 94-98 percent per type, supplementary 46 percent)
  • languages: English plus 448 non-English train papers (de, fr, zh, pt, es, ru, ko, pl)
  • licences: CC BY, CC BY-SA, CC0 only, per row; registry check 0 of 44870 built papers excluded

Every evaluation set in the exclusion registry was kept out of training. Registry: exclusions, build manifest sha256 d8d427de3ea7b954. Split: corpus-split-v1, sha256 c6fc246571f435d7.

Evaluation

Set Metric Value Shipped Difference Interval Notes
silver paired PDF view (398) text_f1_macro 0.908 vs: JATS-trained arm 0.835, rules 0.741
silver paired JATS view (398) text_f1_macro 0.953
silver PDF test (256) block_f1_macro 0.861
silver PDF test_journal (148) block_f1_macro 0.825
statements gold (97 papers, 262 direct statements) found_pooled 0.908 95% [0.868, 0.943] spurious statements not judged (no text layer); vs_bibr_0.8.0_full: 0.744; vs_rules: 0.844
Metacheck psychsci COI (247) found 0.984 95% [0.968, 0.996] vs_rtransparent: 0.729; vs_bibr_full_12.2: 0.891
Metacheck psychsci funding (247, 192 with funding) found 0.833 95% [0.781, 0.883] vs_rtransparent: 0.542; vs_bibr_full_12.2: 0.677
Metacheck MDPI COI (249, GROBID-view rows) found 0.855 95% [0.807, 0.9] vs_rtransparent: 0.876
Metacheck economics COI (249, 68 with COI, GROBID-view rows) found 0.75 95% [0.643, 0.842] worse than the rules here; vs_rtransparent: 0.853
gate192, whole pipeline (65 papers with acknowledgment gold) sec.type:acknowledgment:presence 0.857 0.696 +0.161 difference 95% [0.071, 0.259] 9 papers better, none worse
gate192, whole pipeline (54 papers with author-contribution gold) sec.type:author_contributions:presence 0.9 0.7 +0.2 difference 95% [0.098, 0.314] 10 papers better, none worse
gate192, whole pipeline (99 papers with discussion gold) sec.type:discussion:unigram_precision 0.747 0.737 +0.010 difference 95% [0.003, 0.019] 17 papers better, none worse
gate192, whole pipeline (60 papers with heading gold) header.detect 0.851 0.856 -0.005 difference 95% [-0.008, -0.003] 12 papers one false heading worse, none better: the synthetic headings of new statement sections
gate192 and audience_eval200, whole pipeline (190 and 198 papers) core fields (title, authors, abstract, doi, abstained) identical identical no front-matter field differs on any paper

Known weaknesses

  • the validation scorer judges no spurious statement without a text layer, so the gold numbers are recall-like; precision comes from the silver sets and the Metacheck cohorts only
  • silver labels are JATS markup; ethics paragraphs inside methods and untagged statements are ambiguous and left out of training and of precision
  • supplementary material statements are rarely printed as such (projection recovery 46 percent); weakest type
  • running heads and repeated footers make up 9-13 percent of PDF blocks; JATS-only training typed some as acknowledgment
  • CPU cost 2.9-3.6 ms per block at batch 1 and 4.1-4.3 ms at batch 32 on 4 pinned threads (512 PDF test blocks, tokenisation included; host load 9-11, so not an idle machine); about 300 blocks per paper, about 1 s per paper; 7-17 ms per block under heavier shared load
  • the non-English test slice is small (31 paired papers)
  • on Metacheck's economics COI cohort the rtransparent rules find more statements (0.853 vs 0.750); the cohort is GROBID text, a view the tagger never trained on
  • psychsci and the statements gold may have been seen while bibr's own rules were developed; the export heading filter was fixed after a first look at the gold results
  • in the pipeline a statement with no heading of its own gets a new section with a heading bibr supplies ("Author contributions", "Acknowledgments"). Since export schema 12.4 the export marks such a heading (extraction.diagnostics.section_classification[].header_supplied), but it does not carry the printed run-in label instead (most such papers print one, e.g. "Author Contributions:"). Heading detection reads 0.5 points lower on gate192 for this reason (12 of 60 papers, one heading each), measured with a scorer that does not yet leave the marked headings out

How it was made

  • trainer script: scripts/train_statement_classifier.py
  • data builder: scripts/build_statements_dataset.py (jats, pdf, manifest, ids)
  • export script: scripts/export_statement_classifier_onnx.py
  • recipe: lr 5e-5 encoder / 1e-3 head, batch 64, bf16, BCE, 2 epochs (best = epoch 1 by PDF-view dev macro block F1), length-grouped batches; PDF-view rows repeated 3x
  • commit: 8521e69f6bb540e13e87fa27f785fa518477ed2e
  • seed: 13
  • run record sha256: 8befc900d89ce0d2db94675e16815847ba008f8985c3217c5349e13e45646f50

Licence and attribution

  • model licence: apache-2.0
  • training data licences: CC BY, CC BY-SA, CC0 only, per row; registry check 0 of 44870 built papers excluded
  • starts from: sentence-transformers/all-MiniLM-L6-v2; that model's own licence and card apply