scienceverse/bibr-ref-line-tagger-v1

reference-segmenter-geometry-tagger

reference-segmenter-geometry-tagger is a model for the bibr stage extract/references. Input: the 29 bibr 0.9 geometry + text-pattern line features (sorted FEATURE_KEYS), the whole paper as one sequence (trained on 128-line chunks). bibr loads it through the setting REF_GEOM_TAGGER_MODEL_ID on the default, fast path.

Files

File	Bytes	sha256 (first 16)
bibr_model.json	6,119	2ab5147df2d8714f
gate.json	6,963	3902d750ec2d8bb0
onnx/tagger.json	2,290	3f3822418f7a6afd
onnx/tagger.onnx	2,410,636	fa3a2d07b46bd913

The full checksums are in checksums.json.

Architecture

• encoder: none
• head: per-paper line-sequence tagger, 2-layer bidirectional GRU (hidden 128) over a linear + GELU projection (256) of the 29 line features, B/I per printed line; feature normalisation inside the graph
• input: the 29 bibr 0.9 geometry + text-pattern line features (sorted FEATURE_KEYS), the whole paper as one sequence (trained on 128-line chunks)
• format: ONNX fp32 opset 17 (2.4 MB), onnxruntime only; parity with torch on PMC test (1,196 papers) max |p diff| 6.0e-7, 0 decision flips
• onnx contract: lines first, no batch axis: one paper per call; input features float32 [lines, 29] in feature_keys order; outputs start_prob float32 [lines] and logits float32 [lines, 2], labels [continuation, start]; written into onnx/tagger.json as layout and labels
• runtime parity: stored training features against bibr's runtime feature code 0 mismatches on 1.14 million lines (PMC proto subset, the gated rows, the arXiv rows); runtime features into tagger.onnx against the arm's stored eval logits 0 label flips and 0 gate flips on arXiv OOD (145 papers) and the first 1,500 PMC dev and test rows, largest probability difference 7.4e-4
• gate: temperature 1.551 on the line logits (dev NLL 0.01452 -> 0.01115); paper score expected_entry_precision mapped by isotonic regression (18 breakpoints) onto dev paper entry precision; threshold 0.9667 = the lowest calibrated level whose accepted dev papers reach the GBM's accepted dev precision at margin 0.90 (0.9936); all in gate.json

Training data

• rows: geometry rows v2, re-gated by the exclusion registry (25 papers out) and the corpus-split-v1 split (scripts/build_geom_split_rows.py); plus arXiv rows built here from the arXiv-source pilot (ar5iv reference strings re-projected onto bibr's own reference lines, scripts/build_refseg_probe_rows.py), train-split papers only
• papers: train: 23563 (21798 PMC + 1765 arXiv), dev 1,122, test 1,196, test_journal 812
• lines: train: 4082406 (3600978 PMC + 481428 arXiv), dev 188,683, test 193,004, test_journal 105,805
• labels: JATS / ar5iv reference strings projected onto bibr's reference lines; clean = match rate 1.0 (PMC); arXiv papers kept at the same match-rate gate (1,765 of 2,329 probed)
• languages: mostly English
• licences: PMC (CC BY 21,668, CC0 106, CC BY-SA 24), arXiv (CC BY 1,523, CC BY-SA 136, CC0 106)

Every evaluation set in the exclusion registry was kept out of training. Registry: exclusions, build manifest sha256 d8d427de3ea7b954. Split: corpus-split-v1, sha256 c6fc246571f435d7.

Evaluation

Set	Metric	Value	Shipped	Difference	Interval	Notes
PMC dev (1122)	entry_f1	0.9956	0.9737	+0.0219	95% [0.9901, 0.9986]	vs_gbm_same_rows: 0.9919
PMC test (1196)	entry_f1	0.9987	0.9751	+0.0236	95% [0.9982, 0.9991]	vs_gbm_same_rows: 0.9953
PMC test_journal (812)	entry_f1	0.9946	0.986	+0.0086	95% [0.9906, 0.9977]	vs_gbm_same_rows: 0.9905
gold86 (evaluation only)	entry_f1	0.9987	0.9966	+0.0021	95% [0.9969, 1.0]	shipped model in-sample; vs_gbm_same_rows: 0.9841
shard00 held out (1146)	entry_f1	0.9586	0.9015	+0.0571		vs_gbm_same_rows: 0.9542
shard01 held out (989)	entry_f1	0.9789	0.9412	+0.0377		vs_gbm_same_rows: 0.9742
arXiv pilot dev/test papers, OOD probe (145)	entry_f1	0.9836	0.8742	+0.1094	95% [0.9704, 0.994]	vs_gbm_same_rows: 0.9066; vs_gbm_with_arxiv: 0.9582
OpenAlex/Crossref probe (119)	entry_f1	0.9899	0.98	+0.0099	95% [0.9824, 0.9951]	vs_gbm_same_rows: 0.9849
PMC dev gate	coverage / accepted precision	0.9795, 0.9976				gbm_margin_0.90: 0.9982, 0.9936
PMC test gate	coverage / accepted precision	0.9816, 0.9988				gbm_margin_0.90: 0.9983, 0.9955
PMC test_journal gate	coverage / accepted precision	0.9655, 0.9989				gbm_margin_0.90: 0.9951, 0.9957
EXCITE dev tagger-first replay, floor 0.9 (162)	clean_in_text	0.9434			difference 95% [-0.004, 0.0036]	vs_recorded: 0.9432; papers_better: 7; papers_worse: 5
same, papers the shipped GBM declined (54)	clean_in_text	0.8962			difference 95% [-0.0175, 0.0]	vs_recorded: 0.9016; papers_worse: 1
EXCITE dev tagger-first replay, floor 0.95 (162)	clean_in_text	0.9449			difference 95% [-0.0003, 0.0042]	vs_recorded: 0.9432; papers_better: 7; papers_worse: 4
same, papers the shipped GBM declined (54)	clean_in_text	0.9016				vs_recorded: 0.9016; papers_worse: 0
EXCITE dev replace replay (not the runtime), floor 0.9 (162)	clean_in_text	0.9515			difference 95% [-0.0017, 0.0188]	vs_recorded: 0.9432
same, papers the shipped GBM declined (54)	clean_in_text	0.8962				vs_recorded: 0.9016; papers_worse: 1
EXCITE dev replace replay (not the runtime), floor 0.95 (162)	clean_in_text	0.953			difference 95% [0.0003, 0.0198]	vs_recorded: 0.9432; papers_better: 22; papers_worse: 9
same, papers the shipped GBM declined (54)	clean_in_text	0.9016				vs_recorded: 0.9016; papers_worse: 0
same replace replay, de (112) / en (50)	clean_in_text	0.9353, 0.9838				vs_recorded: 0.9237, 0.9771
audience_eval200 tagger-first replay, floor 0.9 (200)	key_f1	0.9649			difference 95% [0.0002, 0.0017]	vs_recorded: 0.964; papers_better: 12; papers_worse: 4
same, papers the shipped GBM declined (34; 25 with geometry)	key_f1	0.9147				vs_recorded: 0.9147; papers_worse: 0
audience_eval200 tagger-first replay, floor 0.95 (200)	key_f1	0.9648			difference 95% [0.0002, 0.0016]	vs_recorded: 0.964; papers_better: 11; papers_worse: 4
same, papers the shipped GBM declined (34)	key_f1	0.9147				vs_recorded: 0.9147; papers_worse: 0
audience_eval200 replace replay (not the runtime), floor 0.9 and 0.95 (200)	key_f1	0.9657			difference 95% [-0.0003, 0.0036]	vs_recorded: 0.964; papers_better: 27; papers_worse: 8
audience_eval200 coverage, papers with geometry (191)	geom tier accepted	tagger_floor_0.9 139, tagger_floor_0.95 138, gbm 166
scans_eval80 (80; 7 with geometry, scanned PDFs)	key_f1	0.8999				uninformative, both floors identical; vs_recorded: 0.8999
audience 60-paper screening set, whole pipeline	reference entries	0.9235	0.9232	+0.0003	difference 95% [0.0, 0.0012]	asked 56, accepted 40 (final 40, replaced 0, errors 0), declined 16 (low_confidence_or_empty 14, low_alignment_yield 1, low_segment_count 1); gate confidence min 0.672, median 0.981; author +0.0013, volume +0.0018, no paper worse on any reference score; front matter identical; papers_better: 1; papers_worse: 0
EXCITE dev, low-confidence declines (49)	clean_in_text	0.9552			difference 95% [0.0, 0.0658]	vs_runtime: 0.9229; papers_better: 14; papers_worse: 9; all_pass_tier_checks: yes
audience_eval200, low-confidence declines (45)	key_f1	0.936			difference 95% [-0.0214, 0.0062]	vs_runtime: 0.9417; papers_better: 14; papers_worse: 15; all_pass_tier_checks: yes
tagger tier alone, audience_eval200 features (195 papers, 217 lines mean, 7,496 max), onnxruntime	ms_per_paper	4.1				graph only, features precomputed; best of 3, host load average about 25; threads: 1; four_threads: 2.8
CPU time, PMC test (1196 papers, 161 lines mean), onnxruntime	ms_per_paper	1.7				threads: 1; four_threads: 2.0

Known weaknesses

• the replace replay's EXCITE gain (+0.008 / +0.010) came from the 35 papers the tagger declines where the GBM accepted, which it gave an S1 stand-in; the runtime sends those papers to the GBM, which keeps its recorded answer, so under the runtime's semantics the gain is +0.0002 (floor 0.9) and +0.0017 (floor 0.95), both n.s.
• neither replay re-runs the line-stream choice bibr makes after segmentation
• the 0.95 floor (shipped 0.9: spans >= 0.9 x layout reference onsets when >= 90% of onsets align) removes the one GBM-declined paper the tagger made worse on EXCITE dev (44 spans for 47 aligned onsets), where it was chosen; on the second set (audience_eval200) it changes nothing on the GBM-declined papers and declines one more paper overall (+0.0008 against +0.0009), so it does not hold: the candidate ships with the shipped 0.9 floor
• looks per set: EXCITE dev several times (round-1 replays and floor/threshold sweeps, the runtime and runtime2 replays; the 0.95 floor and 0.9667 threshold were looked at there); audience_eval200 twice (replay before and after the line-stream status fix 731b4ba; never tuned on here or by the geometry segmenter work); scans_eval80 once for the tagger (plus a counts-only plumbing dry run)
• replay imports bibr at cb22d3d (pull request 213) for the GBM features and geom post-processing, and bibr at 724cdb47 for the tagger features; runtime semantics 2 and 3 are implemented in this repository's replay (tagger_first arm), not imported from bibr
• threshold: chosen on PMC dev only (the lowest calibrated level whose accepted dev papers reach the GBM's accepted dev precision at margin 0.90); the isotonic map is coarse, and 91 of the 94 low-confidence declines in the ungated diagnostic sit on its 0.672 level (the other 3 at 0.80-0.88), so no threshold between 0.672 and 0.9667 separates them. Accepted ungated, those papers gain on EXCITE dev (+0.032, interval touching 0) and lose on audience_eval200 (-0.006, n.s.): no case for moving the threshold; a finer calibration (more dev papers at the low levels, or a score with more spread) is the place to work. Not retuned; the 60-paper screening set was not used
• the gain under the runtime's semantics comes from papers both accept where the tagger's answer is better (12 better, 4 worse on audience_eval200); the tagger accepts fewer papers than the GBM (139 against 166 of 191) and the GBM keeps those
• PMC dev coverage 0.98 against the GBM's 0.998: the tagger hands about 2% of PMC papers to the fallback
• isotonic levels are coarse (next level below the threshold is 0.672), so the threshold moves in steps; a 0.998 precision target gives dev coverage 0.86
• test_journal is 0.0018 below the PMC-only gru128 arm (0.9964); within the intervals
• training rows are PMC + arXiv (English); EXCITE German papers are out of domain
• CPU timings were taken on cores shared with other training jobs

How it was made

• trainer script: scripts/refseg_line_tagger_train.py train --arm gru128_arxiv=gru,128,all,clean+ext_arxivtrain (gate scripts/refseg_line_tagger_gate.py, export scripts/refseg_line_tagger_onnx.py)
• recipe: optimiser: AdamW (weight decay 0.01), one-cycle schedule, lr 0.002, dropout 0.1, epochs 8, chunk: 128 lines at a random phase, max_lines 8,192, seed 0, selection: dev entry F1 over all arms (gru128_arxiv 0.9956, gru128 0.9955, gru128_noisy 0.9953), wall: 56 s on one shared RTX 5090
• commit: 8c975b95c1a136957df82a5f72300ab11d1f29b1
• seed: 0
• run record sha256: f0bcd8750673c7b64782b492e2bd8a26d10d4edac4d0312cecc9236ad370c03a

Licence and attribution

• model licence: mit
• training data licences: PMC (CC BY 21,668, CC0 106, CC BY-SA 24), arXiv (CC BY 1,523, CC BY-SA 136, CC0 106)