| --- |
| license: mit |
| library_name: onnx |
| tags: |
| - bibr |
| - reference-segmentation-text-fallback |
| --- |
| |
| # reference-segmenter-text-mmbert |
|
|
| `reference-segmenter-text-mmbert` is a model for the bibr stage `extract/references`. Input: the reference-section text joined as bibr joins it at runtime, add_special_tokens false, windows of 1,024 tokens, stride 768, trust merge across windows. bibr loads it through the setting `REF_TEXT_SEG_MODEL_ID` on the text fallback tier path. It replaces `reference-segmenter-crf` (scienceverse/bibr-segmenter-v1 at 344f27d851ffb901b2659629ac879f10b6eba234). |
|
|
| ## Files |
|
|
| | File | Bytes | sha256 (first 16) | |
| |---|---|---| |
| | bibr_model.json | 4,909 | 00b6d3b87e97394b | |
| | onnx/model.onnx | 268,620,732 | a9a5f0ce3ff8f398 | |
| | onnx/segmenter.json | 500 | 964470334fc8f5a6 | |
| | onnx/tokenizer.json | 34,363,188 | 609d8f4c067cd395 | |
| |
| The full checksums are in `checksums.json`. |
| |
| ## Architecture |
| |
| - encoder: jhu-clsp/mmBERT-small (22 layers x 384, multilingual) |
| - head: linear token tagger O / B-REF / I-REF, no CRF |
| - input: the reference-section text joined as bibr joins it at runtime, add_special_tokens false, windows of 1,024 tokens, stride 768, trust merge across windows |
| - decode: argmax per token; a reference opens at each B-REF token and runs to the next one; the first reference is forced open at the first non-space character |
| - format: ONNX opset 17 with int8 token embeddings (268,620,732 bytes; one scale per row, Gather -> Cast -> Mul, the parser's method in scripts/ref_llm/export_parser_onnx_v6.py; every other weight float32; max dequantisation error 0.0013), made by `refseg_text_fallback_train.py quantize` from the fp32 export (562,508,070 bytes); parity with torch on the same 40 EXCITE papers, max |logit diff| 0.068 (fp32 1.1e-5), 0 tag flips, identical segments on 40 of 40 |
| - runtime deps: onnxruntime + tokenizers (`predict-onnx`; no torch) |
| - runtime status: bibr integration 550b2336 has the ONNX runtime (REF_TEXT_SEG_MODEL_ID); its decode never declines (decode.when_nothing_found first_unit_forced); see the decode-variant rows |
| - onnx contract: files: model.onnx, tokenizer.json (HF tokenizers, mmBERT-small's), segmenter.json (tags, window, stride, add_special_tokens, decode, when_nothing_found, model_file, tokenizer_file, files); bibr_model.json in the normalised bundle, tokenize: tokenizers.Tokenizer.from_file(tokenizer.json).encode(text, add_special_tokens=False); keep ids and character offsets; text = the reference-section text as bibr joins it, inputs: input_ids int64 [batch, tokens], attention_mask int64 [batch, tokens] (all ones; one window per call, batch 1), output: logits float32 [batch, tokens, 3]; label order 0 O, 1 B-REF, 2 I-REF, windows: token windows [start, min(start + 1024, n)), start = 0, 768, 1536, ... until a window reaches n; no special tokens, no padding, merge: per window argmax; token i of a window of length L has trust min(i, L - 1 - i); each token keeps the tag from the window where its trust is highest (the first such window on a tie), decode: a reference starts at the first non-space character of every B-REF token (in order, strictly increasing); the first non-space character of the text is always a start; each reference runs to the next start; strip each, drop empty ones; O and I-REF never start one, reference_code: scripts/refseg_text_fallback_train.py (windows, tags_sliding, decode_starts, segments_from_tags, cmd_predict_onnx) |
|
|
| ## Training data |
|
|
| - rows: data/text_v1 from the registry- and split-gated geom rows v2 (clean papers only, match rate 1.0) |
| - papers: train 21,798, dev 1,122, test 1,196, test_journal 812 |
| - references: train 994,664, dev 51,845, test 53,113, test_journal 32,325 |
| - tokens: 113012269 train tokens in 118,517 windows |
| - labels: JATS reference strings projected onto bibr's own reference lines |
| - languages: PMC, mostly English |
| - licences: CC BY 21,668, CC0 106, CC BY-SA 24 |
| - text modes train: region 12,962, lines 5,515, merged 3,321 |
| |
| Every evaluation set in the exclusion registry was kept out of training. Registry: exclusions, build manifest sha256 d8d427de3ea7b954. Split: corpus-split-v1, sha256 c6fc246571f435d7. |
| |
| ## Evaluation |
| |
| | Set | Metric | Value | Shipped | Difference | Interval | Notes | |
| |---|---|---|---|---|---|---| |
| | PMC dev text (1122) | entry_f1 | 0.9955 | | | | | |
| | PMC test text (1196) | entry_f1 | 0.998 | | | | | |
| | PMC test_journal text (812) | entry_f1 | 0.9983 | | | | | |
| | EXCITE dev standalone (162) | clean_in_text | 0.9402 | | | | vs_s1: 0.9353; vs_v1: 0.8034; diff_vs_s1_ci95: -0.0123, 0.0206 | |
| | EXCITE dev geometry-declined (54) | clean_in_text | 0.8775 | | | | vs_s1: 0.8739; vs_v1: 0.6825; diff_vs_s1_ci95: -0.0443, 0.052; vs_recorded_cascade: 0.9016 | |
| | scans proxy (78) | key_f1 | 0.8881 | | | | vs_s1: 0.8686; vs_v1: 0.837 | |
| | gold val no-leak (167 items) | exact_f1 | 0.9024 | | | | vs_s1: 0.9424; vs_v1: 0.9036; diff_vs_s1_ci95: -0.0701, -0.0151; diff_vs_v1_ci95: -0.0239, 0.0163 | |
| | gold val no-leak, mdpi_socsci100 (48) | exact_f1 | 0.8315 | | | | vs_s1: 0.9575; iou50_f1: 0.9812 | |
| | EXCITE dev standalone (162) | clean_in_text | 0.9402 | | | | file: int8; vs_fp32: 0.9402; diff_vs_s1_ci95: -0.0123, 0.0206; papers_differing_from_fp32_onnx: 0 | |
| | EXCITE dev geometry-declined (54) | clean_in_text | 0.8775 | | | | file: int8; vs_fp32: 0.8775; diff_vs_s1_ci95: -0.0443, 0.052 | |
| | scans proxy (78) | key_f1 | 0.888 | | | | file: int8; vs_fp32_onnx: 0.8877; vs_fp32_torch: 0.8881; papers_differing_from_fp32_onnx: 1 | |
| | CPU time, 60 EXCITE papers (74,874 tokens), onnxruntime, cores 0-3 | s_per_paper | | | | | median of 3 alternating rounds; host load average 6.6-11.5, cpu0-3 busy 78-86% during the runs, our run included; torch CPU 4 threads 4.48, S1 torch CPU 6.60 were timed earlier under heavier load; int8: four_threads 0.252, one_thread 0.747; fp32: four_threads 0.26, one_thread 0.751 | |
| | audience_eval200, whole pipeline, only tier against default cascade (198 papers) | refs.entries | 0.983 | 0.983 | +0.000 | difference 95% [-0.002, 0.004] | 24 papers better, 19 worse | |
| | audience_eval200, whole pipeline, only tier against default cascade (198 papers) | refs.title | 0.979 | 0.976 | +0.003 | difference 95% [0.0, 0.005] | 19 papers better, 6 worse | |
| | audience_eval200, whole pipeline, only tier against default cascade (198 papers) | refs.title/gold | 0.959 | 0.953 | +0.006 | difference 95% [-0.0, 0.011] | 27 papers better, 7 worse | |
| | audience_eval200, whole pipeline, only tier against default cascade (198 papers) | core.pass_floors | 0.889 | 0.899 | -0.010 | difference 95% [-0.025, 0.0] | 2 papers worse, none better: a headed list of 11 references with no reference start found (one block), and 14 of 21 entries where the line stream finds 21 | |
| |
| ## Known weaknesses |
| |
| - the shipped CRF segmenter (S1) leads on gold val mostly through mdpi_socsci100, where both find the same references (IoU-0.5 F1 0.981 vs 0.984) but this model places the stray glyph fragments between references at the next reference's start instead of the previous one's end; S1 was trained on that gold's convention (the gold cohorts are evaluation-only here) |
| - synthetic_val exact F1 0.31 (S1 0.999, trained on that format); IoU-0.5 F1 0.91 |
| - PMC-only training; EXCITE (German) and scans are out of domain |
| - int8 token embeddings cut the file from 563 MB to 269 MB (the 34 MB tokenizer.json comes on top); the rest of the graph is fp32, so speed is unchanged (0.25 s per paper at 4 threads either way) |
| - timings were taken on CPU cores shared with other training jobs |
| - how often each set was looked at: EXCITE dev three times (fp32, int8, decode variants), scans proxy three times (fp32, int8, decode variants), PMC dev twice (selection, decode variants), gold val once; no choice was made on the int8 or decode-variant numbers |
| - the model never tags O on EXCITE dev (0 of 225,336 tokens) and rarely on scans (1,252 of 88,140) or PMC dev (O on 26,574 of 10.46 million non-space characters inside gold references, 0.25%): its PMC rows hold reference text only, so it cannot mark non-reference text, and an O-based decode changes nothing; rows with page furniture and headers labelled O would be needed for that |
| - lists with zero B-REF: EXCITE dev 4 of 162 (13 of 4,065 in-text gold refs, 0 found by the forced first reference), scans 4 of 78 (28 of 1,647 gold keys), PMC dev 0 of 1,122; marker_split finds nothing on any of them |
| - the tier answered for a list bibr had only guessed (REF_SECTION_INFERRED, the last unclassified section) and in which it found no reference start: the section came back as one reference, and that one entry kept the citations in the paper's notes from standing in for the list. Seen on 1 of 198 audience_eval200 papers (nine of ten references lost against the shipped CRF, which declines) and on none of 190 gate192 papers. bibr branch fix/ref-not-split-inferred-list (ec89c5b3) declines such a list; a list found by its heading keeps the one-block answer. This corrects the decision recorded under decode_variants, which was measured on stored predictions and could not see the later tier |
| - as the only segmentation tier it is level with bibr's whole default cascade on audience_eval200 (entries 0.983 against 0.983) and has no tier behind it: two papers of 198 drop below the reference floors, which the cascade reads correctly. It stays the last resort of the cascade. On the papers where both answer, neither is always right (24 better, 19 worse), so a choice between the two answers by their own quality figures is the open gain |
| |
| ## How it was made |
| |
| - trainer script: scripts/refseg_text_fallback_train.py train (rows from scripts/refseg_text_rows.py) |
| - recipe: epochs 2, steps 14,816, batch 16, encoder_lr 5e-05, head_lr 0.001, seed 13, selection: dev entry F1 on 400 dev papers every 1,000 steps, best step 11000 (0.9988), wall: 46 min on one shared RTX 5090 |
| - commit: 0b248015bc8c8f858409d8919f95ad6979dd158a |
| - seed: 13 |
| - run record sha256: b5a3cf9c0821f0b674c4e13ba45a9431b984197509c2c4980c6262398ab9fb13 |
|
|
| ## Licence and attribution |
|
|
| - model licence: mit |
| - training data licences: CC BY 21,668, CC0 106, CC BY-SA 24 |
| - starts from: jhu-clsp/mmBERT-small (22 layers x 384, multilingual); that model's own licence and card apply |
|
|
