scienceverse/bibr-parser-v7-mmbert-small

reference-parser-mmbert-small-v7

reference-parser-mmbert-small-v7 is a model for the bibr stage extract/references. Input: one reference string, list number stripped, no special tokens, max 256 tokens. bibr loads it through the setting NER_PARSER_CKPT on the default, fast path. It replaces reference-parser (scienceverse/bibr-parser-v5-1-ultrabert at 97b06cc7dd6511bdd53f8855c93a29020231d215).

Files

File	Bytes	sha256 (first 16)
bibr_model.json	8,730	e45239c56ea132ad
onnx/bibr_onnx.json	34,047	d28762a27475c985
onnx/model.onnx	268,711,556	90111b0a27ddb3fc
onnx/tokenizer.json	34,363,188	609d8f4c067cd395

The full checksums are in checksums.json.

Architecture

• encoder: jhu-clsp/mmBERT-small (multilingual)
• head: linear + CRF, 39 BIO tags over 19 fields (the shipped tag set)
• input: one reference string, list number stripped, no special tokens, max 256 tokens
• format: ONNX with int8 embeddings (268.7 MB; fp32 562.6 MB); loads in bibr's OnnxRefParser unchanged

Training data

• rows: train 1,586,198, main_v7 508,331, pdf 758,115, multilingual 264,470, crossref 55,282
• papers: main_v7 14,480, pdf_before_gate 24,358
• source: main v7: JATS reference strings (PMC, Europe PMC, SciELO, LinkedBooks), no per-paper cap; pdf: the same kind of labels projected onto bibr's own PDF reference strings (parser-pdf-v1); multilingual: references rendered through locale citation styles plus open annotated sets (parser-multiling-v1); crossref: deposited reference strings with distant labels. Every source passed one gate: corpus-split-v1, the exclusion registry (build 7), and a string and title check against every evaluation set, with 0 split mismatches
• labels: JATS mixed-citation and element-citation markup; citation-style variable marks (synthetic rows); aligned Crossref records (distant)
• languages: main v7 is 98% English; the multilingual source covers de, es, fr, it, ja, pl, pt, ru, uk, zh
• licences: main v7 CC BY 499,264 / CC0 8,786 / CC BY-SA 386 rows; PDF source rows (train split) CC BY 961,055 / CC0 4,204 / CC BY-SA 50; multilingual train rows 237,360 renderings of CC0 records (OpenAlex, OpenSSCI) through CSL citation styles, which are CC BY-SA 3.0, and 27,115 CC BY 4.0 rows (8,104 annotated references from GROBID's training data, 19,011 Cyrillic synthetic); AnyStyle (BSD-2-Clause) and SIRIS (MIT) rows were kept out by the CC0 / CC BY / CC BY-SA policy; Crossref rows rest on the metadata basis (3.5% of rows)

Every evaluation set in the exclusion registry was kept out of training. Registry: exclusions, build manifest sha256 d8d427de3ea7b954. Split: corpus-split-v1, sha256 c6fc246571f435d7.

Evaluation

Set	Metric	Value	Shipped	Difference	Interval	Notes
v6_dev (4,976 rows)	ref_exact	0.94	0.864	+0.076	95% [0.927, 0.952]
v6_test (4,959 rows)	ref_exact	0.938	0.841	+0.097	95% [0.922, 0.951]
v6_test_jd (4,367 rows; unseen journals)	ref_exact	0.933	0.822	+0.111	95% [0.917, 0.948]
pdf_test_jd (4,996 rows; PDF strings, unseen journals)	ref_exact	0.952	0.902	+0.050	95% [0.939, 0.963]
multiling_gold_v1 (600 strings, 10 languages; platform gold)	exact_rows all fields	0.513	0.398	+0.115	95% [0.475, 0.553]
multiling_gold_v1	exact_rows core fields	0.56	0.47	+0.09	95% [0.522, 0.6]
multiling_gold_v1	exact_rows all fields by language	de 0.567, es 0.75, fr 0.55, it 0.617, ja 0.417, pl 0.45, pt 0.6, ru 0.417, uk 0.317, zh 0.45
cyrillic_real (716 rows)	ref_exact	0.467	0.106	+0.361	95% [0.422, 0.516]
crossref_test (3,524 rows; distant labels)	ref_exact	0.9	0.553	+0.347	95% [0.873, 0.924]
crossref_test_jd (3,774 rows; distant labels)	ref_exact	0.812	0.672	+0.140	95% [0.775, 0.843]
excite_dev	ref_exact	0.702	0.664	+0.038	95% [0.654, 0.743]
linkedbooks_dev	ref_exact	0.809	0.831	-0.022	95% [0.759, 0.858]
RenoBench	ref_exact	0.727	0.724	+0.003
v4.5 gold val	ref_exact	0.845	0.931	-0.086	95% [0.823, 0.867]	shipped model in-sample
jats_test	record_exact_core	0.61	0.779	-0.169		shipped model in-sample
v6_dev 2,000 rows	cpu_ms_per_ref (ONNX; 4 threads, batch 32)	17.9	39.5	-21.6
v6_dev 2,000 rows	cpu_ms_per_ref (ONNX; 1 thread, batch 32)	36.6	81.4	-44.8
2,000 rows	onnx_tag_parity_vs_torch	1998 of 2000
audience_eval200, whole pipeline (198 papers)	refs.title	0.976	0.971	+0.005	difference 95% [-0.001, 0.011]	arm with all eight candidates of 11 November
audience_eval200, whole pipeline (198 papers)	refs.volume	0.984	0.948	+0.036	difference 95% [0.027, 0.046]
audience_eval200, whole pipeline (198 papers)	refs.journal	0.925	0.919	+0.006	difference 95% [-0.006, 0.019]
audience_eval200, whole pipeline (198 papers)	refs.author	0.988	0.989	-0.001	difference 95% [-0.003, 0.001]
gate192, whole pipeline (190 papers)	refs.volume	0.94	0.915	+0.025	difference 95% [0.017, 0.036]	arm with the parser, line tagger, text segmenter, statement tagger, paper-type and section classifier
gate192, whole pipeline (190 papers)	refs.journal	0.82	0.806	+0.014	difference 95% [0.005, 0.027]
gate192, whole pipeline (190 papers)	refs.title	0.959	0.968	-0.009	difference 95% [-0.025, 0.002]	the set's gold lists the journal as title for 125 title-less journal citations; without the 12 papers concerned 0.974 against 0.971

Known weaknesses

• books cited the way Nature prints them ("Authors. Title (Publisher, year).") come out as a journal name with no title: 12 of 8,531 entries on gate192 and 2 of 8,979 on audience_eval200, confirmed by gold; the shipped model files none of these as a journal. Cause, counted on the mix-v7 rows: Scientific Reports and Nature Communications mark books as journal citations in their JATS, and the row builder labels the source element of a journal-typed citation CONTAINER. Of the rows ending "(Publisher, year)", 211 journal-typed rows carry CONTAINER and no TITLE (90 in v6c, 121 in pdf) against 638 book-typed rows with TITLE (549 and 89)
• relabelling these book rows (282 training rows) and retraining fixes the books: 15 of 15 probe entries with the publisher first in the parentheses get a title and no journal (this model 7 of 15). Over three seeds it leaves two held-out cells below this model's seed floor (v6_dev journal 98.50 against 98.54, v6_test_jd title 96.39 against 96.43) and misses one entry of a colon probe; in the whole pipeline it moves no reference field either way. The relabelled model is not released; the rule waits for the next full parser round
• below the shipped model on v4.5 gold val (-8.6 points) and jats_test (-17 points); the shipped model trained on rows of both families. Of the 330 gold val rows this model misses, 100 are the doi.org convention below, 46 are gold rows that glue two references, 25 are article numbers, 13 are edition and bracket conventions; 144 are unexplained
• two label conventions for a DOI printed as a link are mixed in the training rows: the Crossref (10,025) and multilingual (29,136) rows tag the whole link as URL, the main and PDF rows tag the part from "10." as DOI. A second arm (F2) retrained on rows relabelled to one convention and is ahead on the held-out strings where the convention matters. In the whole pipeline on audience_eval200 its reference titles are lower: 0.965 against 0.976 for this model and 0.971 for the shipped one (paired difference to this model -0.012, interval -0.018 to -0.007; authors, years and pages about +0.002 each). By the rule fixed before that run this entry stays arm F1. Seed repeats (seeds 42, 43, 44 of each arm): of 119 score rows over 15 held-out sets, 25 show a gap between the arms larger than the range of either arm's three seeds. F2 is behind on v6_test_jd title (-0.49 points, 1.9 times the range) and on volume (gold_val -0.11, renobench -0.19); F2 is ahead on gold_val whole records (+1.26 core, +2.20 all fields) and on the Crossref sets relabelled by its own rule. Everything else is inside the seed range, so the pipeline title gap is not explained by the held-out strings and the choice rests on the pipeline rule
• seed spread of one recipe (F1, three seeds), for reading any single run: large sets move by 0.1 to 0.5 points per field (v6_dev title 96.80 to 96.95); small sets by far more: cyrillic_real (716 strings) title 61.4 to 74.4, volume 68.4 to 80.6, whole records 46.5 to 56.7; excite_dev pages 78.0 to 81.9; multiling whole records 54.8 to 56.0. One run on a small set decides nothing between recipes
• a uniform average of the three F1 seeds' weights collapses (v6_dev title 5.9 against 96.9; every set far below any single seed). The encoders differ by 6% of their norm, the linear head and the feature projection by about their own norm: heads initialised from different seeds do not average. A fair test needs a shared head initialisation or an average along one run
• a larger encoder does not help on these rows: the F1 recipe with jhu-clsp/mmBERT-base, one seed, against the three small-encoder seeds on the 15 held-out sets. Journal test sets within 0.3 points per field. gold_val worse: title 91.59 against 93.03 (seed range 0.48), doi 81.30 against 86.75 (1.65), whole records 58.62 against 59.94 (0.17). cyrillic_real and multiling about +1 point, far inside the seed range. One seed, recipe tuned on the small encoder. Distillation from it was not pursued; what limits the parser is the rows (non-English and Cyrillic references, books, DOI forms)
• EDITION is thin (2,338 train spans; entity F1 0.42 on v6_dev)
• linkedbooks_dev is 2 points under the shipped model (the interval includes zero)
• not yet scored on the held-back reference test sets
• the multilingual gold has 60 strings per language; per-language numbers are coarse
• latency was measured on a busy host (load 5 to 8 on a 4-core pin); the ratio is stable across two runs, the absolute numbers are not a budget check
• the baseline first reported on the multilingual gold (0.085 / 0.300) was the public 0.5.1 pin (v4.5-gold), not the shipped v5.1; the v5.1 numbers are the ones above

How it was made

• trainer script: scripts/ref_llm/train_parser_v6.py
• corpus builder: scripts/ref_llm/assemble_parser_corpus_v6.py -> scripts/ref_llm/relabel_parser_corpus.py -> scripts/ref_llm/gate_parser_mix.py
• recipe: 1.5M samples, batch 32, encoder lr 5e-5, head lr 1e-3, warmup 0.05, seed 42; sampling shares main 0.42, PDF 0.35, multilingual 0.15, Crossref 0.08; PDF-text noise on 30% of main rows (line-break, hyphen, identifier-break); best step 44,000 of 46,875 by dev entity F1 0.981
• commit: 83ba758659302b123a0597448cf4e9f7728a53ee
• seed: 42
• run record sha256: f2eebc39be8d540f1154fa328607543a6e0f1446e59d926074a050ed16f1a52b

Licence and attribution

• model licence: mit
• training data licences: main v7 CC BY 499,264 / CC0 8,786 / CC BY-SA 386 rows; PDF source rows (train split) CC BY 961,055 / CC0 4,204 / CC BY-SA 50; multilingual train rows 237,360 renderings of CC0 records (OpenAlex, OpenSSCI) through CSL citation styles, which are CC BY-SA 3.0, and 27,115 CC BY 4.0 rows (8,104 annotated references from GROBID's training data, 19,011 Cyrillic synthetic); AnyStyle (BSD-2-Clause) and SIRIS (MIT) rows were kept out by the CC0 / CC BY / CC BY-SA policy; Crossref rows rest on the metadata basis (3.5% of rows)
• starts from: jhu-clsp/mmBERT-small (multilingual); that model's own licence and card apply