scienceverse/bibr-parser-v5-1-ultrabert

bibr Parser v5.1 (UltraBERT-150M)

UltraBERT-150M + feature-gated CRF reference-field parser for bibr's REF_PARSE_STRATEGY=ner path. Same model class (FeatureGatedEncoderCRF), 39 BIO tags and runtime contract as scienceverse/bibr-parser-v4-5-gold; the encoder and the training data change.

Model

• Encoder: dmis-lab/UltraBERT-150M (Apache-2.0) at revision 545ac9d9c7aece444f014ada5a1e9818132b4d35. Its config.json, configuration_ultrabert.py, modeling_ultrabert.py, tokenizer.json and tokenizer_config.json are copied unchanged into this repo (see LICENSE-UltraBERT), so loading the encoder from this repo at a pinned commit never runs unpinned remote code. The encoder is custom code: the torch path needs trust_remote_code=True (bibr enables it only when the parser revision is a full commit SHA); the ONNX bundle in onnx/ needs no custom code.
• Tokenizer convention: add_special_tokens=False, max sequence length 256. The tokenizer is GPT-2 style BPE that splits digits one per token (about 17 % more tokens per reference than ModernBERT-base).
• Per-token features: all zeros at training and at inference. feature_proj.bias is still part of the forward path, so pass a zeros tensor, not None (as for v4.5).
• Training: from the pretrained encoder (no v4.5 init), 3 epochs (8,844 optimizer steps), batch 16 x grad-accum 2, encoder LR 2e-5, head LR 1e-3, cosine, 5 % warmup, bf16, seed 42; best epoch by validation entity F1 (0.9650 on 2,990 rows; metrics.json, report.txt).

Training data

94,326 rows, all from sources whose licence allows training a published model. 30 % (28,183) carry augment_reference character noise. Fields come from each source's own markup (JATS <mixed-citation> / <element-citation>, refset gold), mapped to v4.5 gold span conventions (first+last page → PAGE_RANGE_START/END, first page only → PAGES, YEAR without its suffix letter) plus EDITOR, PUBLISHER, EDITION and URL.

source	rows	licence
Europe PMC open-access JATS (ref-LLM SFT v1 papers)	45,676	CC BY / CC0 (per paper)
Europe PMC open-access JATS (diverse2 papers, deduplicated vs the above)	21,021	CC BY / CC0 (per paper)
SciELO JATS (Brazil 4,123, Peru 1,354, Chile 1,283, Venezuela 81)	6,841	CC BY (per paper)
bibr-validation refsets dev: LinkedBooks	4,330	CC BY 4.0
v4.5 gold train (psych250 / mdpi_socsci100 / degruyter_econ100)	8,170 / 4,839 / 3,449	in-house annotations, as in the public v4.5 parser

Each JATS paper's licence is read from its own <permissions>/<license> element and classified with bibr-training's corpus.license_policy (permissive = CC BY or CC0, any version). Papers with a non-commercial, no-derivatives, share-alike, publisher-specific or missing licence are dropped, as is the EXCITE refset (no licence stated upstream).

Exclusions. Every training string whose normalized form (plaintext_sha256) is in bibr-validation refsets/holdout_keys.jsonl; the papers of the ref-LLM v1 test, diverse2 test and the diverse2 out-of-distribution sources; sealed gold papers (filter_sealed_papers); RenoBench and the bibr blind-set papers (by DOI). RenoBench is evaluation only.

Evaluation

Against the production parser v4.5 and, for reference, "U2" (the same recipe on a larger corpus that also had non-permissive and EXCITE rows; not released). (b) and (c) run bibr's aligned NER reference path and are scored with bibr-validation scripts/score_refsets.py (aggregates only).

metric	v4.5	v5.1 (this)	U2 (2 seeds, mean)
(a) JATS 2,000-row test, record exact	0.0045	0.7875	0.7785
(b) refsets sealed test (9,728 refs), ref exact	0.424	0.7514	0.7972
· EXCITE part	0.552	0.678	0.789
· LinkedBooks part	0.307	0.818	0.807
(c) RenoBench 10k, ref exact	0.557	0.7262	0.7269
(d) v4.5 gold val (4,031 refs), strict entity F1 on v4.5's fields	0.966	0.9749	0.9716

• (a) mostly measures span conventions: the JATS test shares the training rows' labelling and v4.5 does not (it keeps citation numbers and trailing punctuation in authors, for example).
• (b) EXCITE is now fully out of distribution for this model (U2 trained on its dev split); LinkedBooks dev is in the training data (its sealed test strings are not). v4.5 saw neither.
• Refsets macro field accuracy: 0.851 vs 0.705 for v4.5.

ONNX parity with the torch checkpoint: max abs emission difference below 6e-5 and identical parses on bibr's built-in samples.

Files

• best.pt - PyTorch state dict for FeatureGatedEncoderCRF
• config.json, configuration_ultrabert.py, modeling_ultrabert.py, tokenizer.json, tokenizer_config.json - the UltraBERT-150M encoder config, modeling code and tokenizer (unchanged copies; weights come from best.pt)
• LICENSE-UltraBERT - Apache-2.0 licence and notice for those files
• onnx/ - bibr ONNX Runtime bundle (model.onnx, bibr_onnx.json with the CRF parameters, tokenizer.json); bibr_onnx.json records the source commit of best.pt
• metrics.json - training validation summary (use_features refers to the model class; the feature values were all zero)
• report.txt - per-field validation classification report
• tag_config.json - tag metadata from bibr_training.tags