Add the bibr front-matter tagger bundle (small-v2-embrow8)

13a0f5c verified 1 day ago

12.2 kB

---
license: mit
library_name: onnxruntime
base_model: jhu-clsp/mmBERT-small
pipeline_tag: token-classification
language:
  - en
  - es
  - pt
  - ru
  - uk
  - ja
  - id
tags:
  - bibr
  - scientific-papers
  - front-matter
  - metadata-extraction
  - crf
  - onnx
---
# bibr-front-tagger (small)

An [mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small) encoder with a CRF head that reads the first two pages of a
scientific paper and labels every word as part of the **title, subtitle, author names (given and family), author and
affiliation marks, affiliations, e-mail, ORCID, abstract, keywords, journal data (journal, volume, issue, pages, ISSN,
publisher, date, licence), DOI, or page furniture** (running heads, banners, page numbers).

[bibr](https://github.com/scienceverse/bibr) uses it for front matter in its **fast mode**, where no LLM is called:
the title, abstract, keywords and author list come from this tagger, and the affiliation links are resolved from its
marks. In full mode a confident tagger answer may stand in for the LLM call, and the LLM keeps the fields the tagger is
unsure of. It runs on the CPU through ONNX Runtime; bibr's core install needs no PyTorch.

This repository holds the **small** model (140M parameters, of which 42M outside the embeddings),
stored as **fp32 with int8 token embeddings**: one `model.onnx` of 269 MB.

## Model

- **Architecture:** `jhu-clsp/mmBERT-small` (ModernBERT layers, Gemma 2 tokenizer with 256,000 pieces, revision
  `abc32620`) + a gated residual that adds 32 per-word features + a linear layer to 43 BIO tags + a CRF whose
  parameters are stored in `onnx/bibr_onnx.json` and decoded in numpy (Viterbi for the tags, forward-backward for the
  confidence of each tag).
- **Input** (contract version 2): the text regions of the first pages in reading order, joined with a newline between
  regions, and for every word 32 floats: where its region sits on the page (relative x, y, width, height, centre
  offset, relative font size, bold, first on page, reading-order position), word position and shape (upper case,
  capitalised, digit, punctuation, superscript mark, CJK, length) and the layout label of its region (document title,
  paragraph title, text, abstract, header, footer, number, footnote, aside, reference, other). The regions and their
  layout labels come from bibr's layout detector and OCR or text layer; this model does not read page images. A
  subtoken takes the features of the first word it overlaps; a word no subtoken starts in takes the `I-` form of the
  tag of the subtoken that covers it.
- **Output:** one BIO tag and one posterior per word. bibr's assembler turns the tagged spans back into the printed
  title, the first printed abstract and keyword list, and the author list with their affiliations.
- **Windows:** `max_length` is 4,096 subtokens. A longer input runs in overlapping windows (stride 3/4 of the window);
  every token keeps the emissions of the window it sits deepest in and one CRF pass decodes the stitched sequence.
  bibr reads pages 1-2, and re-tags later windows (pages 2-3, then 3-4) when no abstract was found.
- **Precision:** fp32 with int8 token embeddings. The 256,000-row token-embedding table (70% of this model's weights) is stored as int8 with one fp32 scale per row and expanded right after the lookup (`Gather`, `Cast`, `Mul`; standard opset-17 nodes); every other weight and every computation is fp32. Checked against the fp32 graph on 260 papers outside the training data: this file tags 99.971% of words identically (245 of 260 papers word for word), and the front-matter composite (title match, author-name F1, abstract ROUGE-L) differs by -0.09 points (95% interval -0.30 to +0.04, paired bootstrap over papers). The file is 269 MB; the fp32 graph of the same model is 563 MB.

## Training data

3,960 rows train the model, 233 select the checkpoint and 463 are held out for the test below.
A row is one paper. The split is made by paper; Spanish, Portuguese, Russian, Ukrainian, Japanese and Indonesian papers are
stratified into every split, and roughly two thirds of the rows are English. Every paper is an open-access paper whose
own licence allows training a published model:

| source | what it adds | where the labels come from |
|---|---|---|
| Europe PMC open-access JATS papers and PLOS | English biomedical and life-science papers | the publisher's XML, projected onto the words of the printed page |
| SciELO | Spanish and Portuguese papers | SciELO's JATS metadata, projected the same way |
| J-STAGE | Japanese papers | the landing-page metadata |
| a pool drawn through OpenAlex | psychology, social science, economics, business and other fields; journal articles and preprints | the teacher described below, kept only where it agrees with OpenAlex |
| Russian and Ukrainian papers drawn through OpenAlex | Cyrillic-script papers | machine labels from a larger (27B) model reading the paper |

**Licence policy, in plain words.** A paper is used only when its own licence is CC BY or CC0 (any version) or, for the
OpenAlex pool, CC BY-SA or public domain (about one in eleven pool papers is CC BY-SA, a share-alike licence). Papers licensed non-commercial, no-derivatives, text-and-data-mining-only, with an
unspecified "open access" or with no licence statement are dropped. Papers used to develop and evaluate bibr, including
its withheld test sets, are excluded by DOI, OpenAlex id and title. No page image or paper text is
redistributed with this model.

**Scanned papers.** This recipe uses no synthetic scans.

**How the labels were produced.** For the JATS, SciELO and J-STAGE papers the labels are the publisher's own metadata,
aligned word by word to the page. For the OpenAlex pool the labels are **silver**: bibr's own pipeline, with its 4B
extraction model ([`scienceverse/bibr-extract-qwen35-4b`](https://huggingface.co/scienceverse/bibr-extract-qwen35-4b))
acting as the teacher, read the OCR of the paper, and a paper entered training only when the teacher agreed with its
OpenAlex record: title token F1 of at least 0.9, the same number of authors with family-name F1 of at least 0.9 and,
where OpenAlex has an abstract, abstract token F1 of at least 0.8. Where they disagree, the paper was left out rather
than corrected. Machine labels inherit the teacher's conventions and mistakes, and so does this model.

## Evaluation

**Held-out test split of the training corpus** (463 rows; the same sources as training, so it measures how well
the model reproduces its labels, not how well it does on a new kind of paper; part of the labels are silver):

| metric | value |
|---|---|
| word-level F1 (43 tags) | 0.965 |
| span F1 over all fields | 0.911 |
| span F1, title and authors | 0.956 |
| title exactly right | 0.959 |
| author list exactly right | 0.836 (initial-tolerant 0.838) |
| papers whose assembled front matter raises no problem | 0.801 |

At bibr's default confidence bar (0.9 lowest posterior on the title and names, mean on the abstract) the tagger answers
for the title of 87% of the test papers (exact on 98.0% of those) and for the author list
of 70% (right on 91.4% of those); the remaining papers go to the LLM in full mode.

**Against bibr's 4B extraction model on papers it was not trained on** (front-matter composite: the mean of title
match, author-name F1 and abstract ROUGE-L, with a paired bootstrap over papers):

bibr 0.9.0 fast mode with this bundle, minus bibr 0.8.0's 4B extraction model, in points with the 95% interval. A paper
with no export scores as empty. These are bibr's development sets: the tagger did not train on them, but bibr's
front-matter rules were tuned while looking at them, so read the figures as development results, not a held-out test.

Born-digital papers, text-layer route (no OCR):

| set | papers | fast minus 4B |
|---|---|---|
| 1 | 200 | −0.7 [−3.0, +1.5] |
| 2 | 60 | 0.0 [−2.4, +2.4] |
| 3 | 190 | −1.3 [−3.0, +0.4] |
| 4 | 86 | +0.2 [−0.4, +1.1] |
| 5 | 59 | +1.9 [−4.0, +7.8] |
| all five | 595 | −0.4 [−1.6, +0.7] |

On its own, author-name F1 over all five sets is −1.5 [−3.1, 0.0].

Scanned papers, OCR route, 80 papers: −15.1 [−22.9, −7.8], measured on a pre-release build of 0.9.0. Ten of the 80
gave no export and count as empty; on the 70 that gave one the difference is −4.7 [−9.5, +0.3]. This bundle learned
from born-digital pages only. For scans, use full mode with an
LLM where you can.

**Speed on CPU** (ONNX Runtime, 4 threads, front matter of one paper including the later windows when needed):

0.93 s per paper (median; mean 1.03 s, slowest 2.0 s) over 16 papers and 20,705 words, on 4 reserved cores of an AMD
Ryzen 7 5700X3D with ONNX Runtime set to 4 threads and nothing else running on those cores. Loading the model takes
1.8 s once; peak memory was 1.6 GB. This is the tagger alone: a whole fast-mode paper on the same 4 cores, layout and
reading included, took 31 to 33 s with bibr's default settings (60 papers, run twice), and 28 to 30 s with equation
reading and the corpus tables switched off.

## Limitations

- **Scanned papers are weaker.** The model learned from born-digital pages and, for the scan-aware recipe, from synthetic
  scans; on real scans the abstract and the byline are noticeably less reliable than on born-digital papers. bibr keeps
  the LLM for a scanned first page when one is configured.
- **Abstract position.** The tagger reads two pages at a time. An abstract that is printed on page 3 or 4 (title-page
  manuscripts, preprint cover sheets) is found by bibr re-tagging later windows, not by this model alone.
- **Several printed versions.** When a paper prints parallel abstracts or keyword lists (for example in two languages),
  the model tags all of them and bibr's assembler keeps the first printed one.
- **Names.** Silver labels follow the teacher's conventions for particles, suffixes and middle initials. OCR that glues
  a superscript mark onto a surname can cost the family name.
- **Languages.** Trained on English, Spanish, Portuguese, Russian, Ukrainian, Japanese and Indonesian papers (a few
  more with a handful of examples each). Other scripts and languages are untested, even though the encoder was
  pretrained on 1,800 languages.
- **Not a general tagger.** It needs bibr's region features and layout labels as input. Fed plain text it will not
  reproduce the numbers above.
- **No page images.** Layout and OCR errors upstream of the tagger (merged or split regions, missing text) show up as
  tagging errors.

## Files

| File | Purpose |
|---|---|
| `onnx/model.onnx` | `input_ids [batch, length]`, `attention_mask [batch, length]`, `token_features [batch, length, 32]` to `emissions [batch, length, 43]` (opset 17) |
| `onnx/bibr_onnx.json` | tag set, feature keys, contract version, CRF start / end / transition matrices, encoder revision, precision |
| `onnx/tokenizer.json` | the tokenizer of `jhu-clsp/mmBERT-small`, byte for byte (sha256 `197d4cc5406ee12c...`) |
| `LICENSE` | MIT, with the upstream notice |

## Using it in bibr

bibr resolves the bundle like its other ONNX models: `ML_FRONT_TAGGER_MODEL_ID` takes this repo id or a local directory
that holds `onnx/`, `ML_FRONT_TAGGER_REVISION` pins the commit, and the downloaded bundle is cached in the Hugging Face
cache, so a cached bundle also loads offline.

bibr 0.9.0 and later use this repository by default, at a pinned commit, so fast mode needs no setting:

```bash
bibr chew --fast paper.pdf
```

In full mode the tagger stays off unless `ML_FRONT_TAGGER_MODE=primary` is set.

## License

MIT, inherited from [jhu-clsp/mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small). The training papers keep their own licences and are
not redistributed.

## Citation

The encoder is mmBERT:

```bibtex
@misc{marone2025mmbertmodernmultilingualencoder,
  title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning},
  author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
  year={2025},
  eprint={2509.06888},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2509.06888}
}
```

	---
	license: mit
	library_name: onnxruntime
	base_model: jhu-clsp/mmBERT-small
	pipeline_tag: token-classification
	language:
	- en
	- es
	- pt
	- ru
	- uk
	- ja
	- id
	tags:
	- bibr
	- scientific-papers
	- front-matter
	- metadata-extraction
	- crf
	- onnx
	---

	# bibr-front-tagger (small)

	An [mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small) encoder with a CRF head that reads the first two pages of a
	scientific paper and labels every word as part of the **title, subtitle, author names (given and family), author and
	affiliation marks, affiliations, e-mail, ORCID, abstract, keywords, journal data (journal, volume, issue, pages, ISSN,
	publisher, date, licence), DOI, or page furniture** (running heads, banners, page numbers).

	[bibr](https://github.com/scienceverse/bibr) uses it for front matter in its fast mode, where no LLM is called:
	the title, abstract, keywords and author list come from this tagger, and the affiliation links are resolved from its
	marks. In full mode a confident tagger answer may stand in for the LLM call, and the LLM keeps the fields the tagger is
	unsure of. It runs on the CPU through ONNX Runtime; bibr's core install needs no PyTorch.

	This repository holds the small model (140M parameters, of which 42M outside the embeddings),
	stored as fp32 with int8 token embeddings: one `model.onnx` of 269 MB.

	## Model

	- Architecture: `jhu-clsp/mmBERT-small` (ModernBERT layers, Gemma 2 tokenizer with 256,000 pieces, revision
	`abc32620`) + a gated residual that adds 32 per-word features + a linear layer to 43 BIO tags + a CRF whose
	parameters are stored in `onnx/bibr_onnx.json` and decoded in numpy (Viterbi for the tags, forward-backward for the
	confidence of each tag).
	- Input (contract version 2): the text regions of the first pages in reading order, joined with a newline between
	regions, and for every word 32 floats: where its region sits on the page (relative x, y, width, height, centre
	offset, relative font size, bold, first on page, reading-order position), word position and shape (upper case,
	capitalised, digit, punctuation, superscript mark, CJK, length) and the layout label of its region (document title,
	paragraph title, text, abstract, header, footer, number, footnote, aside, reference, other). The regions and their
	layout labels come from bibr's layout detector and OCR or text layer; this model does not read page images. A
	subtoken takes the features of the first word it overlaps; a word no subtoken starts in takes the `I-` form of the
	tag of the subtoken that covers it.
	- Output: one BIO tag and one posterior per word. bibr's assembler turns the tagged spans back into the printed
	title, the first printed abstract and keyword list, and the author list with their affiliations.
	- Windows: `max_length` is 4,096 subtokens. A longer input runs in overlapping windows (stride 3/4 of the window);
	every token keeps the emissions of the window it sits deepest in and one CRF pass decodes the stitched sequence.
	bibr reads pages 1-2, and re-tags later windows (pages 2-3, then 3-4) when no abstract was found.
	- Precision: fp32 with int8 token embeddings. The 256,000-row token-embedding table (70% of this model's weights) is stored as int8 with one fp32 scale per row and expanded right after the lookup (`Gather`, `Cast`, `Mul`; standard opset-17 nodes); every other weight and every computation is fp32. Checked against the fp32 graph on 260 papers outside the training data: this file tags 99.971% of words identically (245 of 260 papers word for word), and the front-matter composite (title match, author-name F1, abstract ROUGE-L) differs by -0.09 points (95% interval -0.30 to +0.04, paired bootstrap over papers). The file is 269 MB; the fp32 graph of the same model is 563 MB.

	## Training data

	3,960 rows train the model, 233 select the checkpoint and 463 are held out for the test below.
	A row is one paper. The split is made by paper; Spanish, Portuguese, Russian, Ukrainian, Japanese and Indonesian papers are
	stratified into every split, and roughly two thirds of the rows are English. Every paper is an open-access paper whose
	own licence allows training a published model:

	\| source \| what it adds \| where the labels come from \|
	\|---\|---\|---\|
	\| Europe PMC open-access JATS papers and PLOS \| English biomedical and life-science papers \| the publisher's XML, projected onto the words of the printed page \|
	\| SciELO \| Spanish and Portuguese papers \| SciELO's JATS metadata, projected the same way \|
	\| J-STAGE \| Japanese papers \| the landing-page metadata \|
	\| a pool drawn through OpenAlex \| psychology, social science, economics, business and other fields; journal articles and preprints \| the teacher described below, kept only where it agrees with OpenAlex \|
	\| Russian and Ukrainian papers drawn through OpenAlex \| Cyrillic-script papers \| machine labels from a larger (27B) model reading the paper \|

	Licence policy, in plain words. A paper is used only when its own licence is CC BY or CC0 (any version) or, for the
	OpenAlex pool, CC BY-SA or public domain (about one in eleven pool papers is CC BY-SA, a share-alike licence). Papers licensed non-commercial, no-derivatives, text-and-data-mining-only, with an
	unspecified "open access" or with no licence statement are dropped. Papers used to develop and evaluate bibr, including
	its withheld test sets, are excluded by DOI, OpenAlex id and title. No page image or paper text is
	redistributed with this model.

	Scanned papers. This recipe uses no synthetic scans.

	How the labels were produced. For the JATS, SciELO and J-STAGE papers the labels are the publisher's own metadata,
	aligned word by word to the page. For the OpenAlex pool the labels are silver: bibr's own pipeline, with its 4B
	extraction model ([`scienceverse/bibr-extract-qwen35-4b`](https://huggingface.co/scienceverse/bibr-extract-qwen35-4b))
	acting as the teacher, read the OCR of the paper, and a paper entered training only when the teacher agreed with its
	OpenAlex record: title token F1 of at least 0.9, the same number of authors with family-name F1 of at least 0.9 and,
	where OpenAlex has an abstract, abstract token F1 of at least 0.8. Where they disagree, the paper was left out rather
	than corrected. Machine labels inherit the teacher's conventions and mistakes, and so does this model.

	## Evaluation

	Held-out test split of the training corpus (463 rows; the same sources as training, so it measures how well
	the model reproduces its labels, not how well it does on a new kind of paper; part of the labels are silver):

	\| metric \| value \|
	\|---\|---\|
	\| word-level F1 (43 tags) \| 0.965 \|
	\| span F1 over all fields \| 0.911 \|
	\| span F1, title and authors \| 0.956 \|
	\| title exactly right \| 0.959 \|
	\| author list exactly right \| 0.836 (initial-tolerant 0.838) \|
	\| papers whose assembled front matter raises no problem \| 0.801 \|

	At bibr's default confidence bar (0.9 lowest posterior on the title and names, mean on the abstract) the tagger answers
	for the title of 87% of the test papers (exact on 98.0% of those) and for the author list
	of 70% (right on 91.4% of those); the remaining papers go to the LLM in full mode.

	Against bibr's 4B extraction model on papers it was not trained on (front-matter composite: the mean of title
	match, author-name F1 and abstract ROUGE-L, with a paired bootstrap over papers):

	bibr 0.9.0 fast mode with this bundle, minus bibr 0.8.0's 4B extraction model, in points with the 95% interval. A paper
	with no export scores as empty. These are bibr's development sets: the tagger did not train on them, but bibr's
	front-matter rules were tuned while looking at them, so read the figures as development results, not a held-out test.

	Born-digital papers, text-layer route (no OCR):

	\| set \| papers \| fast minus 4B \|
	\|---\|---\|---\|
	\| 1 \| 200 \| −0.7 [−3.0, +1.5] \|
	\| 2 \| 60 \| 0.0 [−2.4, +2.4] \|
	\| 3 \| 190 \| −1.3 [−3.0, +0.4] \|
	\| 4 \| 86 \| +0.2 [−0.4, +1.1] \|
	\| 5 \| 59 \| +1.9 [−4.0, +7.8] \|
	\| all five \| 595 \| −0.4 [−1.6, +0.7] \|

	On its own, author-name F1 over all five sets is −1.5 [−3.1, 0.0].

	Scanned papers, OCR route, 80 papers: −15.1 [−22.9, −7.8], measured on a pre-release build of 0.9.0. Ten of the 80
	gave no export and count as empty; on the 70 that gave one the difference is −4.7 [−9.5, +0.3]. This bundle learned
	from born-digital pages only. For scans, use full mode with an
	LLM where you can.

	Speed on CPU (ONNX Runtime, 4 threads, front matter of one paper including the later windows when needed):

	0.93 s per paper (median; mean 1.03 s, slowest 2.0 s) over 16 papers and 20,705 words, on 4 reserved cores of an AMD
	Ryzen 7 5700X3D with ONNX Runtime set to 4 threads and nothing else running on those cores. Loading the model takes
	1.8 s once; peak memory was 1.6 GB. This is the tagger alone: a whole fast-mode paper on the same 4 cores, layout and
	reading included, took 31 to 33 s with bibr's default settings (60 papers, run twice), and 28 to 30 s with equation
	reading and the corpus tables switched off.

	## Limitations

	- Scanned papers are weaker. The model learned from born-digital pages and, for the scan-aware recipe, from synthetic
	scans; on real scans the abstract and the byline are noticeably less reliable than on born-digital papers. bibr keeps
	the LLM for a scanned first page when one is configured.
	- Abstract position. The tagger reads two pages at a time. An abstract that is printed on page 3 or 4 (title-page
	manuscripts, preprint cover sheets) is found by bibr re-tagging later windows, not by this model alone.
	- Several printed versions. When a paper prints parallel abstracts or keyword lists (for example in two languages),
	the model tags all of them and bibr's assembler keeps the first printed one.
	- Names. Silver labels follow the teacher's conventions for particles, suffixes and middle initials. OCR that glues
	a superscript mark onto a surname can cost the family name.
	- Languages. Trained on English, Spanish, Portuguese, Russian, Ukrainian, Japanese and Indonesian papers (a few
	more with a handful of examples each). Other scripts and languages are untested, even though the encoder was
	pretrained on 1,800 languages.
	- Not a general tagger. It needs bibr's region features and layout labels as input. Fed plain text it will not
	reproduce the numbers above.
	- No page images. Layout and OCR errors upstream of the tagger (merged or split regions, missing text) show up as
	tagging errors.

	## Files

	\| File \| Purpose \|
	\|---\|---\|
	\| `onnx/model.onnx` \| `input_ids [batch, length]`, `attention_mask [batch, length]`, `token_features [batch, length, 32]` to `emissions [batch, length, 43]` (opset 17) \|
	\| `onnx/bibr_onnx.json` \| tag set, feature keys, contract version, CRF start / end / transition matrices, encoder revision, precision \|
	\| `onnx/tokenizer.json` \| the tokenizer of `jhu-clsp/mmBERT-small`, byte for byte (sha256 `197d4cc5406ee12c...`) \|
	\| `LICENSE` \| MIT, with the upstream notice \|

	## Using it in bibr

	bibr resolves the bundle like its other ONNX models: `ML_FRONT_TAGGER_MODEL_ID` takes this repo id or a local directory
	that holds `onnx/`, `ML_FRONT_TAGGER_REVISION` pins the commit, and the downloaded bundle is cached in the Hugging Face
	cache, so a cached bundle also loads offline.

	bibr 0.9.0 and later use this repository by default, at a pinned commit, so fast mode needs no setting:

	```bash
	bibr chew --fast paper.pdf
	```

	In full mode the tagger stays off unless `ML_FRONT_TAGGER_MODE=primary` is set.

	## License

	MIT, inherited from [jhu-clsp/mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small). The training papers keep their own licences and are
	not redistributed.

	## Citation

	The encoder is mmBERT:

	```bibtex
	@misc{marone2025mmbertmodernmultilingualencoder,
	title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning},
	author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
	year={2025},
	eprint={2509.06888},
	archivePrefix={arXiv},
	primaryClass={cs.CL},
	url={https://arxiv.org/abs/2509.06888}
	}
	```