ror-set-point-set-model-mmbert-base

This model is the set model for ROR set point matching. It reads an affiliation string together with its candidate ROR records, and gives each candidate the probability that the string names it. The matcher uses this model.

Part of the ROR set point matching collection.

What the model does

1. The retriever ror-set-point-retriever-e5-base finds the top 32 records for the string.
2. A lexical index adds its top 10 records that are not already there. This gives about 37.5 candidates. They are in a random order.
3. This model reads the string and all candidates in one input: [CLS] string [SEP] record 1 [SEP] record 2 [SEP] ....
4. A linear layer gives a score to each candidate, from the mean of the token states of that candidate.
5. A fixed calibration changes each score into a probability: p = 1 / (1 + exp(-(a × score + b))).
6. The answer is every candidate with p ≥ 0.5. An empty answer means that the string names no organisation.

The model reads the candidates together. Thus it can learn rules that compare candidates, for example "label the hospital, but not the university in the hospital's name".

How to use

set_model.py loads and runs the model. It needs only torch, transformers, safetensors and huggingface_hub. Give it the texts of the ROR records from the config ror_records.

from huggingface_hub import hf_hub_download
from datasets import load_dataset
import importlib.util

path = hf_hub_download("adambuttrick/ror-set-point-set-model-mmbert-base", "set_model.py")
spec = importlib.util.spec_from_file_location("set_model", path)
set_model = importlib.util.module_from_spec(spec); spec.loader.exec_module(set_model)

records = load_dataset("adambuttrick/ror-set-point-matching-train-validation-test", "ror_records", split="train")
texts = dict(zip(records["ror_id"], records["text"]))
m = set_model.SetMatcher.from_pretrained("adambuttrick/ror-set-point-set-model-mmbert-base", texts)
m.predict([("Dept. of Physics, University of Tokyo, Japan", candidates)])
# -> [[("https://ror.org/057zh3y96", 0.99...)]]

Use a GPU for speed. The model answered 130 strings each second on one NVIDIA H200.

Training data

All training data comes from the dataset ror-set-point-matching-train-validation-test. The exact rows are in training_data/.

Stage	Config and split	Strings	Candidates	Targets
1	teacher_judgments train	167,533	The retriever's top 32, then the records that the teacher judged and that are not in it (positives first), to a maximum of 42	The teacher's p. 0 for records with no judgment
1	synthetic_non_latin train	130,882	The retriever's top 32	1 for the labels, 0 for other records
2	affiliations_unseen_institutions train	16,448	The retriever's top 32 and the lexical top 10	1 for the labels, 0 for other records

• A label that is not among the candidates replaces a random candidate that is not a label (663 times).
• These strings are not used: teacher strings that the teacher links to a held-out record (p ≥ 0.5), and every training string that is equal to an evaluation string.
• The synthetic strings that name a held-out institution are not used (2,871 strings). Thus no row has a held-out institution as a positive (a label, or a teacher p ≥ 0.5).
• A teacher string that gives a held-out institution a p below 0.5 stays, with that p as its target. 9,606 strings have a target from 0.1 to 0.49 for a held-out institution. The recall of the held-out institutions does not increase with these targets.
• The retriever trained on these strings. Thus their candidate lists hold the labels more often than the lists of new strings. The replacement of missing labels makes sure that each label is a candidate in training.

Training procedure

1. Stage 1: one epoch of the stage 1 rows.
2. Stage 2, five times: start from the stage 1 weights, train on four folds of the labelled rows, and score the fifth fold. This gives an out-of-fold score for each candidate of each labelled string.
3. Fit the calibration on the out-of-fold scores.
4. Stage 2 on all labelled rows, from the stage 1 weights. This is the final model.

Setting	Value
Base model	jhu-clsp/mmBERT-base, revision c5955035435e, 307M parameters
Input	String: up to 96 tokens. Each record: up to 64 tokens, fewer when the input would be longer than 3,072 tokens
Loss	Binary cross-entropy against the targets, over the real candidates
Stage 1	1 epoch (18,651 steps) at learning rate 2e-5
Stage 2	2 epochs at half that learning rate. The final run has 2,056 steps
Folds	5, by the sha256 of the fold seed (20261022) and the string id
Batch	16 strings
Optimizer	AdamW, weight decay 0.01
Schedule	Linear warm-up for 5% of the steps, then linear decay
Precision	bf16 autocast. The weights in this repository are bf16
Gradient clipping	1.0
Calibration	Platt scaling on 615,010 out-of-fold candidates: a = 0.818, b = -0.071
Out-of-fold result	precision 95.4, recall 94.3, exact match 90.9
Seed	1363
Hardware	1 × NVIDIA H200, 2.8 hours (9,929 s)

Evaluation

The table gives exact match, precision and recall in percent. Precision and recall are micro averages over ROR ids. The last column is the recall of the links to the 1,255 held-out institutions. The baseline, ror-set-point-pair-scorer-baseline, trained on the same labelled strings. The answers on the test split come from the matcher with ror-set-point-retriever-e5-base, on one NVIDIA H200. The answers on the validation splits come from the training run.

Split	Strings	System	Exact match	Precision	Recall	Recall on held-out institutions
affiliations_unseen_institutions test	1,957	v3.0.0, local	77.0	95.9	80.0
		Baseline	79.2	93.6	85.1	86.3
		This model	83.6	94.6	86.9	89.6
affiliations_unseen_institutions validation	1,914	v3.0.0, local	62.7	96.5	67.7
		Baseline	75.0	94.8	84.8	84.9
		This model	73.9	95.4	81.7	78.7
affiliations validation	1,947	v3.0.0, local	80.1	93.5	79.6
		Baseline	85.7	94.7	87.4
		This model	91.0	96.5	91.9

The validation split of affiliations_unseen_institutions contains all 272 links to CNRS (https://ror.org/02feahw73). This model finds 33.8% of these links, and the baseline 93.8%. Without the CNRS strings, the recall on held-out institutions is 86.5 for this model and 83.5 for the baseline.

Other test splits

In this table, the baseline trained on the labelled strings of all institutions (affiliations train and openalex_sample train). This model trained on fewer institutions.

Split	Strings	System	Exact match	Precision	Recall
affiliations test	1,947	v3.0.0, local	78.4	94.3	79.0
		Baseline	86.1	96.1	88.1
		This model	90.0	95.7	92.6
affiliations_crossref test	447	v3.0.0, local	85.9	96.7	88.6
		Baseline	91.7	97.9	93.9
		This model	92.8	97.3	95.5
affiliations_openalex test	751	v3.0.0, local	79.6	94.9	81.7
		Baseline	87.5	96.4	90.3
		This model	90.0	96.0	93.0
affiliations_openalex_non_latin test	749	v3.0.0, local	72.6	91.0	68.3
		Baseline	81.4	94.0	80.8
		This model	88.4	94.1	89.6
datacite test	3,000	v3.0.0, local	92.4	97.7	93.2
		Baseline	91.9	95.1	95.6
		This model	93.5	98.0	94.5
openalex_sample test, sample v1	20,000	v3.0.0, local	81.6	95.3	83.2
		Baseline	86.9	95.6	90.0
		This model	89.9	96.1	92.1
openalex_sample test, sample v2	20,000	v3.0.0, local	82.6	94.9	84.2
		Baseline	88.0	95.2	91.5
		This model	90.4	95.7	93.3

• On affiliations test, this model is right on 150 strings where the baseline is wrong, and wrong on 74 strings where the baseline is right (exact McNemar p = 4 × 10⁻⁷).
• No model, setting or training-data rule was chosen from the scores on a test split. The affiliations_unseen_institutions test split served as a pass-or-fail check for the retriever and for the set model. openalex_sample test and datacite were scored for information before the final scoring.
• "v3.0.0, local" is openalex-affiliation-matcher v3.0.0 without the two candidate generators that need OpenAlex services, and without the teacher.
• Of the 3,000 datacite strings, 2,140 (71%) are equal, after normalisation, to a name, acronym, alias or label of their only gold record. On the other 860 strings, exact match is 88.6 for this model, 86.0 for the baseline and 82.0 for v3.0.0, local.

Limitations

• The model reads only the first 96 tokens of a string. Long strings that name many organisations lose recall.
• Organisations that a string names only with an acronym (for example "CNRS" or "INSERM") have lower recall.
• Strings that name several organisations have lower recall than strings that name one.
• When a string names a parent organisation and its child, the model sometimes names only the child.
• For new institutions, p below 0.5 is often too low. On the affiliations_unseen_institutions validation split, 56% of the candidates with p from 0.35 to 0.5 are correct. Thus the threshold leaves out some correct records.

Files

Path	Content
config.json, model.safetensors, tokenizer files	The encoder
head.safetensors	The linear layer
set_model_config.json	Input limits, calibration and threshold
set_model.py	Code that loads and runs the model
training_data/	The training folder: records.jsonl.gz (ror_id, text), stage1.jsonl.gz (s, c candidates, y targets, src: teacher or synthetic), stage2.jsonl.gz (k, s, c, y, fold), eval.jsonl.gz (set, k, s, c: the evaluation rows, with no labels) and manifest.json
evaluation/predictions/<set>.jsonl	The answers for each evaluation set: id (the id of the dataset row), ror_ids, and for the test sets p (the probability of each named record)
evaluation/probabilities.jsonl.gz	set, k, c, z (scores) and p (probabilities) of each candidate of each row of training_data/eval.jsonl.gz, from the training run
evaluation/oof.jsonl.gz, evaluation/calibration.json	The out-of-fold scores and the calibration fit
logs/	The training log and summary
evaluation/candidates/<set>.jsonl.gz	The candidates of each test string (id, text, candidates), in the order that the model read them

The predictions cover all six evaluation sets. The answers on the validation sets come from the training run. The answers on the test sets come from the matcher, with the candidates of ror-set-point-retriever-e5-base and the lexical index. The candidates of the test splits are in evaluation/candidates/. The ids of affiliations_test and affiliations_validation are also the ids of the rows in the per-source configs (affiliations_crossref, affiliations_openalex and affiliations_openalex_non_latin).

The evaluation sets have these names in the files:

Name	Split
unseen_validation	affiliations_unseen_institutions validation
unseen_test	affiliations_unseen_institutions test
affiliations_validation	affiliations validation
affiliations_test	affiliations test
datacite_test	datacite test
openalex_sample_test	openalex_sample test

The training log uses the names of the training run.

Licence

The weights come from jhu-clsp/mmBERT-base and use the MIT licence.

Related repositories

• Dataset: ror-set-point-matching-train-validation-test.
• Collection: ROR set point matching.
• Retriever: ror-set-point-retriever-e5-base. Set model: ror-set-point-set-model-mmbert-base. Baseline: ror-set-point-pair-scorer-baseline.