ror-set-point-retriever-e5-base

This model is the retriever for ROR set point matching. It finds candidate ROR records for an affiliation string. The matcher uses this model.

Part of the ROR set point matching collection.

What the model does

The model changes text into vectors. It changes an affiliation string into a query vector, and the text of each ROR record into a record vector. A high dot product means a good candidate. The matcher searches all 143,284 records and keeps the top 32. It adds the top 10 records of a lexical index, and a set model then selects the records that the string names.

The model is one shared encoder with mean pooling. The output vectors have a length of 1. The weights are fp16.

How to use

from sentence_transformers import SentenceTransformer
from datasets import load_dataset

model = SentenceTransformer("adambuttrick/ror-set-point-retriever-e5-base")
records = load_dataset("adambuttrick/ror-set-point-matching-train-validation-test", "ror_records", split="train")

model.max_seq_length = 128          # record texts: 128 tokens, as in training
r = model.encode(records["text"], prompt_name="passage", batch_size=256, convert_to_tensor=True)
model.max_seq_length = 512          # strings: up to 512 tokens
q = model.encode(["Dept. of Physics, University of Tokyo, Japan"], prompt_name="query", convert_to_tensor=True)
top = (q @ r.T).topk(32)
candidates = [records[int(i)]["ror_id"] for i in top.indices[0]]

The prompts are "query: " for strings and "passage: " for records.

Training data

All training data comes from the dataset ror-set-point-matching-train-validation-test. The exact rows are in training_data/.

Part	Config and split	Hard negatives	Rows
Teacher	`teacher_judgments` train: 167,533 strings, one row for each record with p ≥ 0.5	The records of the same string with p < 0.1	304,626
Labelled	`affiliations_unseen_institutions` train: 16,452 strings, 16,630 links. Each link is used 3 times	The candidates of the baseline that are not labels	49,890
Synthetic	`synthetic_non_latin` train: 130,886 strings. The 2,871 strings that name a held-out record are not used	Mined (see below)	136,188

The training data has 490,704 rows. In each step, each string has its positive record and 3 hard negatives, drawn at random from the hard negatives of the row. All other records of the batch are also negatives, except the other labels of the string.

These teacher strings are not used:

strings to which the teacher gives p ≥ 0.5 for a held-out record (28,474);
strings that are equal to an evaluation string (11).

The rows contain 8 strings (4 labelled and 4 synthetic) that are equal to affiliations validation strings. No test string is among them.

Hard negatives for synthetic strings

The first retriever, ror-set-point-retriever-e5-base-no-synthetic, searched each synthetic string and kept the top 64 records. The procedure read these records in rank order and dropped:

the labels of the string;
each record that is related to a label (parent, child, related, predecessor or successor);
each record with the same name as the source record;
each record with a score of 0.95 × the score of the source record, or more. The string probably names these records, but they have no label.

The first 10 records that remain are the negatives. The mean is 9.6 for each string. The labels in this step were the labels before the label rules of the dataset (see the synthetic_non_latin section of the dataset card).

Training procedure

Setting	Value
Base model	intfloat/multilingual-e5-base, revision `d12875059715`
Loss	InfoNCE, temperature 0.02, over all records of the batch
Batch	128 strings, each with 1 positive and 3 hard negatives
Epochs	1 (3,833 steps)
Optimizer	AdamW, learning rate 2e-5, weight decay 0.01
Schedule	Linear warm-up for 5% of the steps, then linear decay
Precision	bf16 autocast
Gradient clipping	1.0
Maximum length	96 tokens for strings and 128 tokens for records. The search uses 512 tokens for strings
Gradient checkpointing	Off
Seed	1363
Hardware	1 × NVIDIA H200, 17 minutes (996 s)

Evaluation

Recall of the gold links in the top 32 records, in percent, on affiliations_unseen_institutions. The validation and test splits contain strings for institutions that are not in the train split. "New" links go to records that no training string names. "Known" links go to the other records.

Split	All links	Known	New	Links in non-Latin strings	Links to held-out institutions
test (1,957 strings, 2,620 links)	96.3	93.7	97.2	97.9	98.2
validation (1,914 strings, 3,292 links)	92.8	95.3	91.9	99.2	91.1

"New" refers to the train split only: the teacher and synthetic strings can name a new record. The held-out institutions (1,255) have no string in the train split.

The test row comes from the search in search/topk_test_splits.jsonl.gz, and the validation row from the search of the training run (search/topk.jsonl.gz). For comparison, on the test split:

Candidates	Records for each string	All links	Known	New	Links in non-Latin strings
ror-set-point-retriever-e5-base, top 32	32.0	96.3	93.7	97.2	97.9
The same, plus the lexical top 10 (the matcher's candidates)	37.5	97.6	96.5	98.0	97.9
The candidates of the baseline (lexical, dense, acronym and alias generators)	18.0	94.6	95.6	94.3	85.2

Recall of the gold links on the other test splits, in percent:

Split	Strings	This model, top 32	The matcher's candidates
`affiliations` test	1,947	98.7	98.8
`datacite` test	3,000	99.0	99.4
`openalex_sample` test	40,000	98.6	98.9

The validation split contains all 272 links to CNRS (https://ror.org/02feahw73). Most of these strings name CNRS only with its acronym, inside the name of a laboratory. This decreases the recall on that split.

Files

Path	Content
Model files	The sentence-transformers model, with the prompts `query` and `passage`
`training_data/`	The training folder: `records.jsonl.gz` (`ror_id`, `text`), `train.jsonl.gz` (the rows), `queries.jsonl.gz` (`set`, `id`, `text`: the strings to search) and `manifest.json` (counts and sha256)
`search/topk.jsonl.gz`	`set`, `id`, `ids`, `scores`: the top 64 records for every query in `training_data/queries.jsonl.gz`, from a search on one NVIDIA H200. The set model's rows come from this file
`search/topk_test_splits.jsonl.gz`	The top 64 records for each string of the test splits `affiliations_test`, `unseen_test`, `datacite_test` and `openalex_sample_test`, from a search on one NVIDIA H200
`logs/`	The training log and summary
`conversion_check.json`	A check that this model gives the same vectors as the training code

Each training row has the fields q (the string), pos (the positive record), negs (the hard negatives), pos_all (all labels of the string) and src (teacher, labelled or synthetic). The query sets are teacher, labelled, synthetic (whose id is the line in synthetic_non_latin) and the evaluation sets.

The evaluation sets have these names in the files:

Name	Split
`unseen_validation`	`affiliations_unseen_institutions` validation
`unseen_test`	`affiliations_unseen_institutions` test
`affiliations_validation`	`affiliations` validation
`affiliations_test`	`affiliations` test
`datacite_test`	`datacite` test
`openalex_sample_test`	`openalex_sample` test