Here is MeaningBERT

MeaningBERT is an automatic and trainable metric for assessing meaning preservation between sentences. MeaningBERT was proposed in our article MeaningBERT: assessing meaning preservation between sentences. Its goal is to assess meaning preservation between two sentences that correlate highly with human judgments and sanity checks. For more details, refer to our publicly available article.

This public version of our model uses the best model trained (where in our article, we present the performance results of an average of 10 models) for a more extended period (500 epochs instead of 250). We have observed later that the model can further reduce dev loss and increase performance. Also, we have changed the data augmentation technique used in the article for a more robust one, that also includes the commutative property of the meaning function. Namely, Meaning(Sent_a, Sent_b) = Meaning(Sent_b, Sent_a).

• HuggingFace Model Card
• HuggingFace Metric Card

Two checkpoints in this repository

The repository root holds the v1 model described in the article, unchanged, so anything already using from_pretrained("davebulaval/MeaningBERT") keeps returning exactly the scores it returned before.

The v2 checkpoints live in subfolders:

subfolder	encoder	Pearson r	identical pairs above 95	100 pairs, GPU / CPU
(root)	bert-base-uncased	0.323	0 %	0.36 s / 1.99 s
large	deberta-v3-large	0.704 ± 0.009	97.1 % ± 1.4	1.16 s / 9.49 s
base	bert-base-uncased	coming	coming	0.36 s / 1.99 s

All three are measured on the same held-out test set of 1536 human-annotated pairs, with the control pairs reported separately rather than folded into the correlation.

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# v2, the recommended checkpoint
tokenizer = AutoTokenizer.from_pretrained("davebulaval/MeaningBERT", subfolder="large")
model = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT", subfolder="large")

# v1, as published in the article
model_v1 = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT")

The v2 checkpoints apply their output head outside the model: read config.meaningbert_output_head, which is clamped for large, and map the logit with logit.clamp(0, 1) * 100. Calling logits.tolist() as the v1 snippet below does returns unit-scale values for them, not a 0-100 score.

What v2 changed

Trained on a corpus three times the size of v1's, spanning four sources instead of one, with the label errors corrected and the split done by source sentence rather than by row. Per test corpus:

corpus	v1	v2 large
CSMD (v1's own training corpus)	0.870	0.832
SimpEval	0.057	0.867
SimpleText	0.169	0.393
PLABA	0.087	0.069

The CSMD line favours v1 because part of that test set was in its training data; v2 never saw those rows.

Most of the gain comes from the corpus and the training recipe, not from the larger encoder: at constant recipe, bert-base-uncased trained on the v2 corpus reaches 0.637, so the data accounts for +0.314 and the encoder for +0.067.

Sanity Check

Correlation to human judgment is one way to evaluate the quality of a meaning preservation metric. However, it is inherently subjective, since it uses human judgment as a gold standard, and expensive since it requires a large dataset annotated by several humans. As an alternative, we designed two automated tests: evaluating meaning preservation between identical sentences (which should be 100% preserving) and between unrelated sentences (which should be 0% preserving). In these tests, the meaning preservation target value is not subjective and does not require human annotation to be measured. They represent a trivial and minimal threshold a good automatic meaning preservation metric should be able to achieve. Namely, a metric should be minimally able to return a perfect score (i.e., 100%) if two identical sentences are compared and return a null score (i.e., 0%) if two sentences are completely unrelated.

Identical Sentences

The first test evaluates meaning preservation between identical sentences. To analyze the metrics' capabilities to pass this test, we count the number of times a metric rating was greater or equal to a threshold value X∈[95, 99] and divide It is calculated by the number of sentences to create a ratio of the number of times the metric gives the expected rating. To account for computer floating-point inaccuracy, we round the ratings to the nearest integer and do not use a threshold value of 100%.

Unrelated Sentences

Our second test evaluates meaning preservation between a source sentence and an unrelated sentence generated by a large language model.3 The idea is to verify that the metric finds a meaning preservation rating of 0 when given a completely irrelevant sentence mainly composed of irrelevant words (also known as word soup). Since this test's expected rating is 0, we check that the metric rating is lower or equal to a threshold value X∈[5, 1]. Again, to account for computer floating-point inaccuracy, we round the ratings to the nearest integer and do not use a threshold value of 0%.

Symmetry

Our third test evaluates a property of the metric rather than of a sentence pair. meaning(A, B) and meaning(B, A) ask the same question, how much of the meaning is shared, and the position of a sentence in the call carries no semantic information. The two calls should therefore return the same number. Unlike the two tests above, this one needs no generated data at all: it re-scores the evaluation pairs in the reverse order and measures the absolute difference, which has an expected value of exactly 0.

A metric can pass the first two tests and fail this one. It is worth measuring because the failure is silent: two studies using the same metric, one calling meaning(source, simplification) and the other meaning(simplification, source), would report different numbers with nothing signalling the discrepancy.

Measured on 1652 test pairs, mean absolute difference between the two orders:

model	mean	median	pairs differing by more than 10 points
MeaningBERT v1, as published	6.12	3.12	20.8 %
v2 trained without mirrored pairs	7.79	4.88	31.1 %
v2 as released	1.48	0.74	0.6 %

The property is taught, not enforced: the training corpus contains the mirror of every non-identical pair, with the label carried over unchanged. That brings the violation down by a factor of five, and it does not remove it. The scorer deliberately runs a single direction, the one you pass, rather than averaging both: averaging would make the property exact at twice the inference cost, and would hide in the wrapper a residual violation that belongs in the results. Pass your pairs in a consistent order.

Use MeaningBERT

You can use MeaningBERT as a model that you can retrain or use for inference using the following with HuggingFace

# Load model directly. subfolder="large" is the v2 checkpoint; without it you get v1.
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("davebulaval/MeaningBERT", subfolder="large")
model = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT", subfolder="large")

or you can use MeaningBERT as a metric for evaluation (no retrain) using the following with HuggingFace

import torch

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("davebulaval/MeaningBERT", subfolder="large")
scorer = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT", subfolder="large")
scorer.eval()

documents = ["He wanted to make them pay.", "This sandwich looks delicious.", "He wants to eat."]
simplifications = ["He wanted to make them pay.", "This sandwich looks delicious.",
                   "Whatever, whenever, this is a sentence."]

# We tokenize the text as a pair and return Pytorch Tensors.
# max_length is explicit on purpose: deberta-v3-large declares no maximum length, so
# truncation=True alone silently does nothing and a long pair reaches the model whole.
# 256 is the bound the v2 checkpoints were trained under.
tokenize_text = tokenizer(documents, simplifications, truncation=True, max_length=256,
                          padding=True, return_tensors="pt")

with torch.no_grad():
    # We process the text
    scores = scorer(**tokenize_text)

# The output head is applied OUTSIDE the model, so the logits are NOT the score.
# config.meaningbert_output_head says which one: "clamped" for the v2 checkpoints.
print((scores.logits.squeeze(-1).clamp(0, 1) * 100).tolist())

The v1 snippet does not carry over. v1 was trained with a linear head whose logit is the score, so its model card prints scores.logits.tolist() directly. The v2 checkpoints apply a bounded head outside the model: printing the raw logits returns unit-scale values such as 0.83, silently, with no error. Read config.meaningbert_output_head and map accordingly, or use meaningbert.scorer.MeaningBERTScorer, which reads it for you and always returns 0-100.

# The same thing, without having to know which head the checkpoint uses
from meaningbert import MeaningBERTScorer

scorer = MeaningBERTScorer("davebulaval/MeaningBERT", subfolder="large")
print(scorer.score(documents, simplifications))

or using our HuggingFace Metric module

import evaluate

documents = ["He wanted to make them pay.", "This sandwich looks delicious.", "He wants to eat."]
simplifications = ["He wanted to make them pay.", "This sandwich looks delicious.",
                   "Whatever, whenever, this is a sentence."]

# Three checkpoints, named as the second argument of evaluate.load.
# "large" is the default: deberta-v3-large, the recommended one.
meaning_bert = evaluate.load("davebulaval/meaningbert")
meaning_bert = evaluate.load("davebulaval/meaningbert", "large")  # the same thing

# "base" is bert-base-uncased: 3.2x faster on GPU, 4.8x on CPU, a quarter of the memory.
meaning_bert_base = evaluate.load("davebulaval/meaningbert", "base")

# "v1" is the model published with the 2023 article, unchanged.
meaning_bert_v1 = evaluate.load("davebulaval/meaningbert", "v1")

print(meaning_bert.compute(references=documents, predictions=simplifications))

Which checkpoint to load

The two v2 checkpoints are subfolders of the same repository, davebulaval/MeaningBERT, whose root is still the v1 model. Nothing already loading that repository changes behaviour.

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT", subfolder="large")

name	encoder	Pearson r	identical pairs above 95	100 pairs, GPU / CPU	weights
large (default)	deberta-v3-large	0.704 ± 0.009	97.1 % ± 1.4	1.16 s / 9.49 s	1740 MB
base†	bert-base-uncased	0.637	70.4 %	0.36 s / 1.99 s	438 MB
v1	bert-base-uncased	0.323	0 %	0.36 s / 1.99 s	438 MB

^† base is not published yet: its figures come from a single seed, against ten for large, and loading it raises rather than falling back on another checkpoint. A name that does not exist raises too, so a typo cannot silently swap the model under you.

Load base when you score at volume or have no GPU, large when a wrong score costs you something. On a GPU the speed argument is weak, see A middle option worth knowing about.

The numbers are detailed in MeaningBERT v2: which checkpoint to use.

---

Cite

Use the following citation to cite MeaningBERT

@ARTICLE{10.3389/frai.2023.1223924,
AUTHOR={Beauchemin, David and Saggion, Horacio and Khoury, Richard},    
TITLE={MeaningBERT: assessing meaning preservation between sentences},      
JOURNAL={Frontiers in Artificial Intelligence},      
VOLUME={6},           
YEAR={2023},      
URL={https://www.frontiersin.org/articles/10.3389/frai.2023.1223924},       
DOI={10.3389/frai.2023.1223924},      
ISSN={2624-8212},   
}

---

Contributing to MeaningBERT

We welcome user input, whether it regards bugs found in the library or feature propositions! Make sure to have a look at our contributing guidelines for more details on this matter.

License

MeaningBERT is MIT licensed, as found in the LICENSE file.

---