align the card with the repository: symmetry check, and examples that load the v2 checkpoint

README.md

@@ -104,17 +104,44 @@ irrelevant sentence mainly composed of irrelevant words (also known as word soup
 Again, to account for computer floating-point inaccuracy, we round the ratings to the nearest integer and do not use
 a threshold value of 0%.
 
+### Symmetry
+
+Our third test evaluates a property of the metric rather than of a sentence pair. `meaning(A, B)` and `meaning(B, A)`
+ask the same question, how much of the meaning is shared, and the position of a sentence in the call carries no semantic
+information. The two calls should therefore return the same number. Unlike the two tests above, this one needs no
+generated data at all: it re-scores the evaluation pairs in the reverse order and measures the absolute difference,
+which has an expected value of exactly 0.
+
+A metric can pass the first two tests and fail this one. It is worth measuring because the failure is silent: two
+studies using the same metric, one calling `meaning(source, simplification)` and the other `meaning(simplification,
+source)`, would report different numbers with nothing signalling the discrepancy.
+
+Measured on 1652 test pairs, mean absolute difference between the two orders:
+
+| model | mean | median | pairs differing by more than 10 points |
+|---|---|---|---|
+| MeaningBERT v1, as published | 6.12 | 3.12 | **20.8 %** |
+| v2 trained without mirrored pairs | 7.79 | 4.88 | 31.1 % |
+| **v2 as released** | **1.48** | **0.74** | **0.6 %** |
+
+The property is taught, not enforced: the training corpus contains the mirror of every non-identical pair, with the
+label carried over unchanged. That brings the violation down by a factor of five, and it does not remove it. The scorer
+deliberately runs a single direction, the one you pass, rather than averaging both: averaging would make the property
+exact at twice the inference cost, and would hide in the wrapper a residual violation that belongs in the results. Pass
+your pairs in a consistent order.
+
+
 ## Use MeaningBERT
 
 You can use MeaningBERT as a [model](https://huggingface.co/davebulaval/MeaningBERT) that you can retrain or use for
 inference using the following with HuggingFace
 
 ```python
-# Load model directly
+# Load model directly. subfolder="large" is the v2 checkpoint; without it you get v1.
 from transformers import AutoTokenizer, AutoModelForSequenceClassification
 
-tokenizer = AutoTokenizer.from_pretrained("davebulaval/MeaningBERT")
-model = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT")
+tokenizer = AutoTokenizer.from_pretrained("davebulaval/MeaningBERT", subfolder="large")
+model = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT", subfolder="large")
 ```
 
 or you can use MeaningBERT as a metric for evaluation (no retrain) using the following with HuggingFace
@@ -124,8 +151,8 @@ import torch
 
 from transformers import AutoTokenizer, AutoModelForSequenceClassification
 
-tokenizer = AutoTokenizer.from_pretrained("davebulaval/MeaningBERT")
-scorer = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT")
+tokenizer = AutoTokenizer.from_pretrained("davebulaval/MeaningBERT", subfolder="large")
+scorer = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT", subfolder="large")
 scorer.eval()
 
 documents = ["He wanted to make them pay.", "This sandwich looks delicious.", "He wants to eat."]
@@ -139,9 +166,23 @@ with torch.no_grad():
     # We process the text
     scores = scorer(**tokenize_text)
 
-print(scores.logits.tolist())
+# The output head is applied OUTSIDE the model, so the logits are NOT the score.
+# config.meaningbert_output_head says which one: "clamped" for the v2 checkpoints.
+print((scores.logits.squeeze(-1).clamp(0, 1) * 100).tolist())
 ```
 
+> **The v1 snippet does not carry over.** v1 was trained with a linear head whose logit *is* the score, so its model card
+> prints `scores.logits.tolist()` directly. The v2 checkpoints apply a bounded head outside the model: printing the raw
+> logits returns unit-scale values such as `0.83`, silently, with no error. Read `config.meaningbert_output_head` and map
+> accordingly, or use `meaningbert.scorer.MeaningBERTScorer`, which reads it for you and always returns 0-100.
+
+```python
+# The same thing, without having to know which head the checkpoint uses
+from meaningbert import MeaningBERTScorer
+
+scorer = MeaningBERTScorer("davebulaval/MeaningBERT", subfolder="large")
+print(scorer.score(documents, simplifications))
+```
 or using our HuggingFace Metric module
 
 ```python
@@ -151,11 +192,47 @@ documents = ["He wanted to make them pay.", "This sandwich looks delicious.", "H
 simplifications = ["He wanted to make them pay.", "This sandwich looks delicious.",
                    "Whatever, whenever, this is a sentence."]
 
+# Three checkpoints, named as the second argument of evaluate.load.
+# "large" is the default: deberta-v3-large, the recommended one.
 meaning_bert = evaluate.load("davebulaval/meaningbert")
+meaning_bert = evaluate.load("davebulaval/meaningbert", "large")  # the same thing
+
+# "base" is bert-base-uncased: 3.2x faster on GPU, 4.8x on CPU, a quarter of the memory.
+meaning_bert_base = evaluate.load("davebulaval/meaningbert", "base")
+
+# "v1" is the model published with the 2023 article, unchanged.
+meaning_bert_v1 = evaluate.load("davebulaval/meaningbert", "v1")
 
 print(meaning_bert.compute(references=documents, predictions=simplifications))
 ```
 
+### Which checkpoint to load
+
+The two v2 checkpoints are **subfolders of the same repository**,
+[davebulaval/MeaningBERT](https://huggingface.co/davebulaval/MeaningBERT), whose root is still the v1 model. Nothing
+already loading that repository changes behaviour.
+
+```python
+from transformers import AutoModelForSequenceClassification
+
+model = AutoModelForSequenceClassification.from_pretrained("davebulaval/MeaningBERT", subfolder="large")
+```
+
+| name | encoder | Pearson *r* | identical pairs above 95 | 100 pairs, GPU / CPU | weights |
+|---|---|---|---|---|---|
+| **`large`** (default) | `deberta-v3-large` | **0.704 ± 0.009** | **97.1 % ± 1.4** | 1.16 s / 9.49 s | 1740 MB |
+| `base`<sup>†</sup> | `bert-base-uncased` | 0.637 | 70.4 % | **0.36 s / 1.99 s** | **438 MB** |
+| `v1` | `bert-base-uncased` | 0.323 | 0 % | 0.36 s / 1.99 s | 438 MB |
+
+<sup>†</sup> `base` is **not published yet**: its figures come from a single seed, against ten for `large`, and loading it
+raises rather than falling back on another checkpoint. A name that does not exist raises too, so a typo cannot silently
+swap the model under you.
+
+Load `base` when you score at volume or have no GPU, `large` when a wrong score costs you something. On a GPU the speed
+argument is weak, see [A middle option worth knowing about](#a-middle-option-worth-knowing-about).
+
+The numbers are detailed in [MeaningBERT v2: which checkpoint to use](#meaningbert-v2-which-checkpoint-to-use).
+
 
 ------------------