Singaraj commited on
Commit
25d71b3
·
verified ·
1 Parent(s): d50c668

Drop unscripted figures and the withdrawn bound

Browse files
Files changed (1) hide show
  1. README.md +13 -15
README.md CHANGED
@@ -158,8 +158,8 @@ negative set; Creole→English test ndcg@10 across seeds: **0.9653 ± 0.0002** (
158
 
159
  Every number in the tables above is reproducible from the
160
  [training repository](https://github.com/LK-maker-007/morisien-embed) (Matryoshka figures via
161
- `scripts/evaluate.py --truncate-dim`). The Haitian-proximity and case-sensitivity figures under
162
- Limitations come from an internal adversarial audit of the released checkpoint.
163
 
164
  ## MTEB
165
 
@@ -208,9 +208,9 @@ trained on the MorisienMT corpus this split is drawn from, so MTEB records the r
208
  literature; highly informal or technical registers are less covered.
209
  - **Small evaluation universe.** Retrieval is measured over ~1,000-passage corpora, standard for
210
  bitext benchmarks, but absolute scores would be lower against web-scale corpora.
211
- - **One distribution family.** MorisienMT and Kreyòl-MT overlap heavily, and the only fully
212
- independent evaluation domain for `mfe` (FLORES+) is saturated at this corpus size, so the margin
213
- over LaBSE is demonstrated in-domain only.
214
  - **Haitian Creole proximity.** Like every multilingual embedder we tested, the model embeds Haitian
215
  Creole close to Mauritian Creole. Measured on the 1,012 aligned mfe/hat/eng FLORES+ devtest
216
  triplets (`scripts/probe_haitian.py`): with every same-meaning Haitian twin injected into the
@@ -218,19 +218,17 @@ trained on the MorisienMT corpus this split is drawn from, so MTEB records the r
218
  Asked instead to tell the two creoles apart (is the English sentence closer to its Mauritian or
219
  its Haitian translation?), the fine-tune picks Mauritian 709/1012 times vs LaBSE's 351/1012.
220
  Wrong-meaning Haitian text is never confused; mixed mfe/hat corpora will still degrade retrieval.
221
- - **Case sensitivity.** ALL-CAPS text embeds measurably differently from its lower-case form
222
- (cosine ≈ 0.81 to the same sentence); caps-heavy text retrieves worse.
223
- - **English-only regression.** Fine-tuning costs some pure-English semantic quality: STS-b test
224
- Spearman ≈ 0.79 vs the base model's ≈ 0.85. Use a general model for English-only workloads; this
225
- model is for Creole and Creole↔{English, French} work.
226
  - **Long inputs are truncated** at the encoder's maximum sequence length; chunk long documents
227
  before embedding.
228
  - **Protocol note.** During recipe development the held-out test score was printed at the end of each
229
- training run, so recipe selection had test visibility; an internal adversarial audit bounded the
230
- resulting optimism at ≤ ~0.01 ndcg. The 3-seed replication was run after the recipe was frozen. Leak filtering
231
- reserves the Creole side of every evaluation pair; English/French target texts are not reserved, and
232
- an audit found 1 of 999 benchmark passages also occurring in training as the translation of a
233
- different Creole sentence (dropping it moves ndcg@10 by less than 0.0001). The accent-insensitive
234
  half of the leak check was added after the released run; it verifiably leaves the training set
235
  byte-identical, since the sources were already disjoint at that level.
236
  - **Orthographic variation.** Training data mixes pre- and post-2011 (Lortograf Kreol Morisien)
 
158
 
159
  Every number in the tables above is reproducible from the
160
  [training repository](https://github.com/LK-maker-007/morisien-embed) (Matryoshka figures via
161
+ `scripts/evaluate.py --truncate-dim`). The Haitian-proximity figures under Limitations come from
162
+ `scripts/probe_haitian.py`.
163
 
164
  ## MTEB
165
 
 
208
  literature; highly informal or technical registers are less covered.
209
  - **Small evaluation universe.** Retrieval is measured over ~1,000-passage corpora, standard for
210
  bitext benchmarks, but absolute scores would be lower against web-scale corpora.
211
+ - **The margin over LaBSE is in-domain only.** MorisienMT and Kreyòl-MT overlap heavily. On the
212
+ harder xSIM++ pool built from FLORES+, this model does not beat untrained LaBSE; see the preprint
213
+ and [morisien-embed-v1.5](https://huggingface.co/Singaraj/morisien-embed-v1.5).
214
  - **Haitian Creole proximity.** Like every multilingual embedder we tested, the model embeds Haitian
215
  Creole close to Mauritian Creole. Measured on the 1,012 aligned mfe/hat/eng FLORES+ devtest
216
  triplets (`scripts/probe_haitian.py`): with every same-meaning Haitian twin injected into the
 
218
  Asked instead to tell the two creoles apart (is the English sentence closer to its Mauritian or
219
  its Haitian translation?), the fine-tune picks Mauritian 709/1012 times vs LaBSE's 351/1012.
220
  Wrong-meaning Haitian text is never confused; mixed mfe/hat corpora will still degrade retrieval.
221
+ - **Not measured here:** pure-English quality after fine-tuning, and behaviour on ALL-CAPS text. Use
222
+ a general model for English-only workloads; this model is for Creole and Creole↔{English, French}
223
+ work.
 
 
224
  - **Long inputs are truncated** at the encoder's maximum sequence length; chunk long documents
225
  before embedding.
226
  - **Protocol note.** During recipe development the held-out test score was printed at the end of each
227
+ training run, so recipe selection had test visibility. An earlier version of this card bounded the
228
+ resulting optimism at about 0.01 ndcg; that bound is withdrawn, because no committed script produces
229
+ it. The 3-seed replication was run after the recipe was frozen. Leak filtering reserves the Creole
230
+ side of every evaluation pair; English/French target texts are not reserved, so a benchmark passage
231
+ can also occur in training as the translation of a different Creole sentence. The accent-insensitive
232
  half of the leak check was added after the released run; it verifiably leaves the training set
233
  byte-identical, since the sources were already disjoint at that level.
234
  - **Orthographic variation.** Training data mixes pre- and post-2011 (Lortograf Kreol Morisien)