Sentence Similarity
sentence-transformers
ONNX
Safetensors
Morisyen
English
French
xlm-roberta
feature-extraction
mauritian-creole
kreol-morisien
matryoshka
mteb
Eval Results (legacy)
text-embeddings-inference
Instructions to use Singaraj/morisien-embed with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use Singaraj/morisien-embed with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("Singaraj/morisien-embed") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Notebooks
- Google Colab
- Kaggle
Drop unscripted figures and the withdrawn bound
Browse files
README.md
CHANGED
|
@@ -158,8 +158,8 @@ negative set; Creole→English test ndcg@10 across seeds: **0.9653 ± 0.0002** (
|
|
| 158 |
|
| 159 |
Every number in the tables above is reproducible from the
|
| 160 |
[training repository](https://github.com/LK-maker-007/morisien-embed) (Matryoshka figures via
|
| 161 |
-
`scripts/evaluate.py --truncate-dim`). The Haitian-proximity
|
| 162 |
-
|
| 163 |
|
| 164 |
## MTEB
|
| 165 |
|
|
@@ -208,9 +208,9 @@ trained on the MorisienMT corpus this split is drawn from, so MTEB records the r
|
|
| 208 |
literature; highly informal or technical registers are less covered.
|
| 209 |
- **Small evaluation universe.** Retrieval is measured over ~1,000-passage corpora, standard for
|
| 210 |
bitext benchmarks, but absolute scores would be lower against web-scale corpora.
|
| 211 |
-
- **
|
| 212 |
-
|
| 213 |
-
|
| 214 |
- **Haitian Creole proximity.** Like every multilingual embedder we tested, the model embeds Haitian
|
| 215 |
Creole close to Mauritian Creole. Measured on the 1,012 aligned mfe/hat/eng FLORES+ devtest
|
| 216 |
triplets (`scripts/probe_haitian.py`): with every same-meaning Haitian twin injected into the
|
|
@@ -218,19 +218,17 @@ trained on the MorisienMT corpus this split is drawn from, so MTEB records the r
|
|
| 218 |
Asked instead to tell the two creoles apart (is the English sentence closer to its Mauritian or
|
| 219 |
its Haitian translation?), the fine-tune picks Mauritian 709/1012 times vs LaBSE's 351/1012.
|
| 220 |
Wrong-meaning Haitian text is never confused; mixed mfe/hat corpora will still degrade retrieval.
|
| 221 |
-
- **
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
Spearman ≈ 0.79 vs the base model's ≈ 0.85. Use a general model for English-only workloads; this
|
| 225 |
-
model is for Creole and Creole↔{English, French} work.
|
| 226 |
- **Long inputs are truncated** at the encoder's maximum sequence length; chunk long documents
|
| 227 |
before embedding.
|
| 228 |
- **Protocol note.** During recipe development the held-out test score was printed at the end of each
|
| 229 |
-
training run, so recipe selection had test visibility
|
| 230 |
-
resulting optimism at
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
|
| 234 |
half of the leak check was added after the released run; it verifiably leaves the training set
|
| 235 |
byte-identical, since the sources were already disjoint at that level.
|
| 236 |
- **Orthographic variation.** Training data mixes pre- and post-2011 (Lortograf Kreol Morisien)
|
|
|
|
| 158 |
|
| 159 |
Every number in the tables above is reproducible from the
|
| 160 |
[training repository](https://github.com/LK-maker-007/morisien-embed) (Matryoshka figures via
|
| 161 |
+
`scripts/evaluate.py --truncate-dim`). The Haitian-proximity figures under Limitations come from
|
| 162 |
+
`scripts/probe_haitian.py`.
|
| 163 |
|
| 164 |
## MTEB
|
| 165 |
|
|
|
|
| 208 |
literature; highly informal or technical registers are less covered.
|
| 209 |
- **Small evaluation universe.** Retrieval is measured over ~1,000-passage corpora, standard for
|
| 210 |
bitext benchmarks, but absolute scores would be lower against web-scale corpora.
|
| 211 |
+
- **The margin over LaBSE is in-domain only.** MorisienMT and Kreyòl-MT overlap heavily. On the
|
| 212 |
+
harder xSIM++ pool built from FLORES+, this model does not beat untrained LaBSE; see the preprint
|
| 213 |
+
and [morisien-embed-v1.5](https://huggingface.co/Singaraj/morisien-embed-v1.5).
|
| 214 |
- **Haitian Creole proximity.** Like every multilingual embedder we tested, the model embeds Haitian
|
| 215 |
Creole close to Mauritian Creole. Measured on the 1,012 aligned mfe/hat/eng FLORES+ devtest
|
| 216 |
triplets (`scripts/probe_haitian.py`): with every same-meaning Haitian twin injected into the
|
|
|
|
| 218 |
Asked instead to tell the two creoles apart (is the English sentence closer to its Mauritian or
|
| 219 |
its Haitian translation?), the fine-tune picks Mauritian 709/1012 times vs LaBSE's 351/1012.
|
| 220 |
Wrong-meaning Haitian text is never confused; mixed mfe/hat corpora will still degrade retrieval.
|
| 221 |
+
- **Not measured here:** pure-English quality after fine-tuning, and behaviour on ALL-CAPS text. Use
|
| 222 |
+
a general model for English-only workloads; this model is for Creole and Creole↔{English, French}
|
| 223 |
+
work.
|
|
|
|
|
|
|
| 224 |
- **Long inputs are truncated** at the encoder's maximum sequence length; chunk long documents
|
| 225 |
before embedding.
|
| 226 |
- **Protocol note.** During recipe development the held-out test score was printed at the end of each
|
| 227 |
+
training run, so recipe selection had test visibility. An earlier version of this card bounded the
|
| 228 |
+
resulting optimism at about 0.01 ndcg; that bound is withdrawn, because no committed script produces
|
| 229 |
+
it. The 3-seed replication was run after the recipe was frozen. Leak filtering reserves the Creole
|
| 230 |
+
side of every evaluation pair; English/French target texts are not reserved, so a benchmark passage
|
| 231 |
+
can also occur in training as the translation of a different Creole sentence. The accent-insensitive
|
| 232 |
half of the leak check was added after the released run; it verifiably leaves the training set
|
| 233 |
byte-identical, since the sources were already disjoint at that level.
|
| 234 |
- **Orthographic variation.** Training data mixes pre- and post-2011 (Lortograf Kreol Morisien)
|