BorisTM/loss-guided-static-multi

Loss-Guided Static Multilingual Sentence Embeddings

Overview

A static multilingual sentence encoder: one embedding table, mean pooling, no Transformer at inference. Its vocabulary was extended with loss-guided vocabulary discovery: during contrastive training, adjacent token pairs are scored by how much the sentence objective would suffer without them, and the useful pairs receive their own embedding rows. The base tokenizer stays unchanged. The model is trained on 176 languages and is intended for CPU-efficient semantic similarity, retrieval, clustering, and classification.

Model Specifications

• Embedding dimension: 1,024
• Base vocabulary: 257,914 rows (mmBERT tokenizer + 1,914 language markers)
• Learned pair rows: 56,071
• Parameters: 321.7M (embedding table only), FP32 safetensors, 1.29 GB
• Training languages: 176
• Inference: table lookup and mean pooling; requires a language marker at the start of each text

Performance

MTEB(Multilingual, v2), 131 tasks, primary score (Mean over tasks) 50.13, Mean over task types 43.45:

Task type	Count	Mean score
Bitext mining	13	53.90
Classification	43	52.87
Clustering	16	38.91
Instruction reranking	3	-1.04
Multilabel classification	5	17.64
Pair classification	11	72.69
Reranking	6	49.42
Retrieval	18	41.64
STS	16	64.99

Cross-lingual retrieval (macro F1 over language pairs): Tatoeba 57.26 (106 pairs), BUCC 98.12 (4 pairs), Flores 63.70 (193 pairs).

Installation & Usage

Every input must start with a language marker __<iso639-3>_<script>__, for example __eng_Latn__, __rus_Cyrl__, __arb_Arab__, __cmn_Hani__. A text without a marker raises an error. The repository ships the custom embedding module, so trust_remote_code=True is required.

pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BorisTM/loss-guided-static-multi", trust_remote_code=True)

sentences = [
    "__eng_Latn__ The cat sleeps on the sofa.",
    "__rus_Cyrl__ Кошка спит на диване.",
    "__deu_Latn__ Die Katze schläft auf dem Sofa.",
    "__eng_Latn__ The stock market fell sharply today.",
]

embeddings = model.encode(sentences, normalize_embeddings=True)
similarities = model.similarity(embeddings, embeddings)
# [[1.00, 0.44, 0.80, 0.05],
#  [0.44, 1.00, 0.41, 0.06],
#  [0.80, 0.41, 1.00, 0.08],
#  [0.05, 0.06, 0.08, 1.00]]

License: Apache 2.0