Model card

README.md

@@ -2,101 +2,86 @@
 license: apache-2.0
 library_name: sentence-transformers
 pipeline_tag: sentence-similarity
+language:
+  - multilingual
 tags:
   - sentence-transformers
-  - sentence-embeddings
-  - multilingual
+  - sentence-similarity
   - static-embeddings
-  - research
+  - multilingual
 ---
 
-# StaRSE Multilingual Q42 25k
+# Loss-Guided Static Multilingual Sentence Embeddings
 
-Private technical research checkpoint. This is the best completed multilingual
-StaRSE checkpoint as of 2026-08-11; it is not a polished public release.
+## Overview
 
-## Model
+A static multilingual sentence encoder: one embedding table, mean pooling, no Transformer at inference. Its vocabulary was extended with **loss-guided vocabulary discovery**: during contrastive training, adjacent token pairs are scored by how much the sentence objective would suffer without them, and the useful pairs receive their own embedding rows. The base tokenizer stays unchanged. The model is trained on 176 languages and is intended for CPU-efficient semantic similarity, retrieval, clustering, and classification.
 
-- Architecture: language-conditioned static additive sentence encoder.
-- Output dimension: 1,024.
-- Base token rows: 257,914.
-- Learned adjacent-pair rows: 56,071.
-- Explicit language markers: 1,914.
-- Training population: 176 languages.
-- Training history: the registered 20,000-update Q42 model followed by a
-  separate 5,000-update branch with a fresh AdamW optimizer, inverse-square-root
-  schedule, 1,000 warmup updates, and peak learning rate 1e-4.
-- Model tensor SHA-256:
-  `c28eaa242185aeccbd5ff643bb517b5a93d10b9d5a71c85dd03373ca4b2ad51a`.
+Paper: *Loss-Guided Vocabulary Discovery for Multilingual Static Encoders* (MRL Workshop @ EMNLP 2026). Code: [BlessedTatonka/loss-guided-vocab](https://github.com/BlessedTatonka/loss-guided-vocab).
 
-The checkpoint requires an explicit language marker at the beginning of every
-text. Language IDs use ISO 639-3 plus ISO 15924 script, for example
-`eng_Latn`, `rus_Cyrl`, and `arb_Arab`.
+## Model Specifications
 
-## Usage
+- **Embedding dimension**: 1,024
+- **Base vocabulary**: 257,914 rows (mmBERT tokenizer + 1,914 language markers)
+- **Learned pair rows**: 56,071
+- **Parameters**: 321.7M (embedding table only), FP32 safetensors, 1.29 GB
+- **Training languages**: 176
+- **Inference**: table lookup and mean pooling; requires a language marker at the start of each text
 
-The repository includes the exact `starse` Python module needed to load the
-custom SentenceTransformers component. Loading executes that repository code,
-so keep `trust_remote_code=True` only for a revision you have inspected and
-pinned.
-
-```python
-from sentence_transformers import SentenceTransformer
+## Performance
 
-model = SentenceTransformer(
-    "BorisTM/starse-multilingual-q42-25k",
-    trust_remote_code=True,
-)
+MTEB(Multilingual, v2), 131 tasks, primary score (Mean over tasks) **50.13**, Mean over task types 43.45:
 
-language = "rus_Cyrl"
-sentences = [
-    f"__{language}__ Первое предложение",
-    f"__{language}__ Второе предложение",
-]
-embeddings = model.encode(sentences, normalize_embeddings=True)
-similarities = embeddings @ embeddings.T
-```
+| Task type | Count | Mean score |
+|-----------|-------|-----------|
+| Bitext mining | 13 | 53.90 |
+| Classification | 43 | 52.87 |
+| Clustering | 16 | 38.91 |
+| Instruction reranking | 3 | -1.04 |
+| Multilabel classification | 5 | 17.64 |
+| Pair classification | 11 | 72.69 |
+| Reranking | 6 | 49.42 |
+| Retrieval | 18 | 41.64 |
+| STS | 16 | 64.99 |
 
-Omitting the language marker is a hard error rather than a silent fallback.
-The marker must match one of the entries in `language_conditioning.json`.
+Cross-lingual retrieval (macro F1 over language pairs): Tatoeba 57.26 (106 pairs), BUCC 98.12 (4 pairs), Flores 63.70 (193 pairs).
 
-## Evaluation
+## Installation & Usage
 
-The exact checkpoint completed all 131 tasks in `MTEB(Multilingual, v2)` with
-MTEB 2.12.30 and zero failures:
+Every input must start with a language marker `__<iso639-3>_<script>__`, for example `__eng_Latn__`, `__rus_Cyrl__`, `__arb_Arab__`, `__cmn_Hani__`. A text without a marker raises an error. The repository ships the custom embedding module, so `trust_remote_code=True` is required.
 
-- Mean(Task): 50.13;
-- Mean(TaskType): 43.45.
-
-On the frozen compact multilingual suite it scores:
+```python
+pip install -U sentence-transformers
 
-- Tatoeba, 106 subsets: 57.26;
-- BUCC.v2, 4 subsets: 98.12;
-- FloresBitextMining, 193 subsets: 63.70;
-- unweighted three-task macro: 73.03.
+from sentence_transformers import SentenceTransformer
 
-The repository includes machine-readable summaries under `evaluation/` and an
-exact file manifest under `provenance/`. These are results from one trained
-checkpoint. No training noise floor was measured, so small differences must
-not be interpreted as established effects.
+model = SentenceTransformer("BorisTM/loss-guided-static-multi", trust_remote_code=True)
 
-## Scope and limitations
+sentences = [
+    "__eng_Latn__ The cat sleeps on the sofa.",
+    "__rus_Cyrl__ Кошка спит на диване.",
+    "__deu_Latn__ Die Katze schläft auf dem Sofa.",
+    "__eng_Latn__ The stock market fell sharply today.",
+]
 
-- Inference requires a known language-script ID; automatic language ID and
-  code-switching were not evaluated.
-- Quality varies substantially across languages, scripts, and resource levels.
-- The encoder uses a fixed tokenizer and adjacent-token pair table. It is not a
-  contextual Transformer and does not produce contextual token embeddings.
-- Training data may contain personal, offensive, copyrighted, or socially
-  biased material. Do not use the model for profiling, sensitive-attribute
-  inference, or fully automated high-stakes decisions.
+embeddings = model.encode(sentences, normalize_embeddings=True)
+similarities = model.similarity(embeddings, embeddings)
+# [[1.00, 0.44, 0.80, 0.05],
+#  [0.44, 1.00, 0.41, 0.06],
+#  [0.80, 0.41, 1.00, 0.08],
+#  [0.05, 0.06, 0.08, 1.00]]
+```
 
-## Internal provenance
+## Citation
 
-- Run: `q42-posthoc-lr1e4-5k-v1`.
-- Local source checkpoint:
-  `outputs/q42-posthoc-lr1e4-5k-v1/main/checkpoint-5000`.
-- Cumulative model updates: 25,000.
-- Full evaluation summary protocol:
-  `q42-posthoc-checkpoint5000-full-mmteb-summary-v1`.
+```
+@inproceedings{malashenko2026lossguided,
+  title = {Loss-Guided Vocabulary Discovery for Multilingual Static Encoders},
+  author = {Malashenko, Boris and Jarsky, Ivan and Stumpf, Sviatoslav A. and Efimova, Valeria},
+  booktitle = {Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL)},
+  year = {2026},
+  url = {https://huggingface.co/BorisTM/loss-guided-static-multi}
+}
+```
 
+**License**: Apache 2.0