| --- |
| language: |
| - uk |
| license: mit |
| library_name: model2vec |
| base_model: BAAI/bge-m3 |
| pipeline_tag: sentence-similarity |
| tags: |
| - model2vec |
| - static-embeddings |
| - sentence-transformers |
| - embeddings |
| - retrieval |
| - ukrainian |
| datasets: |
| - wikimedia/wikipedia |
| - HuggingFaceFW/fineweb-2 |
| --- |
| |
| # ukr-static-m3-64k |
|
|
| A static embedding model for Ukrainian: 64,003 tokens × 256 dimensions, 32.8 MB of FP16 weights (about 35 MB with |
| the tokenizer). It was distilled from [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) with |
| [Model2Vec](https://github.com/MinishLab/model2vec) and then trained with |
| [Tokenlearn](https://github.com/MinishLab/tokenlearn). Encoding a text is a table lookup and an average, so it runs |
| on a CPU in about 0.1 ms per query and needs no GPU. |
|
|
| On three Ukrainian retrieval sets it scores 0.07–0.12 nDCG@10 higher than |
| [potion-multilingual-128M](https://huggingface.co/minishlab/potion-multilingual-128M), with weights one-eighth the |
| size at the same precision. It does not replace BM25 for keyword-style questions; see [Results](#results). |
|
|
| ## Usage |
|
|
| ```python |
| from model2vec import StaticModel |
| |
| model = StaticModel.from_pretrained("SyntaxSpirits/ukr-static-m3-64k") |
| embeddings = model.encode(["Київ — столиця України.", "Як оформити закордонний паспорт?"]) |
| ``` |
|
|
| With sentence-transformers: |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| |
| model = SentenceTransformer("SyntaxSpirits/ukr-static-m3-64k").float() |
| embeddings = model.encode(["Київ — столиця України."]) |
| ``` |
|
|
| Keep the `.float()`: the weights are stored in FP16, and without it sentence-transformers returns NaN for inputs |
| that contain no known tokens, such as an empty string. |
|
|
| Embeddings are L2-normalised, so a dot product is the cosine similarity. Inputs that leave no known tokens, such as |
| an empty string or "!!!", get a zero vector. |
|
|
| Long texts are truncated. Both loaders cut at 512 tokens (set in `tokenizer.json`). Model2Vec by default also cuts |
| at 3,584 characters (512 × the median token length of 7) before tokenizing; in the evaluations below this changed |
| the tokens of no document, for this model or POTION. To embed whole texts with Model2Vec, pass `max_length=None` to |
| `model.encode`; otherwise split long documents into chunks. |
|
|
| ## Results |
|
|
| nDCG@10 for question → passage retrieval. |
|
|
| | Model | Weights | Belebele uk | WebFAQ uk | UA-SQuAD val | |
| |---|---|---|---|---| |
| | **ukr-static-m3-64k** | 32.8 MB (FP16) | **0.776** | **0.434** | **0.684** | |
| | potion-multilingual-128M | 512 MB (FP32) | 0.652 | 0.344 | 0.613 | |
| | static-similarity-mrl-multilingual-v1 | 434 MB (FP32) | 0.617 | 0.352 | – | |
| | BM25 | – | 0.693 | 0.490 | 0.761 | |
| | ukr-static-m3-64k + BM25 | – | 0.770 | 0.504 | 0.765 | |
| | potion-multilingual-128M + BM25 | – | 0.716 | 0.462 | 0.734 | |
| | BAAI/bge-m3 (teacher) | 2.27 GB (FP32) | 0.937 | – | – | |
|
|
| - Belebele `ukr_Cyrl`: 900 questions, 488 passages. WebFAQ `ukr` test: 6,851 questions, 68,501 answers. |
| UA-SQuAD `val`: 5,726 answerable questions, 1,160 paragraphs. |
| - Differences from the strongest other static model, with 95% bootstrap intervals: Belebele +0.124 |
| [+0.100, +0.147] (POTION), WebFAQ +0.082 [+0.074, +0.091] (static-similarity-mrl), UA-SQuAD +0.071 |
| [+0.060, +0.081] (POTION). |
| - "+ BM25" is reciprocal-rank fusion (k = 60, top 100 from each system, no tuning). BM25 is Okapi with default |
| parameters on lowercased word tokens. Tied scores are ordered by document index. |
| - Belebele and WebFAQ were used during development (choice of vocabulary and the decision to train). UA-SQuAD was |
| run with a protocol fixed beforehand. 2 of its 1,160 paragraphs share an 8-word sequence with the training text; |
| leaving out their 10 questions changes no score by more than 0.001. |
| - These are the scores of this release. An earlier build had a tokenizer bug: the unknown-token id was shared with |
| the subword `s`. It was fixed and the model retrained before release. All three sets, UA-SQuAD included, were |
| evaluated with both builds, and no score differs by more than 0.0015 nDCG@10. |
| - Size: at FP16, POTION's weights would be 256 MB, 7.8 times this model's. |
| - Speed and memory on an Apple M1 Pro CPU, encoding the 900 Belebele questions one at a time after warm-up: p95 |
| 0.11 ms per query (POTION 0.12–0.14 ms). The model loads in 0.3 s (POTION 3 s). Peak memory of the Python |
| process is about 360 MiB (POTION about 2.0 GiB), of which about 150 MiB is Python and the `datasets` library. |
|
|
| ## Training |
|
|
| 1. **Vocabulary.** The 32,000 most frequent lowercased Ukrainian words in 60 M characters of Ukrainian Wikipedia |
| (dump 20231101) and FineWeb-2 (`ukr_Cyrl`); only Cyrillic letter sequences were counted. Added to them are the |
| 32,000 BGE-M3 subword tokens most frequent in the same text, so unseen word forms still get vectors, plus a |
| dedicated unknown token with a zero vector. Whole words get the highest Unigram score, so the tokenizer prefers |
| them over subword splits. Of the 9,711 documents used for counting, 86 share an 8-word sequence with WebFAQ, |
| 1 with UA-SQuAD and none with Belebele; they were not removed from the counts. |
| 2. **Distillation.** Each token was embedded by BGE-M3 (mean pooling), reduced to 256 dimensions with PCA and |
| weighted by corpus frequency (SIF, 1e-4). |
| 3. **Tokenlearn.** 119,430 chunks of up to 120 words from the same sources; chunks that share an 8-word sequence |
| with Belebele or WebFAQ were removed. Target: the mean BGE-M3 token embedding of each chunk, reduced to 256 |
| dimensions by a PCA fitted on the training chunks. Unknown tokens were removed from the inputs, as Model2Vec |
| does at inference. Tokenlearn defaults: AdamW, learning rate 1e-3, batch 256, MSE loss, early stopping on the |
| last 10,000 chunks. |
|
|
| ## Limitations |
|
|
| - Input is lowercased, word order is ignored, and long texts are truncated (see [Usage](#usage)). |
| - On WebFAQ and UA-SQuAD, BM25 alone scores higher than this model, and fusing the two adds only +0.014 and |
| +0.005 nDCG@10. For keyword-style questions, use it alongside BM25, not instead of it. |
| - It is far behind its teacher (0.776 vs 0.937 on Belebele). |
| - The vocabulary was built for Ukrainian. Other languages, including Russian and English, are covered only by |
| subwords seen in Ukrainian text and were not evaluated. |
| - Only retrieval was evaluated; similarity, clustering and classification were not. |
| - UA-SQuAD is a Ukrainian translation of the SQuAD 2.0 dev set, so its questions share many words with their |
| passages, which favours BM25. |
|
|
| ## License |
|
|
| MIT (see `LICENSE`). The model is derived from BAAI/bge-m3 (MIT) and uses its XLM-RoBERTa tokenizer (MIT); their |
| notices are in `NOTICE`. Training text came from Ukrainian Wikipedia (CC BY-SA 4.0) and FineWeb-2 (ODC-By 1.0, |
| also subject to the Common Crawl Terms of Use). Contains information from FineWeb-2, which is made available under |
| the ODC Attribution License. No training passages are included: the release contains token vectors and a |
| vocabulary of single words and subword pieces. |
|
|
| ## Citation |
|
|
| Oleksandr Kholodniak (ORCID [0009-0000-1012-7361](https://orcid.org/0009-0000-1012-7361)). |
|
|
| ```bibtex |
| @misc{kholodniak2026ukrstatic, |
| author = {Kholodniak, Oleksandr}, |
| title = {ukr-static-m3-64k: a static embedding model for Ukrainian retrieval}, |
| year = {2026}, |
| howpublished = {\url{https://huggingface.co/SyntaxSpirits/ukr-static-m3-64k}} |
| } |
| ``` |
|
|
