--- language: - uk license: mit library_name: model2vec base_model: BAAI/bge-m3 pipeline_tag: sentence-similarity tags: - model2vec - static-embeddings - sentence-transformers - embeddings - retrieval - ukrainian datasets: - wikimedia/wikipedia - HuggingFaceFW/fineweb-2 --- # ukr-static-m3-64k A static embedding model for Ukrainian: 64,003 tokens × 256 dimensions, 32.8 MB of FP16 weights (about 35 MB with the tokenizer). It was distilled from [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) with [Model2Vec](https://github.com/MinishLab/model2vec) and then trained with [Tokenlearn](https://github.com/MinishLab/tokenlearn). Encoding a text is a table lookup and an average, so it runs on a CPU in about 0.1 ms per query and needs no GPU. On three Ukrainian retrieval sets it scores 0.07–0.12 nDCG@10 higher than [potion-multilingual-128M](https://huggingface.co/minishlab/potion-multilingual-128M), with weights one-eighth the size at the same precision. It does not replace BM25 for keyword-style questions; see [Results](#results). ## Usage ```python from model2vec import StaticModel model = StaticModel.from_pretrained("SyntaxSpirits/ukr-static-m3-64k") embeddings = model.encode(["Київ — столиця України.", "Як оформити закордонний паспорт?"]) ``` With sentence-transformers: ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("SyntaxSpirits/ukr-static-m3-64k").float() embeddings = model.encode(["Київ — столиця України."]) ``` Keep the `.float()`: the weights are stored in FP16, and without it sentence-transformers returns NaN for inputs that contain no known tokens, such as an empty string. Embeddings are L2-normalised, so a dot product is the cosine similarity. Inputs that leave no known tokens, such as an empty string or "!!!", get a zero vector. Long texts are truncated. Both loaders cut at 512 tokens (set in `tokenizer.json`). Model2Vec by default also cuts at 3,584 characters (512 × the median token length of 7) before tokenizing; in the evaluations below this changed the tokens of no document, for this model or POTION. To embed whole texts with Model2Vec, pass `max_length=None` to `model.encode`; otherwise split long documents into chunks. ## Results nDCG@10 for question → passage retrieval. | Model | Weights | Belebele uk | WebFAQ uk | UA-SQuAD val | |---|---|---|---|---| | **ukr-static-m3-64k** | 32.8 MB (FP16) | **0.776** | **0.434** | **0.684** | | potion-multilingual-128M | 512 MB (FP32) | 0.652 | 0.344 | 0.613 | | static-similarity-mrl-multilingual-v1 | 434 MB (FP32) | 0.617 | 0.352 | – | | BM25 | – | 0.693 | 0.490 | 0.761 | | ukr-static-m3-64k + BM25 | – | 0.770 | 0.504 | 0.765 | | potion-multilingual-128M + BM25 | – | 0.716 | 0.462 | 0.734 | | BAAI/bge-m3 (teacher) | 2.27 GB (FP32) | 0.937 | – | – | - Belebele `ukr_Cyrl`: 900 questions, 488 passages. WebFAQ `ukr` test: 6,851 questions, 68,501 answers. UA-SQuAD `val`: 5,726 answerable questions, 1,160 paragraphs. - Differences from the strongest other static model, with 95% bootstrap intervals: Belebele +0.124 [+0.100, +0.147] (POTION), WebFAQ +0.082 [+0.074, +0.091] (static-similarity-mrl), UA-SQuAD +0.071 [+0.060, +0.081] (POTION). - "+ BM25" is reciprocal-rank fusion (k = 60, top 100 from each system, no tuning). BM25 is Okapi with default parameters on lowercased word tokens. Tied scores are ordered by document index. - Belebele and WebFAQ were used during development (choice of vocabulary and the decision to train). UA-SQuAD was run with a protocol fixed beforehand. 2 of its 1,160 paragraphs share an 8-word sequence with the training text; leaving out their 10 questions changes no score by more than 0.001. - These are the scores of this release. An earlier build had a tokenizer bug: the unknown-token id was shared with the subword `s`. It was fixed and the model retrained before release. All three sets, UA-SQuAD included, were evaluated with both builds, and no score differs by more than 0.0015 nDCG@10. - Size: at FP16, POTION's weights would be 256 MB, 7.8 times this model's. - Speed and memory on an Apple M1 Pro CPU, encoding the 900 Belebele questions one at a time after warm-up: p95 0.11 ms per query (POTION 0.12–0.14 ms). The model loads in 0.3 s (POTION 3 s). Peak memory of the Python process is about 360 MiB (POTION about 2.0 GiB), of which about 150 MiB is Python and the `datasets` library. ## Training 1. **Vocabulary.** The 32,000 most frequent lowercased Ukrainian words in 60 M characters of Ukrainian Wikipedia (dump 20231101) and FineWeb-2 (`ukr_Cyrl`); only Cyrillic letter sequences were counted. Added to them are the 32,000 BGE-M3 subword tokens most frequent in the same text, so unseen word forms still get vectors, plus a dedicated unknown token with a zero vector. Whole words get the highest Unigram score, so the tokenizer prefers them over subword splits. Of the 9,711 documents used for counting, 86 share an 8-word sequence with WebFAQ, 1 with UA-SQuAD and none with Belebele; they were not removed from the counts. 2. **Distillation.** Each token was embedded by BGE-M3 (mean pooling), reduced to 256 dimensions with PCA and weighted by corpus frequency (SIF, 1e-4). 3. **Tokenlearn.** 119,430 chunks of up to 120 words from the same sources; chunks that share an 8-word sequence with Belebele or WebFAQ were removed. Target: the mean BGE-M3 token embedding of each chunk, reduced to 256 dimensions by a PCA fitted on the training chunks. Unknown tokens were removed from the inputs, as Model2Vec does at inference. Tokenlearn defaults: AdamW, learning rate 1e-3, batch 256, MSE loss, early stopping on the last 10,000 chunks. ## Limitations - Input is lowercased, word order is ignored, and long texts are truncated (see [Usage](#usage)). - On WebFAQ and UA-SQuAD, BM25 alone scores higher than this model, and fusing the two adds only +0.014 and +0.005 nDCG@10. For keyword-style questions, use it alongside BM25, not instead of it. - It is far behind its teacher (0.776 vs 0.937 on Belebele). - The vocabulary was built for Ukrainian. Other languages, including Russian and English, are covered only by subwords seen in Ukrainian text and were not evaluated. - Only retrieval was evaluated; similarity, clustering and classification were not. - UA-SQuAD is a Ukrainian translation of the SQuAD 2.0 dev set, so its questions share many words with their passages, which favours BM25. ## License MIT (see `LICENSE`). The model is derived from BAAI/bge-m3 (MIT) and uses its XLM-RoBERTa tokenizer (MIT); their notices are in `NOTICE`. Training text came from Ukrainian Wikipedia (CC BY-SA 4.0) and FineWeb-2 (ODC-By 1.0, also subject to the Common Crawl Terms of Use). Contains information from FineWeb-2, which is made available under the ODC Attribution License. No training passages are included: the release contains token vectors and a vocabulary of single words and subword pieces. ## Citation Oleksandr Kholodniak (ORCID [0009-0000-1012-7361](https://orcid.org/0009-0000-1012-7361)). ```bibtex @misc{kholodniak2026ukrstatic, author = {Kholodniak, Oleksandr}, title = {ukr-static-m3-64k: a static embedding model for Ukrainian retrieval}, year = {2026}, howpublished = {\url{https://huggingface.co/SyntaxSpirits/ukr-static-m3-64k}} } ```