| --- |
| license: mit |
| tags: |
| - language-detection |
| - cyrillic |
| - fasttext |
| - text-classification |
| language: |
| - rus |
| - ukr |
| - bel |
| - bul |
| - mkd |
| - srp |
| - kaz |
| - kir |
| - tgk |
| - uzn |
| - tat |
| - bak |
| - chv |
| - sah |
| - tyv |
| - mon |
| - oss |
| - che |
| - udm |
| - mhr |
| - kpv |
| - eng |
| - spa |
| - fra |
| - por |
| - deu |
| - cmn |
| - jpn |
| - hin |
| - ara |
| --- |
| |
| # spellman — Cyrillic-optimized language detection |
|
|
| This is the runtime model for **[spellman](https://github.com/vpermilp/spellman)**, |
| a fast Rust language detector covering 30 languages: the big world |
| languages plus twenty Cyrillic-script languages (Yakut, Tuvan, Udmurt, |
| Komi, Mari, Ossetic, Chechen, …) that most detectors don't know exist. |
| It is built for the text that breaks language detectors — tweets, |
| comments, two-word utterances, and close pairs like Russian/Ukrainian — |
| and runs at ~3.3 µs per document on one CPU core (~0.5 µs across 14 |
| cores; Apple M4 Max). |
|
|
| **7.9 MB, one int8 lookup table.** The trained network folds |
| algebraically into a single table, stored and scored as int8 with one |
| scale per language column; inference is pure lookups, no matmul. |
|
|
| ## Usage |
|
|
| The featurizer lives in the Rust runtime, so the model is consumed |
| through the spellman crate or CLI (not a standalone checkpoint): |
|
|
| ```bash |
| echo "Съешь ещё этих мягких французских булок" | spellman detect --model hf:vpermilp/spellman # rus |
| printf 'Қазақша\nHello\n' | spellman detect --model hf:vpermilp/spellman --lines # kaz / eng |
| ``` |
|
|
| ```rust |
| use spellman_detector::SingleDetector; |
| let mut det = SingleDetector::from_hub(1024)?; |
| let d = det.detect("Күн сайын аҕыйах тыл үөрэт")?; // sah |
| ``` |
|
|
| **One storage format.** The model ships once, at the repo root, as int8 |
| with per-column scales — the format the runtime scores directly. It |
| matched f16 accuracy on every eval (within ±0.02pp) at half the size and |
| ran faster on every path measured, so the earlier `f16/`, `int8-row/`, |
| `int8-col/` and `fp8-*` variants were removed (they remain in this repo's |
| history). |
|
|
| ## Languages (30) |
|
|
| - **Cyrillic (21):** Russian, Ukrainian, Belarusian, Bulgarian, |
| Macedonian, Serbian, Kazakh, Kyrgyz, Tajik, Uzbek (Cyrillic), Tatar, |
| Bashkir, Chuvash, Yakut, Tuvan, Mongolian, Ossetic, Chechen, Udmurt, |
| Meadow Mari, Komi-Zyrian |
| - **Latin (5):** English, Spanish, French, Portuguese, German |
| - **Script-routed (4, no model columns):** Mandarin, Japanese, Hindi, Arabic |
|
|
| ## Accuracy (v16, 2026-10-04) |
|
|
| | eval | accuracy | |
| |---|---| |
| | held-out mix (718,647 rows, all registers) | 98.74% | |
| | held-out mix, texts of ≤20 characters (59,932) | 95.06% | |
| | Tatoeba (37,051, out-of-domain) | 99.08% | |
| | single words / pairs / triples (Tatoeba ladder) | 72.04 / 89.78 / 95.37 | |
| | wild Russian tweets (2,606, label-audited) | 96.55% | |
| | COSMUS gold-labeled wild Russian (2,713, unseen) | 99.15% | |
| | real short utterances (574, ≤19 chars) | 95.82% | |
| | literary Russian (2,000 sentences, held-out novel) | 98.40% | |
| | throughput | ~3.3 µs/doc bulk, ~6.7 µs single on one core; ~0.5 µs/doc on 14 cores (Apple M4 Max) | |
|
|
| Every accuracy row is measured with this artifact through the Rust |
| runtime; throughput was measured on the v14 artifact, which has the same |
| shape. All referee files are held out of training by construction since |
| v16 (the mixer drops any row whose text appears in one). |
|
|
| Against v14 on identical rows: Ukrainian 96.1 → 98.6% (texts of ≤20 |
| characters 92.4 → 97.6%), other languages mistaken for Russian on short |
| texts 0.85 → 0.25%, short Russian comments from an unseen source 88.1 → |
| 93.2%. Wild Russian tweets (96.78 → 96.55) and literary Russian (98.95 → |
| 98.40) are slightly lower. |
|
|
| On identical rows it beats fastText lid.176 and open-LID SOTA GlotLID v3 |
| on the held-out mix (98.74 vs 81.97 and 92.92), on wild Russian tweets |
| (96.55 vs GlotLID's 82.73), short utterances (95.82 vs 71.25) and single |
| words (72.04 vs 43.9), while running ~100× faster. GlotLID keeps a small |
| lead on clean out-of-domain sentences (99.25 vs 99.08), and on the |
| cleaned COSMUS Russian set all three are within 0.6pp (lid.176 99.37, |
| spellman 99.15, GlotLID 98.82). Full tables and methodology: |
| [docs/benchmarks.md](https://github.com/vpermilp/spellman/blob/main/docs/benchmarks.md). |
|
|
| ## Files |
|
|
| - `model.json` — the runtime contract: language order, hash id/seed, |
| n-gram config, canonicalization/lexical flags, and θ — the confidence |
| threshold under which a detection is flagged uncertain (0.67, |
| calibrated so the flag actually predicts errors). |
| - `model.safetensors` — the folded table `P` `[262145×30]` as int8, |
| its per-language-column f32 `scales` `[30]`, and the class `bias`. |
|
|
| Features are canonicalized char 1–5-grams plus whole-word and word-bigram |
| keys, sign-hashed into 2^18 buckets (fmix32). |
|
|
| ## Training data |
|
|
| Trained on a 4.2M-row, 26-language mix published with its byte-exact |
| recipe and per-upstream license table at |
| [vpermilp/spellman](https://huggingface.co/datasets/vpermilp/spellman) |
| (dataset repo type). Sources include FineWeb-2 line-windows, Tatoeba, |
| per-language community and parallel corpora, a wild social-media lane, |
| a verified short-utterance lane, and the six open crawl datasets |
| (`vpermilp/lid-*`: news/library sites + Telegram/VK for |
| sah/tyv/kpv/mhr/oss/udm) built for this project. Since v16 Russian's |
| short texts come from open chat and forum dumps (Telegram dialogues, the |
| Dvach chat, forum messages) instead of a tweet corpus whose `ru` tag |
| turned out to be mostly Ukrainian. Every source passes LID hygiene; twin |
| languages are arbitrated by spelling and by vocabulary (rows whose |
| letters or words belong to a twin language of their label are dropped). |
| No upstream is NC-licensed (2026-08 license audit); the per-source |
| license table, including sources whose license is the uploader's own |
| label, is on the dataset card. |
|
|
| ## Version history |
|
|
| - **v16** (2026-10-04): label-noise cleanup — orthographic and lexical |
| twin gates, a 41k-row tweet lane that was mostly Ukrainian under a |
| Russian label retired, short Russian from chat/forum sources instead, referees held out of training by |
| construction, θ 0.67 by error-detection F1. Same architecture and |
| format as v14. |
| - **2026-10-01**: one storage format — the int8-col table at the root; |
| the f16 / int8-row / int8-col / fp8 variant directories removed. Same |
| v14 weights. |
| - **v14** (2026-08-31): architecture-review retrain — 2^18 buckets, |
| full-length training truncation, θ by error-detection F1; int8-row |
| root artifact. |
| - **v13c** (2026-08-30): the six crawl datasets + per-language cap |
| 32k → 120k (+3.4pp wild Russian, +4.0pp short vs v12). |
| - **v12** (2026-08-24): commercial-clean rebuild — every NC-licensed |
| upstream replaced or row-filtered. |
|
|
| The full measured experiment log (including what failed): |
| [docs/experiments.md](https://github.com/vpermilp/spellman/blob/main/docs/experiments.md). |
|
|
