README.md

---
license: mit
tags:
- language-detection
- cyrillic
- fasttext
- text-classification
language:
- rus
- ukr
- bel
- bul
- mkd
- srp
- kaz
- kir
- tgk
- uzn
- tat
- bak
- chv
- sah
- tyv
- mon
- oss
- che
- udm
- mhr
- kpv
- eng
- spa
- fra
- por
- deu
- cmn
- jpn
- hin
- ara
---

# spellman — Cyrillic-optimized language detection

This is the runtime model for **[spellman](https://github.com/vpermilp/spellman)**,
a fast Rust language detector covering 30 languages: the big world
languages plus twenty Cyrillic-script languages (Yakut, Tuvan, Udmurt,
Komi, Mari, Ossetic, Chechen, …) that most detectors don't know exist.
It is built for the text that breaks language detectors — tweets,
comments, two-word utterances, and close pairs like Russian/Ukrainian —
and runs at ~3.3 µs per document on one CPU core (~0.5 µs across 14
cores; Apple M4 Max).

**7.9 MB, one int8 lookup table.** The trained network folds
algebraically into a single table, stored and scored as int8 with one
scale per language column; inference is pure lookups, no matmul.

## Usage

The featurizer lives in the Rust runtime, so the model is consumed
through the spellman crate or CLI (not a standalone checkpoint):

```bash
echo "Съешь ещё этих мягких французских булок" | spellman detect --model hf:vpermilp/spellman   # rus
printf 'Қазақша\nHello\n' | spellman detect --model hf:vpermilp/spellman --lines               # kaz / eng
```

```rust
use spellman_detector::SingleDetector;
let mut det = SingleDetector::from_hub(1024)?;
let d = det.detect("Күн сайын аҕыйах тыл үөрэт")?;   // sah
```

**One storage format.** The model ships once, at the repo root, as int8
with per-column scales — the format the runtime scores directly. It
matched f16 accuracy on every eval (within ±0.02pp) at half the size and
ran faster on every path measured, so the earlier `f16/`, `int8-row/`,
`int8-col/` and `fp8-*` variants were removed (they remain in this repo's
history).

## Languages (30)

- **Cyrillic (21):** Russian, Ukrainian, Belarusian, Bulgarian,
  Macedonian, Serbian, Kazakh, Kyrgyz, Tajik, Uzbek (Cyrillic), Tatar,
  Bashkir, Chuvash, Yakut, Tuvan, Mongolian, Ossetic, Chechen, Udmurt,
  Meadow Mari, Komi-Zyrian
- **Latin (5):** English, Spanish, French, Portuguese, German
- **Script-routed (4, no model columns):** Mandarin, Japanese, Hindi, Arabic

## Accuracy (v16, 2026-10-04)

| eval | accuracy |
|---|---|
| held-out mix (718,647 rows, all registers) | 98.74% |
| held-out mix, texts of ≤20 characters (59,932) | 95.06% |
| Tatoeba (37,051, out-of-domain) | 99.08% |
| single words / pairs / triples (Tatoeba ladder) | 72.04 / 89.78 / 95.37 |
| wild Russian tweets (2,606, label-audited) | 96.55% |
| COSMUS gold-labeled wild Russian (2,713, unseen) | 99.15% |
| real short utterances (574, ≤19 chars) | 95.82% |
| literary Russian (2,000 sentences, held-out novel) | 98.40% |
| throughput | ~3.3 µs/doc bulk, ~6.7 µs single on one core; ~0.5 µs/doc on 14 cores (Apple M4 Max) |

Every accuracy row is measured with this artifact through the Rust
runtime; throughput was measured on the v14 artifact, which has the same
shape. All referee files are held out of training by construction since
v16 (the mixer drops any row whose text appears in one).

Against v14 on identical rows: Ukrainian 96.1 → 98.6% (texts of ≤20
characters 92.4 → 97.6%), other languages mistaken for Russian on short
texts 0.85 → 0.25%, short Russian comments from an unseen source 88.1 →
93.2%. Wild Russian tweets (96.78 → 96.55) and literary Russian (98.95 →
98.40) are slightly lower.

On identical rows it beats fastText lid.176 and open-LID SOTA GlotLID v3
on the held-out mix (98.74 vs 81.97 and 92.92), on wild Russian tweets
(96.55 vs GlotLID's 82.73), short utterances (95.82 vs 71.25) and single
words (72.04 vs 43.9), while running ~100× faster. GlotLID keeps a small
lead on clean out-of-domain sentences (99.25 vs 99.08), and on the
cleaned COSMUS Russian set all three are within 0.6pp (lid.176 99.37,
spellman 99.15, GlotLID 98.82). Full tables and methodology:
[docs/benchmarks.md](https://github.com/vpermilp/spellman/blob/main/docs/benchmarks.md).

## Files

- `model.json` — the runtime contract: language order, hash id/seed,
  n-gram config, canonicalization/lexical flags, and θ — the confidence
  threshold under which a detection is flagged uncertain (0.67,
  calibrated so the flag actually predicts errors).
- `model.safetensors` — the folded table `P` `[262145×30]` as int8,
  its per-language-column f32 `scales` `[30]`, and the class `bias`.

Features are canonicalized char 1–5-grams plus whole-word and word-bigram
keys, sign-hashed into 2^18 buckets (fmix32).

## Training data

Trained on a 4.2M-row, 26-language mix published with its byte-exact
recipe and per-upstream license table at
[vpermilp/spellman](https://huggingface.co/datasets/vpermilp/spellman)
(dataset repo type). Sources include FineWeb-2 line-windows, Tatoeba,
per-language community and parallel corpora, a wild social-media lane,
a verified short-utterance lane, and the six open crawl datasets
(`vpermilp/lid-*`: news/library sites + Telegram/VK for
sah/tyv/kpv/mhr/oss/udm) built for this project. Since v16 Russian's
short texts come from open chat and forum dumps (Telegram dialogues, the
Dvach chat, forum messages) instead of a tweet corpus whose `ru` tag
turned out to be mostly Ukrainian. Every source passes LID hygiene; twin
languages are arbitrated by spelling and by vocabulary (rows whose
letters or words belong to a twin language of their label are dropped).
No upstream is NC-licensed (2026-08 license audit); the per-source
license table, including sources whose license is the uploader's own
label, is on the dataset card.

## Version history

- **v16** (2026-10-04): label-noise cleanup — orthographic and lexical
  twin gates, a 41k-row tweet lane that was mostly Ukrainian under a
  Russian label retired, short Russian from chat/forum sources instead, referees held out of training by
  construction, θ 0.67 by error-detection F1. Same architecture and
  format as v14.
- **2026-10-01**: one storage format — the int8-col table at the root;
  the f16 / int8-row / int8-col / fp8 variant directories removed. Same
  v14 weights.
- **v14** (2026-08-31): architecture-review retrain — 2^18 buckets,
  full-length training truncation, θ by error-detection F1; int8-row
  root artifact.
- **v13c** (2026-08-30): the six crawl datasets + per-language cap
  32k → 120k (+3.4pp wild Russian, +4.0pp short vs v12).
- **v12** (2026-08-24): commercial-clean rebuild — every NC-licensed
  upstream replaced or row-filtered.

The full measured experiment log (including what failed):
[docs/experiments.md](https://github.com/vpermilp/spellman/blob/main/docs/experiments.md).