Instructions to use vpermilp/spellman with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- fastText
How to use vpermilp/spellman with fastText:
from huggingface_hub import hf_hub_download import fasttext model = fasttext.load_model(hf_hub_download("vpermilp/spellman", "model.bin")) - Notebooks
- Google Colab
- Kaggle
docs: v11c card fixes — tatoeba ladder, 12 diverse lanes (algo=6), literary lane, 32k cap, dataset link (was: stale data_mix4 pointer)
Browse files
README.md
CHANGED
|
@@ -106,7 +106,7 @@ printf 'Қазақша\nHello\n' | spellman detect --lines # kaz /
|
|
| 106 |
|---|---|
|
| 107 |
| held-out mix (352,470 rows, pristine test) | 98.56% |
|
| 108 |
| Tatoeba (37,051, out-of-domain) | 98.78% |
|
| 109 |
-
| Tatoeba granularity ladder (word / pair / triple) |
|
| 110 |
| wild Russian tweets (2,606, label-audited; 61% @mentions, 20% URLs) | 94.47% |
|
| 111 |
| COSMUS gold-labeled wild Russian (2,808, unseen) | 97.15% |
|
| 112 |
| short-utterance referee (orthography-certified, ≤19 chars) | 91.29% |
|
|
@@ -124,13 +124,19 @@ set still leads clean out-of-domain sentences (99.25 vs 98.78).
|
|
| 124 |
|
| 125 |
## Training data
|
| 126 |
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
|---|---|
|
| 107 |
| held-out mix (352,470 rows, pristine test) | 98.56% |
|
| 108 |
| Tatoeba (37,051, out-of-domain) | 98.78% |
|
| 109 |
+
| Tatoeba granularity ladder (word / pair / triple) | 70.6 / 88.2 / 94.5 |
|
| 110 |
| wild Russian tweets (2,606, label-audited; 61% @mentions, 20% URLs) | 94.47% |
|
| 111 |
| COSMUS gold-labeled wild Russian (2,808, unseen) | 97.15% |
|
| 112 |
| short-utterance referee (orthography-certified, ≤19 chars) | 91.29% |
|
|
|
|
| 124 |
|
| 125 |
## Training data
|
| 126 |
|
| 127 |
+
The full training mix (1,535,170 rows, 26 languages) is published at
|
| 128 |
+
[vpermilp/spellman](https://huggingface.co/datasets/vpermilp/spellman)
|
| 129 |
+
(dataset repo type) — the byte-exact recipe lives in its `manifest.json`.
|
| 130 |
+
|
| 131 |
+
Built from FineWeb-2 per-language configs (32k rows/language cap),
|
| 132 |
+
~104k Tatoeba sentences (CC BY 2.0 FR), per-language community/parallel
|
| 133 |
+
corpora, a wild social-media lane (1.85M Ukrainian tweets, Telegram
|
| 134 |
+
Uzbek, Chuvash CC0 web, Kazakh reviews, Leipzig community crawls,
|
| 135 |
+
cardiffnlp multilingual tweets, sentiment140, 2025 German-politics
|
| 136 |
+
tweets), a verified short-utterance lane (3-19 char wild rows under
|
| 137 |
+
2-of-3 judge consensus and orthographic twin rules), a literary Russian
|
| 138 |
+
lane (wikisource classic prose, 6k rows), and 12 lexically-diverse lanes
|
| 139 |
+
selected by banded lemma coverage (algo=6) over wild pools, with real
|
| 140 |
+
normalizers — pymorphy3 for rus/ukr, Stanza for bel/bul/srp/kaz/kir,
|
| 141 |
+
Apertium for mkd, corpus-derived prefix stems for the agglutinative
|
| 142 |
+
set. Every source is LID-hygiene-filtered with twin-language protection.
|