vpermilp commited on
Commit
5cb3fe4
·
verified ·
1 Parent(s): ad0304c

docs: v11c card fixes — tatoeba ladder, 12 diverse lanes (algo=6), literary lane, 32k cap, dataset link (was: stale data_mix4 pointer)

Browse files
Files changed (1) hide show
  1. README.md +17 -11
README.md CHANGED
@@ -106,7 +106,7 @@ printf 'Қазақша\nHello\n' | spellman detect --lines # kaz /
106
  |---|---|
107
  | held-out mix (352,470 rows, pristine test) | 98.56% |
108
  | Tatoeba (37,051, out-of-domain) | 98.78% |
109
- | Tatoeba granularity ladder (word / pair / triple) | 66.7 / 86.2 / 93.3 |
110
  | wild Russian tweets (2,606, label-audited; 61% @mentions, 20% URLs) | 94.47% |
111
  | COSMUS gold-labeled wild Russian (2,808, unseen) | 97.15% |
112
  | short-utterance referee (orthography-certified, ≤19 chars) | 91.29% |
@@ -124,13 +124,19 @@ set still leads clean out-of-domain sentences (99.25 vs 98.78).
124
 
125
  ## Training data
126
 
127
- FineWeb-2 per-language configs, ~104k Tatoeba sentences (CC BY 2.0 FR),
128
- per-language community/parallel corpora, a wild social-media lane (1.85M
129
- Ukrainian tweets, Telegram Uzbek, Chuvash CC0 web, Kazakh reviews,
130
- Leipzig community crawls, cardiffnlp multilingual tweets, sentiment140,
131
- 2025 German-politics tweets), a verified short-utterance lane (3-19 char
132
- wild rows under 2-of-3 judge consensus and orthographic twin rules), and
133
- lexically-diverse rus/ukr sets selected by greedy lemma coverage
134
- (pymorphy3; 93% of the top-15k lemmas in 14k rows). Every source
135
- LID-hygiene-filtered with twin-language protection; the exact recipe is
136
- recorded in the repository (data_mix4/manifest.json).
 
 
 
 
 
 
 
106
  |---|---|
107
  | held-out mix (352,470 rows, pristine test) | 98.56% |
108
  | Tatoeba (37,051, out-of-domain) | 98.78% |
109
+ | Tatoeba granularity ladder (word / pair / triple) | 70.6 / 88.2 / 94.5 |
110
  | wild Russian tweets (2,606, label-audited; 61% @mentions, 20% URLs) | 94.47% |
111
  | COSMUS gold-labeled wild Russian (2,808, unseen) | 97.15% |
112
  | short-utterance referee (orthography-certified, ≤19 chars) | 91.29% |
 
124
 
125
  ## Training data
126
 
127
+ The full training mix (1,535,170 rows, 26 languages) is published at
128
+ [vpermilp/spellman](https://huggingface.co/datasets/vpermilp/spellman)
129
+ (dataset repo type) — the byte-exact recipe lives in its `manifest.json`.
130
+
131
+ Built from FineWeb-2 per-language configs (32k rows/language cap),
132
+ ~104k Tatoeba sentences (CC BY 2.0 FR), per-language community/parallel
133
+ corpora, a wild social-media lane (1.85M Ukrainian tweets, Telegram
134
+ Uzbek, Chuvash CC0 web, Kazakh reviews, Leipzig community crawls,
135
+ cardiffnlp multilingual tweets, sentiment140, 2025 German-politics
136
+ tweets), a verified short-utterance lane (3-19 char wild rows under
137
+ 2-of-3 judge consensus and orthographic twin rules), a literary Russian
138
+ lane (wikisource classic prose, 6k rows), and 12 lexically-diverse lanes
139
+ selected by banded lemma coverage (algo=6) over wild pools, with real
140
+ normalizers — pymorphy3 for rus/ukr, Stanza for bel/bul/srp/kaz/kir,
141
+ Apertium for mkd, corpus-derived prefix stems for the agglutinative
142
+ set. Every source is LID-hygiene-filtered with twin-language protection.