| --- |
| language: |
| - ka |
| license: apache-2.0 |
| base_model: google/byt5-small |
| pipeline_tag: translation |
| tags: |
| - transliteration |
| - georgian |
| - byt5 |
| --- |
| |
| # saba-ab/ka_normalize |
| |
| Converts Latin-typed Georgian ("translit") into Georgian script (Mkhedruli). |
| |
| | input | output | |
| |---|---| |
| | `gamarjoba` | `გამარჯობა` | |
| | `ra xdeba` | `რა ხდება` | |
| | `modi 4emtan` | `მოდი ჩემთან` | |
| |
| Latin→Georgian is ambiguous (`t`→თ/ტ, `k`→კ/ქ, `ch`→ჩ/ჭ, `w`→შ/წ, `ts`→ც/წ, `q`→ქ/ყ, `z`→ზ/ძ), |
| so the right letter depends on the word. This model is `google/byt5-small` fine-tuned on synthetic pairs. |
| |
| ## Intended use |
| |
| Normalizing short, informal Latin-typed Georgian (chat messages, comments, support requests) |
| before search, classification, or display. Not intended for English or mixed-language text beyond |
| passing through short Latin tokens. |
| |
| ## Training data |
| |
| - 500k Georgian sentences from `wikimedia/wikipedia` (20231101.ka) and `HuggingFaceFW/fineweb-2` (kat_Geor), |
| filtered to modern Mkhedruli and deduplicated. |
| - Latin sources generated by a stochastic transliterator that samples one consistent "writer profile" |
| per example (e.g. შ as `sh`/`w`/`S`, ჩ as `ch`/`4`), with ~50% short spans (1–8 words), |
| ~50% punctuation stripped, ~30% lowercased, ~10% typo noise. |
| - Trained on 200,000 pairs; base model revision `68377bdc18a2ffec8a0533fef03b1c513a4dd49d`. |
|
|
| ## Evaluation |
|
|
| Rule baseline = greedy longest-match reverse transliteration picking the most frequent letter per token. |
| word acc = 1 − WER. |
|
|
| | set | system | CER ↓ | word acc ↑ | exact ↑ | n | |
| |---|---|---|---|---|---| |
| | synthetic valid | model | 0.67% | 96.68% | 79.05% | 9992 | |
| | synthetic valid | baseline | 4.45% | 73.82% | 21.37% | 9992 | |
| | real test | model | 5.53% | 84.73% | 58.52% | 135 | |
| | real test | baseline | 8.70% | 77.85% | 45.19% | 135 | |
|
|
| Real-test sources are genuinely human-typed chat messages (two group chats). Their Mkhedruli references were drafted by an AI assistant from the raw text (not by this model) and have not been fully human-reviewed; the set is small, so treat real-test numbers as indicative. |
|
|
| ## Limitations |
|
|
| - Training text is formal (Wikipedia/news register); slang, abbreviations and code-switching are under-represented. |
| - Synthetic Latin spellings follow fixed conventions; unusual personal spellings may fail. |
| - Long inputs are split into ~240-byte chunks at word boundaries and whitespace is normalized. |
| - Outputs can contain plausible-but-wrong letters on rare words/names; don't use for legal/medical text unreviewed. |
|
|
| ## Usage |
|
|
| Lowercase the first letter of each sentence before generating. Phones auto-capitalize |
| message starts ("Araa", "Kii"), but the training data never had that, so without this step |
| those words tend to come back in Latin. The reported metrics include this step. Only |
| sentence-initial letters are changed, so `T`=თ / `W`=წ / `S`=შ inside words still work. |
|
|
| ```python |
| import re |
| from transformers import AutoTokenizer, T5ForConditionalGeneration |
| |
| tok = AutoTokenizer.from_pretrained("saba-ab/ka_normalize") |
| model = T5ForConditionalGeneration.from_pretrained("saba-ab/ka_normalize") |
| |
| def fold_sentence_starts(text: str) -> str: |
| return re.sub(r"(^|[.!?…]\s+)([A-Z])", lambda m: m.group(1) + m.group(2).lower(), text) |
| |
| texts = [fold_sentence_starts(t) for t in ["Ra xdeba?", "modi 4emtan"]] |
| ids = model.generate(**tok(texts, return_tensors="pt", padding=True), max_length=768) |
| print(tok.batch_decode(ids, skip_special_tokens=True)) |
| ``` |
|
|
| Inputs longer than ~240 UTF-8 bytes should be split at word boundaries first (training used |
| ≤256-byte sources). |
|
|
