---
language:
- ka
license: apache-2.0
base_model: google/byt5-small
pipeline_tag: translation
tags:
- transliteration
- georgian
- byt5
---

# saba-ab/ka_normalize

Converts Latin-typed Georgian ("translit") into Georgian script (Mkhedruli).

| input | output |
|---|---|
| `gamarjoba` | `გამარჯობა` |
| `ra xdeba` | `რა ხდება` |
| `modi 4emtan` | `მოდი ჩემთან` |

Latin→Georgian is ambiguous (`t`→თ/ტ, `k`→კ/ქ, `ch`→ჩ/ჭ, `w`→შ/წ, `ts`→ც/წ, `q`→ქ/ყ, `z`→ზ/ძ),
so the right letter depends on the word. This model is `google/byt5-small` fine-tuned on synthetic pairs.

## Intended use

Normalizing short, informal Latin-typed Georgian (chat messages, comments, support requests)
before search, classification, or display. Not intended for English or mixed-language text beyond
passing through short Latin tokens.

## Training data

- 500k Georgian sentences from `wikimedia/wikipedia` (20231101.ka) and `HuggingFaceFW/fineweb-2` (kat_Geor),
  filtered to modern Mkhedruli and deduplicated.
- Latin sources generated by a stochastic transliterator that samples one consistent "writer profile"
  per example (e.g. შ as `sh`/`w`/`S`, ჩ as `ch`/`4`), with ~50% short spans (1–8 words),
  ~50% punctuation stripped, ~30% lowercased, ~10% typo noise.
- Trained on 200,000 pairs; base model revision `68377bdc18a2ffec8a0533fef03b1c513a4dd49d`.

## Evaluation

Rule baseline = greedy longest-match reverse transliteration picking the most frequent letter per token.
word acc = 1 − WER.

| set | system | CER ↓ | word acc ↑ | exact ↑ | n |
|---|---|---|---|---|---|
| synthetic valid | model | 0.67% | 96.68% | 79.05% | 9992 |
| synthetic valid | baseline | 4.45% | 73.82% | 21.37% | 9992 |
| real test | model | 5.53% | 84.73% | 58.52% | 135 |
| real test | baseline | 8.70% | 77.85% | 45.19% | 135 |

Real-test sources are genuinely human-typed chat messages (two group chats). Their Mkhedruli references were drafted by an AI assistant from the raw text (not by this model) and have not been fully human-reviewed; the set is small, so treat real-test numbers as indicative.

## Limitations

- Training text is formal (Wikipedia/news register); slang, abbreviations and code-switching are under-represented.
- Synthetic Latin spellings follow fixed conventions; unusual personal spellings may fail.
- Long inputs are split into ~240-byte chunks at word boundaries and whitespace is normalized.
- Outputs can contain plausible-but-wrong letters on rare words/names; don't use for legal/medical text unreviewed.

## Usage

Lowercase the first letter of each sentence before generating. Phones auto-capitalize
message starts ("Araa", "Kii"), but the training data never had that, so without this step
those words tend to come back in Latin. The reported metrics include this step. Only
sentence-initial letters are changed, so `T`=თ / `W`=წ / `S`=შ inside words still work.

```python
import re
from transformers import AutoTokenizer, T5ForConditionalGeneration

tok = AutoTokenizer.from_pretrained("saba-ab/ka_normalize")
model = T5ForConditionalGeneration.from_pretrained("saba-ab/ka_normalize")

def fold_sentence_starts(text: str) -> str:
    return re.sub(r"(^|[.!?…]\s+)([A-Z])", lambda m: m.group(1) + m.group(2).lower(), text)

texts = [fold_sentence_starts(t) for t in ["Ra xdeba?", "modi 4emtan"]]
ids = model.generate(**tok(texts, return_tensors="pt", padding=True), max_length=768)
print(tok.batch_decode(ids, skip_special_tokens=True))
```

Inputs longer than ~240 UTF-8 bytes should be split at word boundaries first (training used
≤256-byte sources).