--- license: apache-2.0 language: - km library_name: transformers pipeline_tag: translation tags: - transliteration - romanization - romanized-khmer - khmer - character-level - pytorch - safetensors - pretrained datasets: - samdei/samdei-type-pretraining-data metrics: - cer --- # Samdei Type Base (pretrained) This is the pretrained (stage-1) checkpoint of [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base), a 18.2M-parameter character-level encoder-decoder that converts romanized Khmer into Khmer script. Pretraining taught it Khmer words and spelling from millions of real Khmer sentences spelled by a rule-based typing generator. It has not yet learned how people actually type in chat. **Use it to fine-tune your own converter:** for another typing habit, a domain, or data you collected. To convert chat as it is, use [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base). The Mini-size checkpoint is [`samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained). Release article: [Samdei Type: open models that turn romanized Khmer into Khmer script](https://huggingface.co/blog/mengsay/samdei-type). ## Quick start With transformers alone (`pip install transformers torch`): ```python from transformers import AutoModelForSeq2SeqLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained") model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained") batch = tok(["knhom jong tov phsar"], return_tensors="pt") print(tok.batch_decode(model.generate(**batch), skip_special_tokens=True)) # ['ខ្ញុំចង់ទៅផ្សារ'] ``` Inputs should be short pieces. The [Samdei Type Base card](https://huggingface.co/samdei/samdei-type-base) shows how to cut a long message into pieces of at most eight words. ## Fine-tune The folder is a standard transformers model, so any seq2seq training loop works. Two details matter: - **Labels.** Output symbols are stored in the tokenizer as `"\ue000"` + character. - **`dropout=0.0`.** Blenderbot's `dropout` also drops the embeddings, which this model was never trained with. With it on, held-out loss rises during fine-tuning; with it off, the loss falls. Attention and FFN dropout (0.1) stay on. ```python import random import torch from transformers import AutoModelForSeq2SeqLM, AutoTokenizer repo = "samdei/samdei-type-base-pretrained" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForSeq2SeqLM.from_pretrained(repo, dropout=0.0) vocab = tok.get_vocab() def labels(khmer: list[str]) -> torch.Tensor: ids = [[vocab.get("\ue000" + c, vocab[""]) for c in y] + [tok.eos_token_id] for y in khmer] n = max(map(len, ids)) return torch.tensor([x + [-100] * (n - len(x)) for x in ids]) # -100: padding, ignored by the loss pairs = [("knhom jong tov phsar", "ខ្ញុំចង់ទៅផ្សារ")] # your (romanized, Khmer) pairs, short pieces opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.01) model.train() for step in range(1000): batch = random.sample(pairs, min(64, len(pairs))) x = tok([s for s, _ in batch], padding=True, truncation=True, return_tensors="pt") loss = model(**x, labels=labels([y for _, y in batch])).loss loss.backward() opt.step() opt.zero_grad() model.save_pretrained("my-converter") tok.save_pretrained("my-converter") ``` The released model was fine-tuned from this checkpoint with AdamW at 2e-4, 300 warmup steps and cosine decay, label smoothing 0.1 and about 12k tokens per batch, for 6k steps. Mixing in some pretraining-style pairs ([`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data)) keeps the model's general Khmer. The saved folder loads with transformers and with [`samdei-type`](https://github.com/Samdei-Labs/samdei-type). To rerank, add a `rerank.json` and its LM files, as in the released model. ## Results [Samdei Type Bench](https://huggingface.co/datasets/samdei/samdei-type-bench), beam 5, inputs in pieces of at most eight words: | Checkpoint | CER | Exact | |---|---|---| | this checkpoint | 0.097 | 15% | | Samdei Type Base (fine-tuned from it) | 0.062 | 24% | | Samdei Type Base, with LM reranking | 0.052 | 30% | Fine-tuning brings the largest single gain. Part of the gain from fine-tuning is convention: this checkpoint keeps many place names in Latin script, as its pretraining data does, while the bench writes them in Khmer script. ## Model and training - **Architecture.** Pre-LN Transformer encoder-decoder: 6 encoder and 3 decoder layers, d_model 384, 6 heads, FFN 1536; 216 input and 228 output symbols. It loads as `BlenderbotForConditionalGeneration` because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 18.63M parameters. - **Data.** 12.8M pairs: the first version of the typing generator applied twice (two seeds) to 1.41M web sentences, plus 2.4k LLM-written chat sentences spelled the same way (×40). The files are in [`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data), configs `generator_v1` and `llm_chat_sentences`, with the exact mix in `recipes.json`. The Khmer side is real text from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT, plus the chat sentences. - **Training.** 60k steps (best at 60k), peak lr 7e-4, 4.9 hours, on one laptop GPU (Apple M5 Pro). The checkpoint was picked on Dev-Sentences and Akkhara-Preview, a public 110-row preview that was never trained on. No evaluation data was trained on. ## Limitations - Its romanized inputs follow the generator's typing personas, not real chat. It handles careful spelling far better than heavy shortcuts. - It was trained on short inputs only (at most 132 characters), so decode longer text in pieces. - The Khmer side is mostly web text, with its topics and biases. ## License Apache-2.0 (`LICENSE`; keep `NOTICE` when redistributing). The training data is CC BY-SA 4.0, with its sources in the [dataset card](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data). ## Citation ```bibtex @misc{samdei-type-2026, title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration}, author = {Mengsay Loem}, year = {2026}, note = {Technical report}, url = {https://github.com/Samdei-Labs/samdei-type/blob/main/paper/samdei-type.pdf} } ```