Translation
Transformers
Safetensors
PyTorch
Khmer
blenderbot
text2text-generation
transliteration
romanization
romanized-khmer
khmer
character-level
pretrained
Instructions to use samdei/samdei-type-base-pretrained with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use samdei/samdei-type-base-pretrained with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="samdei/samdei-type-base-pretrained")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained") model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained", device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from samdei/samdei-type-base-pretrained: direct link, hf CLI and curl.
- Browser
- Download file 6.54 kB
-
https://huggingface.co/samdei/samdei-type-base-pretrained/resolve/main/README.md
- Command line
-
hf download hf://samdei/samdei-type-base-pretrained/README.md
-
curl -L -o README.md https://huggingface.co/samdei/samdei-type-base-pretrained/resolve/main/README.md
6.54 kB
| license: apache-2.0 | |
| language: | |
| - km | |
| library_name: transformers | |
| pipeline_tag: translation | |
| tags: | |
| - transliteration | |
| - romanization | |
| - romanized-khmer | |
| - khmer | |
| - character-level | |
| - pytorch | |
| - safetensors | |
| - pretrained | |
| datasets: | |
| - samdei/samdei-type-pretraining-data | |
| metrics: | |
| - cer | |
| # Samdei Type Base (pretrained) | |
| This is the pretrained (stage-1) checkpoint of [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base), a 18.2M-parameter character-level encoder-decoder that converts romanized Khmer into Khmer script. Pretraining taught it Khmer words and spelling from millions of real Khmer sentences spelled by a rule-based typing generator. It has not yet learned how people actually type in chat. | |
| **Use it to fine-tune your own converter:** for another typing habit, a domain, or data you collected. To convert chat as it is, use [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base). The Mini-size checkpoint is [`samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained). | |
| Release article: [Samdei Type: open models that turn romanized Khmer into Khmer script](https://huggingface.co/blog/mengsay/samdei-type). | |
| ## Quick start | |
| With transformers alone (`pip install transformers torch`): | |
| ```python | |
| from transformers import AutoModelForSeq2SeqLM, AutoTokenizer | |
| tok = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained") | |
| model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained") | |
| batch = tok(["knhom jong tov phsar"], return_tensors="pt") | |
| print(tok.batch_decode(model.generate(**batch), skip_special_tokens=True)) # ['αααα»αα αααα ααααΆα'] | |
| ``` | |
| Inputs should be short pieces. The [Samdei Type Base card](https://huggingface.co/samdei/samdei-type-base) shows how to cut a long message into pieces of at most eight words. | |
| ## Fine-tune | |
| The folder is a standard transformers model, so any seq2seq training loop works. Two details matter: | |
| - **Labels.** Output symbols are stored in the tokenizer as `"\ue000"` + character. | |
| - **`dropout=0.0`.** Blenderbot's `dropout` also drops the embeddings, which this model was never trained with. With it on, held-out loss rises during fine-tuning; with it off, the loss falls. Attention and FFN dropout (0.1) stay on. | |
| ```python | |
| import random | |
| import torch | |
| from transformers import AutoModelForSeq2SeqLM, AutoTokenizer | |
| repo = "samdei/samdei-type-base-pretrained" | |
| tok = AutoTokenizer.from_pretrained(repo) | |
| model = AutoModelForSeq2SeqLM.from_pretrained(repo, dropout=0.0) | |
| vocab = tok.get_vocab() | |
| def labels(khmer: list[str]) -> torch.Tensor: | |
| ids = [[vocab.get("\ue000" + c, vocab["<unk>"]) for c in y] + [tok.eos_token_id] for y in khmer] | |
| n = max(map(len, ids)) | |
| return torch.tensor([x + [-100] * (n - len(x)) for x in ids]) # -100: padding, ignored by the loss | |
| pairs = [("knhom jong tov phsar", "αααα»αα αααα ααααΆα")] # your (romanized, Khmer) pairs, short pieces | |
| opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.01) | |
| model.train() | |
| for step in range(1000): | |
| batch = random.sample(pairs, min(64, len(pairs))) | |
| x = tok([s for s, _ in batch], padding=True, truncation=True, return_tensors="pt") | |
| loss = model(**x, labels=labels([y for _, y in batch])).loss | |
| loss.backward() | |
| opt.step() | |
| opt.zero_grad() | |
| model.save_pretrained("my-converter") | |
| tok.save_pretrained("my-converter") | |
| ``` | |
| The released model was fine-tuned from this checkpoint with AdamW at 2e-4, 300 warmup steps and cosine decay, label smoothing 0.1 and about 12k tokens per batch, for 6k steps. Mixing in some pretraining-style pairs ([`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data)) keeps the model's general Khmer. The saved folder loads with transformers and with [`samdei-type`](https://github.com/Samdei-Labs/samdei-type). To rerank, add a `rerank.json` and its LM files, as in the released model. | |
| ## Results | |
| [Samdei Type Bench](https://huggingface.co/datasets/samdei/samdei-type-bench), beam 5, inputs in pieces of at most eight words: | |
| | Checkpoint | CER | Exact | | |
| |---|---|---| | |
| | this checkpoint | 0.097 | 15% | | |
| | Samdei Type Base (fine-tuned from it) | 0.062 | 24% | | |
| | Samdei Type Base, with LM reranking | 0.052 | 30% | | |
| Fine-tuning brings the largest single gain. Part of the gain from fine-tuning is convention: this checkpoint keeps many place names in Latin script, as its pretraining data does, while the bench writes them in Khmer script. | |
| ## Model and training | |
| - **Architecture.** Pre-LN Transformer encoder-decoder: 6 encoder and 3 decoder layers, d_model 384, 6 heads, FFN 1536; 216 input and 228 output symbols. It loads as `BlenderbotForConditionalGeneration` because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 18.63M parameters. | |
| - **Data.** 12.8M pairs: the first version of the typing generator applied twice (two seeds) to 1.41M web sentences, plus 2.4k LLM-written chat sentences spelled the same way (Γ40). The files are in [`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data), configs `generator_v1` and `llm_chat_sentences`, with the exact mix in `recipes.json`. The Khmer side is real text from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT, plus the chat sentences. | |
| - **Training.** 60k steps (best at 60k), peak lr 7e-4, 4.9 hours, on one laptop GPU (Apple M5 Pro). The checkpoint was picked on Dev-Sentences and Akkhara-Preview, a public 110-row preview that was never trained on. No evaluation data was trained on. | |
| ## Limitations | |
| - Its romanized inputs follow the generator's typing personas, not real chat. It handles careful spelling far better than heavy shortcuts. | |
| - It was trained on short inputs only (at most 132 characters), so decode longer text in pieces. | |
| - The Khmer side is mostly web text, with its topics and biases. | |
| ## License | |
| Apache-2.0 (`LICENSE`; keep `NOTICE` when redistributing). The training data is CC BY-SA 4.0, with its sources in the [dataset card](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data). | |
| ## Citation | |
| ```bibtex | |
| @misc{samdei-type-2026, | |
| title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration}, | |
| author = {Mengsay Loem}, | |
| year = {2026}, | |
| note = {Technical report}, | |
| url = {https://github.com/Samdei-Labs/samdei-type/blob/main/paper/samdei-type.pdf} | |
| } | |
| ``` | |