Instructions to use samdei/samdei-type-base-pretrained with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use samdei/samdei-type-base-pretrained with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="samdei/samdei-type-base-pretrained")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained") model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained", device_map="auto") - Notebooks
- Google Colab
- Kaggle
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained")
model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained", device_map="auto")Samdei Type Base (pretrained)
This is the pretrained (stage-1) checkpoint of Samdei Type Base, a 18.2M-parameter character-level encoder-decoder that converts romanized Khmer into Khmer script. Pretraining taught it Khmer words and spelling from millions of real Khmer sentences spelled by a rule-based typing generator. It has not yet learned how people actually type in chat.
Use it to fine-tune your own converter: for another typing habit, a domain, or data you collected. To convert chat as it is, use Samdei Type Base. The Mini-size checkpoint is samdei-type-mini-pretrained.
Release article: Samdei Type: open models that turn romanized Khmer into Khmer script.
Quick start
With transformers alone (pip install transformers torch):
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained")
model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained")
batch = tok(["knhom jong tov phsar"], return_tensors="pt")
print(tok.batch_decode(model.generate(**batch), skip_special_tokens=True)) # ['αααα»αα
αααα
ααααΆα']
Inputs should be short pieces. The Samdei Type Base card shows how to cut a long message into pieces of at most eight words.
Fine-tune
The folder is a standard transformers model, so any seq2seq training loop works. Two details matter:
- Labels. Output symbols are stored in the tokenizer as
"\ue000"+ character. dropout=0.0. Blenderbot'sdropoutalso drops the embeddings, which this model was never trained with. With it on, held-out loss rises during fine-tuning; with it off, the loss falls. Attention and FFN dropout (0.1) stay on.
import random
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
repo = "samdei/samdei-type-base-pretrained"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo, dropout=0.0)
vocab = tok.get_vocab()
def labels(khmer: list[str]) -> torch.Tensor:
ids = [[vocab.get("\ue000" + c, vocab["<unk>"]) for c in y] + [tok.eos_token_id] for y in khmer]
n = max(map(len, ids))
return torch.tensor([x + [-100] * (n - len(x)) for x in ids]) # -100: padding, ignored by the loss
pairs = [("knhom jong tov phsar", "αααα»αα
αααα
ααααΆα")] # your (romanized, Khmer) pairs, short pieces
opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.01)
model.train()
for step in range(1000):
batch = random.sample(pairs, min(64, len(pairs)))
x = tok([s for s, _ in batch], padding=True, truncation=True, return_tensors="pt")
loss = model(**x, labels=labels([y for _, y in batch])).loss
loss.backward()
opt.step()
opt.zero_grad()
model.save_pretrained("my-converter")
tok.save_pretrained("my-converter")
The released model was fine-tuned from this checkpoint with AdamW at 2e-4, 300 warmup steps and cosine decay, label smoothing 0.1 and about 12k tokens per batch, for 6k steps. Mixing in some pretraining-style pairs (samdei-type-pretraining-data) keeps the model's general Khmer. The saved folder loads with transformers and with samdei-type. To rerank, add a rerank.json and its LM files, as in the released model.
Results
Samdei Type Bench, beam 5, inputs in pieces of at most eight words:
| Checkpoint | CER | Exact |
|---|---|---|
| this checkpoint | 0.097 | 15% |
| Samdei Type Base (fine-tuned from it) | 0.062 | 24% |
| Samdei Type Base, with LM reranking | 0.052 | 30% |
Fine-tuning brings the largest single gain. Part of the gain from fine-tuning is convention: this checkpoint keeps many place names in Latin script, as its pretraining data does, while the bench writes them in Khmer script.
Model and training
- Architecture. Pre-LN Transformer encoder-decoder: 6 encoder and 3 decoder layers, d_model 384, 6 heads, FFN 1536; 216 input and 228 output symbols. It loads as
BlenderbotForConditionalGenerationbecause the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 18.63M parameters. - Data. 12.8M pairs: the first version of the typing generator applied twice (two seeds) to 1.41M web sentences, plus 2.4k LLM-written chat sentences spelled the same way (Γ40). The files are in
samdei-type-pretraining-data, configsgenerator_v1andllm_chat_sentences, with the exact mix inrecipes.json. The Khmer side is real text from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT, plus the chat sentences. - Training. 60k steps (best at 60k), peak lr 7e-4, 4.9 hours, on one laptop GPU (Apple M5 Pro). The checkpoint was picked on Dev-Sentences and Akkhara-Preview, a public 110-row preview that was never trained on. No evaluation data was trained on.
Limitations
- Its romanized inputs follow the generator's typing personas, not real chat. It handles careful spelling far better than heavy shortcuts.
- It was trained on short inputs only (at most 132 characters), so decode longer text in pieces.
- The Khmer side is mostly web text, with its topics and biases.
License
Apache-2.0 (LICENSE; keep NOTICE when redistributing). The training data is CC BY-SA 4.0, with its sources in the dataset card.
Citation
@misc{samdei-type-2026,
title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
author = {Mengsay Loem},
year = {2026},
note = {Technical report},
url = {https://github.com/Samdei-Labs/samdei-type/blob/main/paper/samdei-type.pdf}
}
- Downloads last month
- 39
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="samdei/samdei-type-base-pretrained")