Instructions to use samdei/samdei-type-mini-pretrained with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use samdei/samdei-type-mini-pretrained with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="samdei/samdei-type-mini-pretrained")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("samdei/samdei-type-mini-pretrained") model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-mini-pretrained", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download README.md from samdei/samdei-type-mini-pretrained: direct link, hf CLI and curl.
- Browser
- Download file 6.43 kB
-
https://huggingface.co/samdei/samdei-type-mini-pretrained/resolve/main/README.md
- Command line
-
hf download hf://samdei/samdei-type-mini-pretrained/README.md
-
curl -L -o README.md https://huggingface.co/samdei/samdei-type-mini-pretrained/resolve/main/README.md
license: apache-2.0
language:
- km
library_name: transformers
pipeline_tag: translation
tags:
- transliteration
- romanization
- romanized-khmer
- khmer
- character-level
- pytorch
- safetensors
- pretrained
datasets:
- samdei/samdei-type-pretraining-data
metrics:
- cer
Samdei Type Mini (pretrained)
This is the pretrained (stage-1) checkpoint of Samdei Type Mini, a 7.1M-parameter character-level encoder-decoder that converts romanized Khmer into Khmer script. Pretraining taught it Khmer words and spelling from millions of real Khmer sentences spelled by a rule-based typing generator. It has not yet learned how people actually type in chat.
Use it to fine-tune your own converter: for another typing habit, a domain, or data you collected. It is the size to fine-tune for on-device use, such as a phone keyboard. To convert chat as it is, use Samdei Type Mini. The Base-size checkpoint is samdei-type-base-pretrained.
Release article: Samdei Type: open models that turn romanized Khmer into Khmer script.
Quick start
With transformers alone (pip install transformers torch):
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("samdei/samdei-type-mini-pretrained")
model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-mini-pretrained")
batch = tok(["knhom jong tov phsar"], return_tensors="pt")
print(tok.batch_decode(model.generate(**batch), skip_special_tokens=True)) # ['αααα»αα
αααα
ααααΆα']
Inputs should be short pieces. The Samdei Type Mini card shows how to cut a long message into pieces of at most eight words.
Fine-tune
The folder is a standard transformers model, so any seq2seq training loop works. Two details matter:
- Labels. Output symbols are stored in the tokenizer as
"\ue000"+ character. dropout=0.0. Blenderbot'sdropoutalso drops the embeddings, which this model was never trained with. With it on, held-out loss rises during fine-tuning; with it off, the loss falls. Attention and FFN dropout (0.1) stay on.
import random
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
repo = "samdei/samdei-type-mini-pretrained"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo, dropout=0.0)
vocab = tok.get_vocab()
def labels(khmer: list[str]) -> torch.Tensor:
ids = [[vocab.get("\ue000" + c, vocab["<unk>"]) for c in y] + [tok.eos_token_id] for y in khmer]
n = max(map(len, ids))
return torch.tensor([x + [-100] * (n - len(x)) for x in ids]) # -100: padding, ignored by the loss
pairs = [("knhom jong tov phsar", "αααα»αα
αααα
ααααΆα")] # your (romanized, Khmer) pairs, short pieces
opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.01)
model.train()
for step in range(1000):
batch = random.sample(pairs, min(64, len(pairs)))
x = tok([s for s, _ in batch], padding=True, truncation=True, return_tensors="pt")
loss = model(**x, labels=labels([y for _, y in batch])).loss
loss.backward()
opt.step()
opt.zero_grad()
model.save_pretrained("my-converter")
tok.save_pretrained("my-converter")
The released model was fine-tuned from this checkpoint with AdamW at 2e-4, 300 warmup steps and cosine decay, label smoothing 0.1 and about 12k tokens per batch, for 6k steps. Mixing in some pretraining-style pairs (samdei-type-pretraining-data) keeps the model's general Khmer. The saved folder loads with transformers and with samdei-type. To rerank, add a rerank.json and its LM files, as in the released model.
Results
Samdei Type Bench, beam 5, inputs in pieces of at most eight words:
| Checkpoint | CER | Exact |
|---|---|---|
| this checkpoint | 0.084 | 14% |
| Samdei Type Mini (fine-tuned from it) | 0.072 | 17% |
| Samdei Type Mini, with LM reranking | 0.058 | 24% |
Fine-tuning brings the largest single gain.
Model and training
- Architecture. Pre-LN Transformer encoder-decoder: 6 encoder and 2 decoder layers, d_model 256, 4 heads, FFN 1024; 215 input and 228 output symbols. It loads as
BlenderbotForConditionalGenerationbecause the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 7.43M parameters. - Data. 6.5M pairs: the second version of the typing generator applied to 1.41M web sentences and 2.4k LLM-written chat sentences, plus Gemma 4 romanizations of 2,000 real sentences (Γ3). The files are in
samdei-type-pretraining-data, configsgenerator_v2andllm_romanized, with the exact mix inrecipes.json. The Khmer side is real text from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT, plus the chat sentences. - Training. 40k steps (best at 35k), peak lr 1e-3, 2.2 hours, on one laptop GPU (Apple M5 Pro). The checkpoint was picked on Dev-Sentences and Akkhara-Preview, a public 110-row preview that was never trained on. No evaluation data was trained on.
Limitations
- Its romanized inputs follow the generator's typing personas, not real chat. It handles careful spelling far better than heavy shortcuts.
- It was trained on short inputs only (at most 132 characters), so decode longer text in pieces.
- The Khmer side is mostly web text, with its topics and biases.
License
Apache-2.0 (LICENSE; keep NOTICE when redistributing). The training data is CC BY-SA 4.0, with its sources in the dataset card.
Citation
@misc{samdei-type-2026,
title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
author = {Mengsay Loem},
year = {2026},
note = {Technical report},
url = {https://github.com/Samdei-Labs/samdei-type/blob/main/paper/samdei-type.pdf}
}