Samdei Type Mini (pretrained)

This is the pretrained (stage-1) checkpoint of Samdei Type Mini, a 7.1M-parameter character-level encoder-decoder that converts romanized Khmer into Khmer script. Pretraining taught it Khmer words and spelling from millions of real Khmer sentences spelled by a rule-based typing generator. It has not yet learned how people actually type in chat.

Use it to fine-tune your own converter: for another typing habit, a domain, or data you collected. It is the size to fine-tune for on-device use, such as a phone keyboard. To convert chat as it is, use Samdei Type Mini. The Base-size checkpoint is samdei-type-base-pretrained.

Release article: Samdei Type: open models that turn romanized Khmer into Khmer script.

Quick start

With transformers alone (pip install transformers torch):

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("samdei/samdei-type-mini-pretrained")
model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-mini-pretrained")
batch = tok(["knhom jong tov phsar"], return_tensors="pt")
print(tok.batch_decode(model.generate(**batch), skip_special_tokens=True))  # ['αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš']

Inputs should be short pieces. The Samdei Type Mini card shows how to cut a long message into pieces of at most eight words.

Fine-tune

The folder is a standard transformers model, so any seq2seq training loop works. Two details matter:

  • Labels. Output symbols are stored in the tokenizer as "\ue000" + character.
  • dropout=0.0. Blenderbot's dropout also drops the embeddings, which this model was never trained with. With it on, held-out loss rises during fine-tuning; with it off, the loss falls. Attention and FFN dropout (0.1) stay on.
import random
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo = "samdei/samdei-type-mini-pretrained"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo, dropout=0.0)
vocab = tok.get_vocab()

def labels(khmer: list[str]) -> torch.Tensor:
    ids = [[vocab.get("\ue000" + c, vocab["<unk>"]) for c in y] + [tok.eos_token_id] for y in khmer]
    n = max(map(len, ids))
    return torch.tensor([x + [-100] * (n - len(x)) for x in ids])  # -100: padding, ignored by the loss

pairs = [("knhom jong tov phsar", "αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš")]  # your (romanized, Khmer) pairs, short pieces
opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.01)
model.train()
for step in range(1000):
    batch = random.sample(pairs, min(64, len(pairs)))
    x = tok([s for s, _ in batch], padding=True, truncation=True, return_tensors="pt")
    loss = model(**x, labels=labels([y for _, y in batch])).loss
    loss.backward()
    opt.step()
    opt.zero_grad()

model.save_pretrained("my-converter")
tok.save_pretrained("my-converter")

The released model was fine-tuned from this checkpoint with AdamW at 2e-4, 300 warmup steps and cosine decay, label smoothing 0.1 and about 12k tokens per batch, for 6k steps. Mixing in some pretraining-style pairs (samdei-type-pretraining-data) keeps the model's general Khmer. The saved folder loads with transformers and with samdei-type. To rerank, add a rerank.json and its LM files, as in the released model.

Results

Samdei Type Bench, beam 5, inputs in pieces of at most eight words:

Checkpoint CER Exact
this checkpoint 0.084 14%
Samdei Type Mini (fine-tuned from it) 0.072 17%
Samdei Type Mini, with LM reranking 0.058 24%

Fine-tuning brings the largest single gain.

Model and training

  • Architecture. Pre-LN Transformer encoder-decoder: 6 encoder and 2 decoder layers, d_model 256, 4 heads, FFN 1024; 215 input and 228 output symbols. It loads as BlenderbotForConditionalGeneration because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 7.43M parameters.
  • Data. 6.5M pairs: the second version of the typing generator applied to 1.41M web sentences and 2.4k LLM-written chat sentences, plus Gemma 4 romanizations of 2,000 real sentences (Γ—3). The files are in samdei-type-pretraining-data, configs generator_v2 and llm_romanized, with the exact mix in recipes.json. The Khmer side is real text from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT, plus the chat sentences.
  • Training. 40k steps (best at 35k), peak lr 1e-3, 2.2 hours, on one laptop GPU (Apple M5 Pro). The checkpoint was picked on Dev-Sentences and Akkhara-Preview, a public 110-row preview that was never trained on. No evaluation data was trained on.

Limitations

  • Its romanized inputs follow the generator's typing personas, not real chat. It handles careful spelling far better than heavy shortcuts.
  • It was trained on short inputs only (at most 132 characters), so decode longer text in pieces.
  • The Khmer side is mostly web text, with its topics and biases.

License

Apache-2.0 (LICENSE; keep NOTICE when redistributing). The training data is CC BY-SA 4.0, with its sources in the dataset card.

Citation

@misc{samdei-type-2026,
  title  = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
  author = {Mengsay Loem},
  year   = {2026},
  note   = {Technical report},
  url    = {https://github.com/Samdei-Labs/samdei-type/blob/main/paper/samdei-type.pdf}
}
Downloads last month
42
Safetensors
Model size
7.43M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for samdei/samdei-type-mini-pretrained

Finetunes
1 model

Dataset used to train samdei/samdei-type-mini-pretrained

Collection including samdei/samdei-type-mini-pretrained