samdei-type-base / README.md
mengsay's picture
Link the release article
06a3e56 verified
|
Raw History Blame Contribute Delete
7.73 kB
---
license: apache-2.0
language:
- km
library_name: transformers
pipeline_tag: translation
tags:
- transliteration
- romanization
- romanized-khmer
- khmer
- character-level
- pytorch
- onnx
- int8
- safetensors
base_model: samdei/samdei-type-base-pretrained
base_model_relation: finetune
datasets:
- samdei/samdei-type-pretraining-data
- samdei/samdei-type-finetuning-data
metrics:
- cer
- exact_match
model-index:
- name: Samdei Type Base
results:
- task:
type: translation
name: Romanized Khmer to Khmer script
dataset:
type: samdei/samdei-type-bench
name: Samdei Type Bench
split: test
metrics:
- type: cer
value: 0.052
name: CER (beam 5, char LM + word LM + OOV + channel rerank, 8-word pieces)
- type: exact_match
value: 0.303
name: Exact match
---
# Samdei Type Base
Samdei Type Base converts romanized Khmer, as people type it in chat, into Khmer script. It is an 18.2M-parameter character-level encoder-decoder, the most accurate open Samdei Type model, for desktop and server use. For a phone, use [Samdei Type Mini](https://huggingface.co/samdei/samdei-type-mini).
| Input | Output |
|---|---|
| `knhom jong tov phsar` | αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš |
| `ot mean wifi te` | αž’αžαŸ‹αž˜αžΆαž“ wifi αž‘αŸ |
Release article: [Samdei Type: open models that turn romanized Khmer into Khmer script](https://huggingface.co/blog/mengsay/samdei-type).
## Quick start
With transformers alone (`pip install transformers torch`):
```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
repo = "samdei/samdei-type-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo)
def pieces(text: str, max_words: int = 8) -> list[str]:
"""At most max_words words per piece, with a break after punctuation, as in the paper."""
out, cur = [], []
for w in text.split():
cur.append(w)
if len(cur) == max_words or w[-1] in ",.;:?!αŸ”":
out.append(" ".join(cur))
cur = []
if cur:
out.append(" ".join(cur))
return out or [text]
def convert(text: str) -> str:
batch = tok(pieces(text), padding=True, truncation=True, return_tensors="pt")
out = model.generate(**batch) # beam search, k = 5 (generation_config.json)
return " ".join(tok.batch_decode(out, skip_special_tokens=True))
print(convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
```
This is beam search alone (CER 0.062 on Samdei Type Bench). For the best quality, rerank with the two LMs and the channel model of this repo (0.052), using the [`samdei-type`](https://github.com/Samdei-Labs/samdei-type) package:
```bash
pip install samdei-type
```
```python
from samdei_type.pipeline import Converter
conv = Converter("samdei/samdei-type-base") # PyTorch weights + full rerank, as in the paper
print(conv.convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
```
Or on the command line: `samdei-type -m samdei/samdei-type-base "knhom jong tov phsar"`.
## Files
```
config.json, generation_config.json, model.safetensors (71.1 MiB), tokenizer*.json fp32 model (transformers)
onnx/encoder_kv.int8.onnx (11.3 MiB), onnx/decoder_step.int8.onnx (6.3 MiB), onnx/vocab.json int8 ONNX
channel/ (28.4 MiB) reverse model (Khmer -> romanized) for reranking, same format
lm/char5_nowiki.npz (8.9 MiB) character 5-gram LM
lm/wordfull_nowiki.json.gz (12.2 MiB) word bigram LM
rerank.json decoding and rerank settings
```
Reranking scores each beam hypothesis y for input x as `log P(y|x) + 0.3 log P_char(y) + 0.3 log P_word(y) βˆ’ 1.0 #OOV(y) + 0.3 log P_ch(x|y)`. The channel term checks that y explains all of the input, so the LMs cannot swap a rare word for a frequent one. Nothing in this repo is a pickle.
## Results
[Samdei Type Bench](https://huggingface.co/datasets/samdei/samdei-type-bench): 2,155 everyday messages and words with human-checked Khmer. The metrics are CER after normalization and exact match.
| System | CER | Exact |
|---|---|---|
| **Samdei Type Base**, reranked | **0.052** | **30%** |
| Samdei Type Base, beam only (the transformers example above) | 0.062 | 24% |
| [Samdei Type Mini](https://huggingface.co/samdei/samdei-type-mini), reranked | 0.058 | 24% |
| Samdei Type Prime (not released) | 0.027 | 61% |
| Akkhara-mini, 6-word pieces (0.159 on whole inputs) | 0.066 | 23% |
| AkaraAlpha | 0.281 | 2% |
- Prime is Base trained further on scenario-designed, LLM-drafted, human-audited data. Most bench items share that scenario design, so its lead is an upper bound.
- Without punctuation, Base's CER is 0.046. Decode long messages in pieces, as above. On whole inputs, beam-only CER is 0.133.
- On one Apple M5 Pro CPU thread with int8 ONNX, beam 5 takes 9.3 ms per input (p50).
## Model and training
- **Architecture.** Pre-LN Transformer encoder-decoder: 6 encoder and 3 decoder layers, d_model 384, 6 heads, FFN 1536. It has 216 input and 228 output symbols. The channel model is Mini-sized (7.1M) and reads in the reverse direction.
- **transformers format.** It loads as `BlenderbotForConditionalGeneration` because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 18.63M parameters.
- **Training.** Three stages on one laptop GPU (Apple M5 Pro), about 6 hours in all:
1. pretraining on 12.8M pairs: real Khmer sentences spelled by a rule-based typing generator ([`samdei-type-base-pretrained`](https://huggingface.co/samdei/samdei-type-base-pretrained));
2. fine-tuning on 1.16M chat-like pairs;
3. adaptation to casual register (1.59M pairs, with replay).
The rerank LMs are built from HPLT v2, FineWeb-2, ALT and our LLM-written chat sentences, without Wikipedia.
- **Data.** The exact training files are in [`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data) and [`samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data), with `recipes.json`. No evaluation data and no other romanization system's data was trained on. Akkhara-Preview, a public 110-row preview, was used only to pick stage 1–2 checkpoints; stage 3 used a closed development set.
## Limitations
- Every romanized input seen in training is synthetic (rules, mined spelling tables, an LLM). Real typing appears only in the evaluation.
- Homophones that need more context than a word bigram: ដើរ / αžŠαŸ‚αžš (`der`), αžŠαžΉαž€ / αžŠαŸ‚αž€ / αžŠαŸαž€ (`dek`).
- Loanwords typed in English (`check` for αž†αŸ‚αž€) stay English. A Latin comma is copied, and a missing final αŸ” is never added.
- Rare dictionary words without context are hard, many of them Pali-Sanskrit.
- The Khmer side is mostly web text, with its topics and biases.
- Not for Khmer β†’ Latin, official romanization of names, or output nobody checks.
## License
Apache-2.0 (`LICENSE`; keep `NOTICE` when redistributing).
- **LMs:** built without Wikipedia, from HPLT v2, FineWeb-2 (ODC-By), ALT (CC BY) and our LLM-written chat sentences.
- **Training data:** CC BY-SA 4.0, with its sources in the dataset cards.
- **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
## Citation
```bibtex
@misc{samdei-type-2026,
title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
author = {Mengsay Loem},
year = {2026},
note = {Technical report},
url = {https://github.com/Samdei-Labs/samdei-type/blob/main/paper/samdei-type.pdf}
}
```