Translation
Transformers
Safetensors
PyTorch
Khmer
blenderbot
text2text-generation
transliteration
romanization
romanized-khmer
khmer
character-level
pretrained
Instructions to use samdei/samdei-type-base-pretrained with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use samdei/samdei-type-base-pretrained with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="samdei/samdei-type-base-pretrained")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained") model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Shorter cards with a transformers quick start; translation task tag; author; samdei-type package
Browse files
README.md
CHANGED
|
@@ -3,7 +3,7 @@ license: apache-2.0
|
|
| 3 |
language:
|
| 4 |
- km
|
| 5 |
library_name: transformers
|
| 6 |
-
pipeline_tag:
|
| 7 |
tags:
|
| 8 |
- transliteration
|
| 9 |
- romanization
|
|
@@ -21,156 +21,98 @@ metrics:
|
|
| 21 |
|
| 22 |
# Samdei Type Base (pretrained)
|
| 23 |
|
| 24 |
-
This is the
|
| 25 |
|
| 26 |
-
|
| 27 |
|
| 28 |
-
|
| 29 |
-
- a different romanization habit or scheme;
|
| 30 |
-
- a domain;
|
| 31 |
-
- data you collected.
|
| 32 |
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
Samdei (ααααΈ, "what you say") is our team, and Samdei Type is the converter. Paper: `<paper-url>`. Code: `<code-repo-url>`.
|
| 36 |
-
|
| 37 |
-
## Files
|
| 38 |
-
|
| 39 |
-
```
|
| 40 |
-
config.json configuration and vocabularies (transformers format)
|
| 41 |
-
generation_config.json beam search, k = 5
|
| 42 |
-
model.safetensors 71.1 MiB weights (fp32)
|
| 43 |
-
tokenizer.json character tokenizer
|
| 44 |
-
tokenizer_config.json its settings
|
| 45 |
-
```
|
| 46 |
-
|
| 47 |
-
## How to use
|
| 48 |
-
|
| 49 |
-
Install the `rkhmer` package from the code repository:
|
| 50 |
-
|
| 51 |
-
```bash
|
| 52 |
-
pip install "rkhmer @ git+<code-repo-url>"
|
| 53 |
-
```
|
| 54 |
-
|
| 55 |
-
**Fine-tune** with the training script of the code repository, as the released models were. For example, stage 2 of Base, with the data of [`samdei/samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data) exported to TSV, or with your own `latin<TAB>khmer` file:
|
| 56 |
-
|
| 57 |
-
```bash
|
| 58 |
-
hf download samdei/samdei-type-base-pretrained --local-dir samdei-base-pretrained
|
| 59 |
-
python scripts/train.py --init samdei-base-pretrained --train my_pairs.tsv \
|
| 60 |
-
--dev_sets my_dev.tsv --select my_dev.tsv \
|
| 61 |
-
--steps 6000 --warmup 300 --lr 2e-4 --eval_every 1000 --out runs/my_finetune
|
| 62 |
-
```
|
| 63 |
-
|
| 64 |
-
The model configuration comes from the checkpoint. `best.pt` is picked on the greedy CER of `--select`. The settings above are those of stage 2: AdamW, cosine decay to 5% of the peak, label smoothing 0.1, about 12k tokens per batch, bf16 autocast. Replaying some pretraining-style data alongside new data keeps the model's general Khmer.
|
| 65 |
-
|
| 66 |
-
`train.py` writes PyTorch checkpoints (`best.pt`, a pickle). To share a fine-tuned model safely, convert it to the format of this repository:
|
| 67 |
-
|
| 68 |
-
```python
|
| 69 |
-
from rkhmer.model import load, save_pretrained
|
| 70 |
-
save_pretrained("my_model", *load("runs/my_finetune/best.pt")[:3]) # my_model/: the five files of this repository
|
| 71 |
-
```
|
| 72 |
-
|
| 73 |
-
**Convert** text directly (for comparison):
|
| 74 |
-
|
| 75 |
-
```python
|
| 76 |
-
from rkhmer.evaluation import chunk_source, translate
|
| 77 |
-
from rkhmer.model import load_pretrained
|
| 78 |
-
|
| 79 |
-
model, sv, tv, _ = load_pretrained("samdei/samdei-type-base-pretrained")
|
| 80 |
-
pieces = chunk_source("knhom jong tov phsar", 8)
|
| 81 |
-
best, _ = translate(model, sv, tv, pieces, "cpu", beam=5)
|
| 82 |
-
print(" ".join(best))
|
| 83 |
-
```
|
| 84 |
-
|
| 85 |
-
With transformers instead (see Model):
|
| 86 |
|
| 87 |
```python
|
| 88 |
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
|
|
|
| 89 |
tok = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained")
|
| 90 |
model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained")
|
| 91 |
batch = tok(["knhom jong tov phsar"], return_tensors="pt")
|
| 92 |
-
print(tok.batch_decode(model.generate(**batch), skip_special_tokens=True)) #
|
| 93 |
```
|
| 94 |
|
| 95 |
-
The [Samdei Type Base card](https://huggingface.co/samdei/samdei-type-base) shows how to cut long
|
| 96 |
-
|
| 97 |
-
**No pickles.** The weights are safetensors and the configuration is JSON, so loading runs no code from this repository. `load_pretrained` downloads only `config.json` and `model.safetensors`.
|
| 98 |
|
| 99 |
-
##
|
| 100 |
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
| d_model / heads | 384 / 6 |
|
| 105 |
-
| Encoder / decoder layers | 6 / 3 |
|
| 106 |
-
| FFN width | 1536 |
|
| 107 |
-
| Parameters | 18,202,980 |
|
| 108 |
-
| Vocabulary (incl. special tokens) | 216 input, 228 output |
|
| 109 |
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
| | Stage 1: pretraining |
|
| 115 |
-
|---|---|
|
| 116 |
-
| Data | the first generator version on 1.41M web sentences, two seeds (6.29M pairs each), + 2.4k LLM-written chat sentences spelled the same way, Γ40 (189k pairs) |
|
| 117 |
-
| Total pairs | 12.8M |
|
| 118 |
-
| Steps (best checkpoint) | 60k (60k) |
|
| 119 |
-
| Peak lr, warmup | 7e-4, 3k |
|
| 120 |
-
| Wall clock | 4.9 h |
|
| 121 |
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
|
| 133 |
-
|
| 134 |
|
| 135 |
-
|
| 136 |
|
| 137 |
-
|
| 138 |
-
- **Metric:** CER after normalization, and exact match.
|
| 139 |
|
| 140 |
| Checkpoint | CER | Exact |
|
| 141 |
|---|---|---|
|
| 142 |
-
| this checkpoint
|
| 143 |
-
| Samdei Type Base (
|
|
|
|
|
|
|
|
|
|
| 144 |
|
| 145 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
|
| 147 |
## Limitations
|
| 148 |
|
| 149 |
-
-
|
| 150 |
-
-
|
| 151 |
-
-
|
| 152 |
|
| 153 |
-
## License
|
| 154 |
|
| 155 |
-
|
| 156 |
-
- **Training data** (CC BY-SA 4.0, [`samdei/samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data)). Its sources are:
|
| 157 |
-
- HPLT v2 (CC0 packaging; Common Crawl terms apply to the text);
|
| 158 |
-
- FineWeb-2 (ODC-By 1.0);
|
| 159 |
-
- Khmer Wikipedia (CC BY-SA 4.0);
|
| 160 |
-
- ALT (CC BY 4.0);
|
| 161 |
-
- Google's Khmer pronunciation lexicon (CC BY 4.0);
|
| 162 |
-
- Gemma 4 output (Apache-2.0).
|
| 163 |
-
- **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
|
| 164 |
|
| 165 |
## Citation
|
| 166 |
|
| 167 |
```bibtex
|
| 168 |
@misc{samdei-type-2026,
|
| 169 |
title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
|
| 170 |
-
author = {
|
| 171 |
year = {2026},
|
| 172 |
url = {<paper-url>}
|
| 173 |
}
|
| 174 |
```
|
| 175 |
-
|
| 176 |
-
Contact: `<contact>`
|
|
|
|
| 3 |
language:
|
| 4 |
- km
|
| 5 |
library_name: transformers
|
| 6 |
+
pipeline_tag: translation
|
| 7 |
tags:
|
| 8 |
- transliteration
|
| 9 |
- romanization
|
|
|
|
| 21 |
|
| 22 |
# Samdei Type Base (pretrained)
|
| 23 |
|
| 24 |
+
This is the pretrained (stage-1) checkpoint of [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base), a 18.2M-parameter character-level encoder-decoder that converts romanized Khmer into Khmer script. Pretraining taught it Khmer words and spelling from millions of real Khmer sentences spelled by a rule-based typing generator. It has not yet learned how people actually type in chat.
|
| 25 |
|
| 26 |
+
**Use it to fine-tune your own converter:** for another typing habit, a domain, or data you collected. To convert chat as it is, use [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base). The Mini-size checkpoint is [`samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained).
|
| 27 |
|
| 28 |
+
## Quick start
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
+
With transformers alone (`pip install transformers torch`):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
```python
|
| 33 |
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
| 34 |
+
|
| 35 |
tok = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained")
|
| 36 |
model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained")
|
| 37 |
batch = tok(["knhom jong tov phsar"], return_tensors="pt")
|
| 38 |
+
print(tok.batch_decode(model.generate(**batch), skip_special_tokens=True)) # ['αααα»αα
αααα
ααααΆα']
|
| 39 |
```
|
| 40 |
|
| 41 |
+
Inputs should be short pieces. The [Samdei Type Base card](https://huggingface.co/samdei/samdei-type-base) shows how to cut a long message into pieces of at most eight words.
|
|
|
|
|
|
|
| 42 |
|
| 43 |
+
## Fine-tune
|
| 44 |
|
| 45 |
+
The folder is a standard transformers model, so any seq2seq training loop works. Two details matter:
|
| 46 |
+
- **Labels.** Output symbols are stored in the tokenizer as `"\ue000"` + character.
|
| 47 |
+
- **`dropout=0.0`.** Blenderbot's `dropout` also drops the embeddings, which this model was never trained with. With it on, held-out loss rises during fine-tuning; with it off, the loss falls. Attention and FFN dropout (0.1) stay on.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
+
```python
|
| 50 |
+
import random
|
| 51 |
+
import torch
|
| 52 |
+
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
|
| 54 |
+
repo = "samdei/samdei-type-base-pretrained"
|
| 55 |
+
tok = AutoTokenizer.from_pretrained(repo)
|
| 56 |
+
model = AutoModelForSeq2SeqLM.from_pretrained(repo, dropout=0.0)
|
| 57 |
+
vocab = tok.get_vocab()
|
| 58 |
+
|
| 59 |
+
def labels(khmer: list[str]) -> torch.Tensor:
|
| 60 |
+
ids = [[vocab.get("\ue000" + c, vocab["<unk>"]) for c in y] + [tok.eos_token_id] for y in khmer]
|
| 61 |
+
n = max(map(len, ids))
|
| 62 |
+
return torch.tensor([x + [-100] * (n - len(x)) for x in ids]) # -100: padding, ignored by the loss
|
| 63 |
+
|
| 64 |
+
pairs = [("knhom jong tov phsar", "αααα»αα
αααα
ααααΆα")] # your (romanized, Khmer) pairs, short pieces
|
| 65 |
+
opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.01)
|
| 66 |
+
model.train()
|
| 67 |
+
for step in range(1000):
|
| 68 |
+
batch = random.sample(pairs, min(64, len(pairs)))
|
| 69 |
+
x = tok([s for s, _ in batch], padding=True, truncation=True, return_tensors="pt")
|
| 70 |
+
loss = model(**x, labels=labels([y for _, y in batch])).loss
|
| 71 |
+
loss.backward()
|
| 72 |
+
opt.step()
|
| 73 |
+
opt.zero_grad()
|
| 74 |
+
|
| 75 |
+
model.save_pretrained("my-converter")
|
| 76 |
+
tok.save_pretrained("my-converter")
|
| 77 |
+
```
|
| 78 |
|
| 79 |
+
The released model was fine-tuned from this checkpoint with AdamW at 2e-4, 300 warmup steps and cosine decay, label smoothing 0.1 and about 12k tokens per batch, for 6k steps. Mixing in some pretraining-style pairs ([`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data)) keeps the model's general Khmer. The saved folder loads with transformers and with [`samdei-type`](https://github.com/Samdei-Labs/samdei-type). To rerank, add a `rerank.json` and its LM files, as in the released model.
|
| 80 |
|
| 81 |
+
## Results
|
| 82 |
|
| 83 |
+
[Samdei Type Bench](https://huggingface.co/datasets/samdei/samdei-type-bench), beam 5, inputs in pieces of at most eight words:
|
|
|
|
| 84 |
|
| 85 |
| Checkpoint | CER | Exact |
|
| 86 |
|---|---|---|
|
| 87 |
+
| this checkpoint | 0.097 | 15% |
|
| 88 |
+
| Samdei Type Base (fine-tuned from it) | 0.062 | 24% |
|
| 89 |
+
| Samdei Type Base, with LM reranking | 0.052 | 30% |
|
| 90 |
+
|
| 91 |
+
Fine-tuning brings the largest single gain. Part of the gain from fine-tuning is convention: this checkpoint keeps many place names in Latin script, as its pretraining data does, while the bench writes them in Khmer script.
|
| 92 |
|
| 93 |
+
## Model and training
|
| 94 |
+
|
| 95 |
+
- **Architecture.** Pre-LN Transformer encoder-decoder: 6 encoder and 3 decoder layers, d_model 384, 6 heads, FFN 1536; 216 input and 228 output symbols. It loads as `BlenderbotForConditionalGeneration` because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 18.63M parameters.
|
| 96 |
+
- **Data.** 12.8M pairs: the first version of the typing generator applied twice (two seeds) to 1.41M web sentences, plus 2.4k LLM-written chat sentences spelled the same way (Γ40). The files are in [`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data), configs `generator_v1` and `llm_chat_sentences`, with the exact mix in `recipes.json`. The Khmer side is real text from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT, plus the chat sentences.
|
| 97 |
+
- **Training.** 60k steps (best at 60k), peak lr 7e-4, 4.9 hours, on one laptop GPU (Apple M5 Pro). The checkpoint was picked on Dev-Sentences and Akkhara-Preview, a public 110-row preview that was never trained on. No evaluation data was trained on.
|
| 98 |
|
| 99 |
## Limitations
|
| 100 |
|
| 101 |
+
- Its romanized inputs follow the generator's typing personas, not real chat. It handles careful spelling far better than heavy shortcuts.
|
| 102 |
+
- It was trained on short inputs only (at most 132 characters), so decode longer text in pieces.
|
| 103 |
+
- The Khmer side is mostly web text, with its topics and biases.
|
| 104 |
|
| 105 |
+
## License
|
| 106 |
|
| 107 |
+
Apache-2.0 (`LICENSE`; keep `NOTICE` when redistributing). The training data is CC BY-SA 4.0, with its sources in the [dataset card](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
|
| 109 |
## Citation
|
| 110 |
|
| 111 |
```bibtex
|
| 112 |
@misc{samdei-type-2026,
|
| 113 |
title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
|
| 114 |
+
author = {Mengsay Loem},
|
| 115 |
year = {2026},
|
| 116 |
url = {<paper-url>}
|
| 117 |
}
|
| 118 |
```
|
|
|
|
|
|