Translation
Transformers
ONNX
Safetensors
PyTorch
Khmer
blenderbot
text2text-generation
transliteration
romanization
romanized-khmer
khmer
character-level
int8
Eval Results (legacy)
Instructions to use samdei/samdei-type-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use samdei/samdei-type-base with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="samdei/samdei-type-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("samdei/samdei-type-base") model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Shorter cards with a transformers quick start; translation task tag; author; samdei-type package
Browse files- README.md +58 -186
- rerank.json +1 -1
README.md
CHANGED
|
@@ -3,7 +3,7 @@ license: apache-2.0
|
|
| 3 |
language:
|
| 4 |
- km
|
| 5 |
library_name: transformers
|
| 6 |
-
pipeline_tag:
|
| 7 |
tags:
|
| 8 |
- transliteration
|
| 9 |
- romanization
|
|
@@ -26,7 +26,7 @@ model-index:
|
|
| 26 |
- name: Samdei Type Base
|
| 27 |
results:
|
| 28 |
- task:
|
| 29 |
-
type:
|
| 30 |
name: Romanized Khmer to Khmer script
|
| 31 |
dataset:
|
| 32 |
type: samdei/samdei-type-bench
|
|
@@ -43,97 +43,16 @@ model-index:
|
|
| 43 |
|
| 44 |
# Samdei Type Base
|
| 45 |
|
| 46 |
-
Samdei Type Base converts romanized Khmer, as people type it in chat, into Khmer script:
|
| 47 |
|
| 48 |
| Input | Output |
|
| 49 |
|---|---|
|
| 50 |
| `knhom jong tov phsar` | αααα»αα
αααα
ααααΆα |
|
| 51 |
| `ot mean wifi te` | α’ααααΆα wifi αα |
|
| 52 |
|
| 53 |
-
|
| 54 |
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
| Model | Parameters | Released as | Samdei Type Bench CER / exact |
|
| 58 |
-
|---|---|---|---|
|
| 59 |
-
| Samdei Type Mini | 7.1M | int8 ONNX + pruned word LM: [`samdei/samdei-type-mini`](https://huggingface.co/samdei/samdei-type-mini) | 0.058 / 24% |
|
| 60 |
-
| **Samdei Type Base** (this repo) | 18.2M | PyTorch + int8 ONNX + LMs + channel model | 0.052 / 30% |
|
| 61 |
-
| Samdei Type Prime | 18.2M | not released | 0.027 / 61% |
|
| 62 |
-
|
| 63 |
-
Samdei Type Prime is Base trained further on scenario-designed, LLM-drafted, human-audited data. It is not released.
|
| 64 |
-
|
| 65 |
-
The pretrained (stage-1) checkpoints, for fine-tuning your own converter, are [`samdei/samdei-type-base-pretrained`](https://huggingface.co/samdei/samdei-type-base-pretrained) (this size) and [`samdei/samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained).
|
| 66 |
-
|
| 67 |
-
Paper: *Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration* (`<paper-url>`). Code: `<code-repo-url>`.
|
| 68 |
-
|
| 69 |
-
## Files
|
| 70 |
-
|
| 71 |
-
```
|
| 72 |
-
config.json configuration and vocabularies (transformers format)
|
| 73 |
-
generation_config.json beam search, k = 5
|
| 74 |
-
model.safetensors 71.1 MiB weights (fp32)
|
| 75 |
-
tokenizer.json character tokenizer
|
| 76 |
-
tokenizer_config.json its settings
|
| 77 |
-
onnx/encoder_kv.int8.onnx 11.3 MiB input characters -> cross-attention K/V of every decoder layer
|
| 78 |
-
onnx/decoder_step.int8.onnx 6.3 MiB one decoder step with a self-attention KV cache
|
| 79 |
-
onnx/vocab.json vocabularies and graph shapes
|
| 80 |
-
channel/ 28.4 MiB reverse channel model (Khmer -> romanized), 7.1M parameters, used in reranking; the same five files
|
| 81 |
-
lm/char5_nowiki.npz 8.9 MiB hashed character 5-gram LM
|
| 82 |
-
lm/wordfull_nowiki.json.gz 12.2 MiB word bigram LM over khmercut words (counts, gzipped JSON)
|
| 83 |
-
rerank.json decoding and rerank settings used for every number below
|
| 84 |
-
```
|
| 85 |
-
|
| 86 |
-
## Intended use
|
| 87 |
-
|
| 88 |
-
- **Converting romanized Khmer chat into Khmer script** on a desktop or server, where tens of MB of LMs and a second model for reranking are affordable.
|
| 89 |
-
- **A starting point for fine-tuning:** the weights can be trained further with `scripts/train.py --init <downloaded folder>`.
|
| 90 |
-
- **Mixed text:** English words, numbers and brand names are copied through unchanged.
|
| 91 |
-
|
| 92 |
-
For a phone keyboard, use [Samdei Type Mini](https://huggingface.co/samdei/samdei-type-mini).
|
| 93 |
-
|
| 94 |
-
**Out of scope:**
|
| 95 |
-
- the reverse direction (Khmer β Latin);
|
| 96 |
-
- official romanization standards;
|
| 97 |
-
- any setting where the user cannot see and correct the output (see Limitations).
|
| 98 |
-
|
| 99 |
-
## How to use
|
| 100 |
-
|
| 101 |
-
Install the `rkhmer` package from the code repository:
|
| 102 |
-
|
| 103 |
-
```bash
|
| 104 |
-
pip install "rkhmer[deploy] @ git+<code-repo-url>"
|
| 105 |
-
```
|
| 106 |
-
|
| 107 |
-
**PyTorch, with the full reranker (the setting of every number below):**
|
| 108 |
-
|
| 109 |
-
```python
|
| 110 |
-
from huggingface_hub import snapshot_download
|
| 111 |
-
from rkhmer.evaluation import chunk_source, translate
|
| 112 |
-
from rkhmer.model import load
|
| 113 |
-
from rkhmer.rerank import Reranker, load_weights
|
| 114 |
-
|
| 115 |
-
d = snapshot_download("samdei/samdei-type-base")
|
| 116 |
-
model, sv, tv, _ = load(d, "cpu") # config.json + model.safetensors
|
| 117 |
-
rr = Reranker(load_weights(f"{d}/rerank.json"), charlm=f"{d}/lm/char5_nowiki.npz",
|
| 118 |
-
wordlm=f"{d}/lm/wordfull_nowiki.json.gz", channel=f"{d}/channel")
|
| 119 |
-
|
| 120 |
-
def convert(text: str) -> str:
|
| 121 |
-
pieces = chunk_source(text, 8) # at most 8 words per piece, break after punctuation
|
| 122 |
-
_, nbest = translate(model, sv, tv, pieces, "cpu", beam=5)
|
| 123 |
-
return " ".join(rr.rerank(p, hyps)[0][0] for p, hyps in zip(pieces, nbest))
|
| 124 |
-
|
| 125 |
-
print(convert("knhom jong tov phsar")) # αααα»αα
αααα
ααααΆα
|
| 126 |
-
```
|
| 127 |
-
|
| 128 |
-
**Int8 ONNX** (no torch for the converter itself; the channel model still needs torch):
|
| 129 |
-
|
| 130 |
-
```python
|
| 131 |
-
from rkhmer.onnx_infer import OnnxConverter
|
| 132 |
-
conv = OnnxConverter(f"{d}/onnx")
|
| 133 |
-
hyps = conv.beam("knhom jong tov phsar", k=5) # rerank with rr.rerank(...) as above
|
| 134 |
-
```
|
| 135 |
-
|
| 136 |
-
**transformers, without rkhmer** (`pip install transformers torch`; see Model). This is beam search with k = 5 and no reranking: 0.062 CER / 24% exact on Samdei Type Bench, against 0.052 / 30% with the reranker above.
|
| 137 |
|
| 138 |
```python
|
| 139 |
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
|
@@ -162,122 +81,77 @@ def convert(text: str) -> str:
|
|
| 162 |
print(convert("knhom jong tov phsar")) # αααα»αα
αααα
ααααΆα
|
| 163 |
```
|
| 164 |
|
| 165 |
-
|
| 166 |
-
- Beam search with k = 5.
|
| 167 |
-
- The input is cut into pieces of at most eight words, with a break after any punctuation mark. Each piece is decoded and reranked on its own, and the outputs are joined with a space.
|
| 168 |
-
- Each hypothesis y for input x is scored as `log P(y|x) + 0.3 Β· log P_char(y) + 0.3 Β· log P_word(y) β 1.0 Β· #OOV(y) + 0.3 Β· log P_ch(x|y)`. The weights were tuned on a closed development set, and `rerank.json` holds them.
|
| 169 |
-
- The channel term checks that a hypothesis explains all of the input. It keeps the LMs from replacing a rare word with a frequent one.
|
| 170 |
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
| | Samdei Type Base | channel model |
|
| 176 |
-
|---|---|---|
|
| 177 |
-
| Direction | romanized β Khmer | Khmer β romanized |
|
| 178 |
-
| Architecture | pre-LN Transformer encoder-decoder, GELU, learned positions up to 256, untied output projection | same |
|
| 179 |
-
| d_model / heads | 384 / 6 | 256 / 4 |
|
| 180 |
-
| Encoder / decoder layers | 6 / 3 | 6 / 2 |
|
| 181 |
-
| FFN width | 1536 | 1024 |
|
| 182 |
-
| Parameters | 18,202,980 | 7,146,712 |
|
| 183 |
-
| Vocabulary | 216 input, 228 output symbols (incl. special tokens) | 228 input, 216 output |
|
| 184 |
-
| int8 ONNX | 18.2 MiB | β |
|
| 185 |
-
|
| 186 |
-
Int8 quantization does not change CER on Dev-Sentences with beam 5 and no LMs (0.007 in fp32 and in int8). The cached ONNX decoder reproduces full re-decoding exactly.
|
| 187 |
-
|
| 188 |
-
**transformers format.** The model folders load with `AutoModelForSeq2SeqLM` and `AutoTokenizer` and need no custom code. They use transformers' Blenderbot classes only because the architecture is the same (a pre-LN encoder-decoder with scaled embeddings, learned positions from 0 and no embedding LayerNorm). The model has nothing else in common with BlenderBot. Blenderbot has one vocabulary for both sides, so the tokenizer stacks the two: ids 0β227 are the output symbols and the input symbols follow. The output layer gives input ids a fixed logit of β10β΄, so they are never generated. For the same reason transformers reports 18.63M (7.43M for the channel model) parameters: the stacked embedding tables and the output layer carry rows that are never used. `rkhmer` reads the same files and maps them back to its own layout.
|
| 189 |
-
## Training
|
| 190 |
|
| 191 |
-
|
|
|
|
| 192 |
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
| Replayed | β | β | the stage-2 mix |
|
| 197 |
-
| Total pairs | 12.8M | 1.16M | 1.59M |
|
| 198 |
-
| Steps (best checkpoint) | 60k (60k) | 6k (4k) | 3k (3k) |
|
| 199 |
-
| Peak lr, warmup | 7e-4, 3k | 2e-4, 300 | 1e-4, 100 |
|
| 200 |
-
| Wall clock | 4.9 h | about 31β36 min | about 10β12 min |
|
| 201 |
|
| 202 |
-
|
| 203 |
-
- AdamW (Ξ² 0.9 / 0.98, weight decay 0.01), cosine decay to 5% of the peak, label smoothing 0.1;
|
| 204 |
-
- about 12k tokens per batch, bf16 autocast;
|
| 205 |
-
- one laptop GPU (Apple M5 Pro).
|
| 206 |
|
| 207 |
-
|
| 208 |
-
- Stages 1β2: the lowest mean greedy CER on Dev-Sentences and Akkhara-Preview. Akkhara-Preview is the 110-row public preview of Akkhara-Sing-Khmer; it was used for this selection only and never trained on.
|
| 209 |
-
- Stage 3: the lowest greedy CER on the first 400 messages of a closed development set.
|
| 210 |
|
| 211 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 212 |
|
| 213 |
-
|
| 214 |
|
| 215 |
-
|
| 216 |
-
- **Stage 1:** [`samdei/samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data), configs `generator_v1` and `llm_chat_sentences`. The pretrained checkpoint is [`samdei/samdei-type-base-pretrained`](https://huggingface.co/samdei/samdei-type-base-pretrained).
|
| 217 |
-
- **Stages 2β3:** [`samdei/samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data). Base uses `mined_tables_chh_wd`.
|
| 218 |
|
| 219 |
-
|
| 220 |
-
- **Khmer targets.** Always real text, except the 2,483 LLM-written chat sentences. The text comes from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT.
|
| 221 |
-
- **Romanized inputs.** These come from:
|
| 222 |
-
- the persona generator, which uses Google's Khmer pronunciation lexicon;
|
| 223 |
-
- spelling tables mined from public word lists;
|
| 224 |
-
- Gemma 4 romanizations of real sentences;
|
| 225 |
-
- chat-like abbreviations.
|
| 226 |
-
- **Not used:** no evaluation set, and no data of other romanization systems (in particular, no Akkhara-Sing-Khmer data).
|
| 227 |
|
| 228 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 229 |
|
| 230 |
-
|
|
|
|
|
|
|
| 231 |
|
| 232 |
-
|
| 233 |
-
- **Same code for every system:** `rkhmer.evaluation.score`.
|
| 234 |
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
| Akkhara-mini, 6-word pieces | 17.8M | 17.6 MiB | 0.066 | 23% |
|
| 242 |
-
| AkaraAlpha, word by word | 0.08M | β | 0.281 | 2% |
|
| 243 |
|
| 244 |
-
|
| 245 |
-
- **
|
| 246 |
-
- **Punctuation.** Without punctuation in the target and the output, Base's CER is 0.046.
|
| 247 |
-
- **Decode long messages in pieces.** Base was trained on short pieces (inputs of at most 132 characters, median 26). With beam 5 and no LMs, whole-input CER is 0.133 overall and 0.399 on inputs over 132 characters, against 0.062 and 0.070 in pieces.
|
| 248 |
-
- **Reference sets** (CER / exact): Dev-Sentences 0.004 / 96%, Chat-Words 0.077 / 84%, Dict-Words 0.246 / 41%, Casual-Synth 0.044 / 69%.
|
| 249 |
-
- **Speed.** One M5 Pro CPU thread, int8 ONNX, without reranking:
|
| 250 |
-
- greedy: 4.4 ms p50 / 9.1 ms p95 per input;
|
| 251 |
-
- beam 5: 9.3 / 19.1 ms.
|
| 252 |
|
| 253 |
## Limitations
|
| 254 |
|
| 255 |
-
-
|
| 256 |
-
-
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
|
| 261 |
-
- **Spoken-register words that share a spelling with a written word:** `nung` (α αααΉα in chat) can come out as αα·α ("and").
|
| 262 |
-
- **Loanwords typed in English** (`check` for ααα) are copied through as English.
|
| 263 |
-
- **Punctuation.** Base copies a Latin comma where Khmer uses a space, and never adds a final α that the input lacks.
|
| 264 |
-
- **Rare dictionary words without context**, many of them Pali-Sanskrit, remain hard (Dict-Words CER 0.246).
|
| 265 |
-
- **Web text.** The Khmer side comes mostly from web text and carries its topics and biases.
|
| 266 |
|
| 267 |
-
## License
|
| 268 |
|
| 269 |
-
|
| 270 |
-
- **
|
| 271 |
-
|
| 272 |
-
- FineWeb-2 (ODC-By 1.0);
|
| 273 |
-
- ALT (CC BY 4.0);
|
| 274 |
-
- our LLM-written chat sentences (CC BY 4.0; written with Gemma 4, Apache-2.0).
|
| 275 |
-
- **Training data:** CC BY-SA 4.0; see the dataset cards. Its sources are:
|
| 276 |
-
- HPLT v2, FineWeb-2, Khmer Wikipedia (CC BY-SA) and ALT;
|
| 277 |
-
- Google's Khmer pronunciation lexicon (CC BY 4.0);
|
| 278 |
-
- the Chhunneng romanized Khmer dataset (MIT, Β© 2025 Chrea Chanchhunneng);
|
| 279 |
-
- Wikidata (CC0);
|
| 280 |
-
- Gemma 4 output (Apache-2.0).
|
| 281 |
- **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
|
| 282 |
|
| 283 |
## Citation
|
|
@@ -285,10 +159,8 @@ Each repo's `recipes.json` gives the exact mix.
|
|
| 285 |
```bibtex
|
| 286 |
@misc{samdei-type-2026,
|
| 287 |
title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
|
| 288 |
-
author = {
|
| 289 |
year = {2026},
|
| 290 |
url = {<paper-url>}
|
| 291 |
}
|
| 292 |
```
|
| 293 |
-
|
| 294 |
-
Contact: `<contact>`
|
|
|
|
| 3 |
language:
|
| 4 |
- km
|
| 5 |
library_name: transformers
|
| 6 |
+
pipeline_tag: translation
|
| 7 |
tags:
|
| 8 |
- transliteration
|
| 9 |
- romanization
|
|
|
|
| 26 |
- name: Samdei Type Base
|
| 27 |
results:
|
| 28 |
- task:
|
| 29 |
+
type: translation
|
| 30 |
name: Romanized Khmer to Khmer script
|
| 31 |
dataset:
|
| 32 |
type: samdei/samdei-type-bench
|
|
|
|
| 43 |
|
| 44 |
# Samdei Type Base
|
| 45 |
|
| 46 |
+
Samdei Type Base converts romanized Khmer, as people type it in chat, into Khmer script. It is an 18.2M-parameter character-level encoder-decoder, the most accurate open Samdei Type model, for desktop and server use. For a phone, use [Samdei Type Mini](https://huggingface.co/samdei/samdei-type-mini).
|
| 47 |
|
| 48 |
| Input | Output |
|
| 49 |
|---|---|
|
| 50 |
| `knhom jong tov phsar` | αααα»αα
αααα
ααααΆα |
|
| 51 |
| `ot mean wifi te` | α’ααααΆα wifi αα |
|
| 52 |
|
| 53 |
+
## Quick start
|
| 54 |
|
| 55 |
+
With transformers alone (`pip install transformers torch`):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
|
| 57 |
```python
|
| 58 |
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
|
|
|
| 81 |
print(convert("knhom jong tov phsar")) # αααα»αα
αααα
ααααΆα
|
| 82 |
```
|
| 83 |
|
| 84 |
+
This is beam search alone (CER 0.062 on Samdei Type Bench). For the best quality, rerank with the two LMs and the channel model of this repo (0.052), using the [`samdei-type`](https://github.com/Samdei-Labs/samdei-type) package:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
+
```bash
|
| 87 |
+
pip install "samdei-type @ git+https://github.com/Samdei-Labs/samdei-type.git"
|
| 88 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
+
```python
|
| 91 |
+
from samdei_type.pipeline import Converter
|
| 92 |
|
| 93 |
+
conv = Converter("samdei/samdei-type-base") # PyTorch weights + full rerank, as in the paper
|
| 94 |
+
print(conv.convert("knhom jong tov phsar")) # αααα»αα
αααα
ααααΆα
|
| 95 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 96 |
|
| 97 |
+
Or on the command line: `samdei-type -m samdei/samdei-type-base "knhom jong tov phsar"`.
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
+
## Files
|
|
|
|
|
|
|
| 100 |
|
| 101 |
+
```
|
| 102 |
+
config.json, generation_config.json, model.safetensors (71.1 MiB), tokenizer*.json fp32 model (transformers)
|
| 103 |
+
onnx/encoder_kv.int8.onnx (11.3 MiB), onnx/decoder_step.int8.onnx (6.3 MiB), onnx/vocab.json int8 ONNX
|
| 104 |
+
channel/ (28.4 MiB) reverse model (Khmer -> romanized) for reranking, same format
|
| 105 |
+
lm/char5_nowiki.npz (8.9 MiB) character 5-gram LM
|
| 106 |
+
lm/wordfull_nowiki.json.gz (12.2 MiB) word bigram LM
|
| 107 |
+
rerank.json decoding and rerank settings
|
| 108 |
+
```
|
| 109 |
|
| 110 |
+
Reranking scores each beam hypothesis y for input x as `log P(y|x) + 0.3 log P_char(y) + 0.3 log P_word(y) β 1.0 #OOV(y) + 0.3 log P_ch(x|y)`. The channel term checks that y explains all of the input, so the LMs cannot swap a rare word for a frequent one. Nothing in this repo is a pickle.
|
| 111 |
|
| 112 |
+
## Results
|
|
|
|
|
|
|
| 113 |
|
| 114 |
+
[Samdei Type Bench](https://huggingface.co/datasets/samdei/samdei-type-bench): 2,155 everyday messages and words with human-checked Khmer. The metrics are CER after normalization and exact match.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 115 |
|
| 116 |
+
| System | CER | Exact |
|
| 117 |
+
|---|---|---|
|
| 118 |
+
| **Samdei Type Base**, reranked | **0.052** | **30%** |
|
| 119 |
+
| Samdei Type Base, beam only (the transformers example above) | 0.062 | 24% |
|
| 120 |
+
| [Samdei Type Mini](https://huggingface.co/samdei/samdei-type-mini), reranked | 0.058 | 24% |
|
| 121 |
+
| Samdei Type Prime (not released) | 0.027 | 61% |
|
| 122 |
+
| Akkhara-mini, 6-word pieces (0.159 on whole inputs) | 0.066 | 23% |
|
| 123 |
+
| AkaraAlpha | 0.281 | 2% |
|
| 124 |
|
| 125 |
+
- Prime is Base trained further on scenario-designed, LLM-drafted, human-audited data. Most bench items share that scenario design, so its lead is an upper bound.
|
| 126 |
+
- Without punctuation, Base's CER is 0.046. Decode long messages in pieces, as above. On whole inputs, beam-only CER is 0.133.
|
| 127 |
+
- On one Apple M5 Pro CPU thread with int8 ONNX, beam 5 takes 9.3 ms per input (p50).
|
| 128 |
|
| 129 |
+
## Model and training
|
|
|
|
| 130 |
|
| 131 |
+
- **Architecture.** Pre-LN Transformer encoder-decoder: 6 encoder and 3 decoder layers, d_model 384, 6 heads, FFN 1536. It has 216 input and 228 output symbols. The channel model is Mini-sized (7.1M) and reads in the reverse direction.
|
| 132 |
+
- **transformers format.** It loads as `BlenderbotForConditionalGeneration` because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 18.63M parameters.
|
| 133 |
+
- **Training.** Three stages on one laptop GPU (Apple M5 Pro), about 6 hours in all:
|
| 134 |
+
1. pretraining on 12.8M pairs: real Khmer sentences spelled by a rule-based typing generator ([`samdei-type-base-pretrained`](https://huggingface.co/samdei/samdei-type-base-pretrained));
|
| 135 |
+
2. fine-tuning on 1.16M chat-like pairs;
|
| 136 |
+
3. adaptation to casual register (1.59M pairs, with replay).
|
|
|
|
|
|
|
| 137 |
|
| 138 |
+
The rerank LMs are built from HPLT v2, FineWeb-2, ALT and our LLM-written chat sentences, without Wikipedia.
|
| 139 |
+
- **Data.** The exact training files are in [`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data) and [`samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data), with `recipes.json`. No evaluation data and no other romanization system's data was trained on. Akkhara-Preview, a public 110-row preview, was used only to pick stage 1β2 checkpoints; stage 3 used a closed development set.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
|
| 141 |
## Limitations
|
| 142 |
|
| 143 |
+
- Every romanized input seen in training is synthetic (rules, mined spelling tables, an LLM). Real typing appears only in the evaluation.
|
| 144 |
+
- Homophones that need more context than a word bigram: ααΎα / ααα (`der`), ααΉα / ααα / ααα (`dek`).
|
| 145 |
+
- Loanwords typed in English (`check` for ααα) stay English. A Latin comma is copied, and a missing final α is never added.
|
| 146 |
+
- Rare dictionary words without context are hard, many of them Pali-Sanskrit.
|
| 147 |
+
- The Khmer side is mostly web text, with its topics and biases.
|
| 148 |
+
- Not for Khmer β Latin, official romanization of names, or output nobody checks.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 149 |
|
| 150 |
+
## License
|
| 151 |
|
| 152 |
+
Apache-2.0 (`LICENSE`; keep `NOTICE` when redistributing).
|
| 153 |
+
- **LMs:** built without Wikipedia, from HPLT v2, FineWeb-2 (ODC-By), ALT (CC BY) and our LLM-written chat sentences.
|
| 154 |
+
- **Training data:** CC BY-SA 4.0, with its sources in the dataset cards.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 155 |
- **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
|
| 156 |
|
| 157 |
## Citation
|
|
|
|
| 159 |
```bibtex
|
| 160 |
@misc{samdei-type-2026,
|
| 161 |
title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
|
| 162 |
+
author = {Mengsay Loem},
|
| 163 |
year = {2026},
|
| 164 |
url = {<paper-url>}
|
| 165 |
}
|
| 166 |
```
|
|
|
|
|
|
rerank.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
| 2 |
"model": "Samdei Type Base",
|
| 3 |
"beam": 5,
|
| 4 |
"chunk_max_words": 8,
|
| 5 |
-
"chunk_rule": "
|
| 6 |
"rerank_weights": {"char": 0.3, "word": 0.3, "oov": 1.0, "channel": 0.3},
|
| 7 |
"score": "log P(y|x) + char*log P_char(y) + word*log P_word(y) - oov*#OOV(y) + channel*log P_ch(x|y)",
|
| 8 |
"char_lm": "lm/char5_nowiki.npz",
|
|
|
|
| 2 |
"model": "Samdei Type Base",
|
| 3 |
"beam": 5,
|
| 4 |
"chunk_max_words": 8,
|
| 5 |
+
"chunk_rule": "samdei_type.evaluation.chunk_source: at most 8 words per piece, break after any punctuation mark; pieces are decoded and reranked on their own and joined with a space",
|
| 6 |
"rerank_weights": {"char": 0.3, "word": 0.3, "oov": 1.0, "channel": 0.3},
|
| 7 |
"score": "log P(y|x) + char*log P_char(y) + word*log P_word(y) - oov*#OOV(y) + channel*log P_ch(x|y)",
|
| 8 |
"char_lm": "lm/char5_nowiki.npz",
|