Translation
Transformers
ONNX
Safetensors
Khmer
blenderbot
text2text-generation
transliteration
romanization
romanized-khmer
khmer
character-level
int8
keyboard
on-device
Eval Results (legacy)
Instructions to use samdei/samdei-type-mini with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use samdei/samdei-type-mini with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="samdei/samdei-type-mini")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("samdei/samdei-type-mini") model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-mini", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Shorter cards with a transformers quick start; translation task tag; author; samdei-type package
Browse files- README.md +54 -173
- rerank.json +1 -1
README.md
CHANGED
|
@@ -3,7 +3,7 @@ license: apache-2.0
|
|
| 3 |
language:
|
| 4 |
- km
|
| 5 |
library_name: transformers
|
| 6 |
-
pipeline_tag:
|
| 7 |
tags:
|
| 8 |
- transliteration
|
| 9 |
- romanization
|
|
@@ -27,7 +27,7 @@ model-index:
|
|
| 27 |
- name: Samdei Type Mini
|
| 28 |
results:
|
| 29 |
- task:
|
| 30 |
-
type:
|
| 31 |
name: Romanized Khmer to Khmer script
|
| 32 |
dataset:
|
| 33 |
type: samdei/samdei-type-bench
|
|
@@ -44,97 +44,16 @@ model-index:
|
|
| 44 |
|
| 45 |
# Samdei Type Mini
|
| 46 |
|
| 47 |
-
Samdei Type Mini converts romanized Khmer, as people type it in chat, into Khmer script:
|
| 48 |
|
| 49 |
| Input | Output |
|
| 50 |
|---|---|
|
| 51 |
| `knhom jong tov phsar` | αααα»αα
αααα
ααααΆα |
|
| 52 |
| `ot mean wifi te` | α’ααααΆα wifi αα |
|
| 53 |
|
| 54 |
-
|
| 55 |
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
| Model | Parameters | Released as | Samdei Type Bench CER / exact |
|
| 59 |
-
|---|---|---|---|
|
| 60 |
-
| **Samdei Type Mini** (this repo) | 7.1M | int8 ONNX + pruned word LM + PyTorch | 0.058 / 24% |
|
| 61 |
-
| Samdei Type Base | 18.2M | PyTorch + int8 ONNX + LMs + channel model: [`samdei/samdei-type-base`](https://huggingface.co/samdei/samdei-type-base) | 0.052 / 30% |
|
| 62 |
-
| Samdei Type Prime | 18.2M | not released | 0.027 / 61% |
|
| 63 |
-
|
| 64 |
-
Samdei Type Prime is Base trained further on scenario-designed, LLM-drafted, human-audited data. It is not released.
|
| 65 |
-
|
| 66 |
-
The pretrained (stage-1) checkpoints, for fine-tuning your own converter, are [`samdei/samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained) (this size) and [`samdei/samdei-type-base-pretrained`](https://huggingface.co/samdei/samdei-type-base-pretrained).
|
| 67 |
-
|
| 68 |
-
Paper: *Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration* (`<paper-url>`). Code: `<code-repo-url>`.
|
| 69 |
-
|
| 70 |
-
## Files
|
| 71 |
-
|
| 72 |
-
```
|
| 73 |
-
config.json configuration and vocabularies (transformers format)
|
| 74 |
-
generation_config.json beam search, k = 5
|
| 75 |
-
model.safetensors 28.4 MiB weights (fp32), for transformers, fine-tuning or the PyTorch reranker
|
| 76 |
-
tokenizer.json character tokenizer
|
| 77 |
-
tokenizer_config.json its settings
|
| 78 |
-
onnx/encoder_kv.int8.onnx 5.0 MiB input characters -> cross-attention K/V of every decoder layer
|
| 79 |
-
onnx/decoder_step.int8.onnx 2.0 MiB one decoder step with a self-attention KV cache
|
| 80 |
-
onnx/vocab.json vocabularies and graph shapes (layers, heads, head_dim, bos, eos, max_src_len)
|
| 81 |
-
lm/wordfull_nowiki_p25.json.gz 1.2 MiB word bigram LM over khmercut words, pruned to bigram count >= 25 (gzipped JSON)
|
| 82 |
-
rerank.json decoding and rerank settings used for every number below
|
| 83 |
-
```
|
| 84 |
-
|
| 85 |
-
## Intended use
|
| 86 |
-
|
| 87 |
-
- **Inside a keyboard:** turn what a user types in Latin letters into a list of Khmer-script candidates on every keystroke, offline.
|
| 88 |
-
- **Reading:** show someone else's romanized message in Khmer script.
|
| 89 |
-
- **Mixed text:** English words, numbers and brand names are copied through unchanged.
|
| 90 |
-
|
| 91 |
-
**Out of scope:**
|
| 92 |
-
- the reverse direction (Khmer β Latin);
|
| 93 |
-
- official romanization standards, such as names in documents;
|
| 94 |
-
- any setting where the user cannot see and correct the output. The model makes mistakes on homophones and rare words (see Limitations).
|
| 95 |
-
|
| 96 |
-
## How to use
|
| 97 |
-
|
| 98 |
-
Install the `rkhmer` package from the code repository:
|
| 99 |
-
|
| 100 |
-
```bash
|
| 101 |
-
pip install "rkhmer[deploy] @ git+<code-repo-url>"
|
| 102 |
-
```
|
| 103 |
-
|
| 104 |
-
```python
|
| 105 |
-
from huggingface_hub import snapshot_download
|
| 106 |
-
from rkhmer.evaluation import chunk_source
|
| 107 |
-
from rkhmer.onnx_infer import OnnxConverter
|
| 108 |
-
from rkhmer.rerank import Reranker, load_weights
|
| 109 |
-
|
| 110 |
-
d = snapshot_download("samdei/samdei-type-mini", allow_patterns=["onnx/*", "lm/*", "rerank.json"])
|
| 111 |
-
conv = OnnxConverter(f"{d}/onnx") # int8 graphs + vocab.json, numpy + onnxruntime only
|
| 112 |
-
rr = Reranker(load_weights(f"{d}/rerank.json"), wordlm=f"{d}/lm/wordfull_nowiki_p25.json.gz")
|
| 113 |
-
|
| 114 |
-
def convert(text: str) -> str:
|
| 115 |
-
out = []
|
| 116 |
-
for piece in chunk_source(text, 8): # at most 8 words per piece, break after punctuation
|
| 117 |
-
hyps = conv.beam(piece, k=5) # [(khmer, logprob), ...], best first
|
| 118 |
-
out.append(rr.rerank(piece, hyps)[0][0])
|
| 119 |
-
return " ".join(out)
|
| 120 |
-
|
| 121 |
-
print(convert("knhom jong tov phsar")) # αααα»αα
αααα
ααααΆα
|
| 122 |
-
```
|
| 123 |
-
|
| 124 |
-
**Decoding, as in the paper.**
|
| 125 |
-
- Beam search with k = 5.
|
| 126 |
-
- The input is cut into pieces of at most eight words, with a break after any punctuation mark. Each piece is decoded and reranked on its own, and the outputs are joined with a space.
|
| 127 |
-
- Each hypothesis y for input x is scored as `log P(y|x) + 0.3 Β· log P_word(y) β 1.0 Β· #OOV(y)`. The weights were tuned on a closed development set, and `rerank.json` holds them.
|
| 128 |
-
- Inputs are truncated to 249 characters.
|
| 129 |
-
|
| 130 |
-
**On a phone.**
|
| 131 |
-
- The two graphs are all a runtime needs.
|
| 132 |
-
- `encoder_kv` runs once per input.
|
| 133 |
-
- `decoder_step` maps the newest token, its position, the past self-attention K/V and the cross-attention K/V to the logits and the new K/V. It has a dynamic batch axis, so all beams run in one call.
|
| 134 |
-
- `rkhmer/onnx_infer.py` is a reference implementation in plain numpy (no torch) to port from.
|
| 135 |
-
- The cached decoder reproduces full re-decoding exactly. Int8 quantization does not change CER on Dev-Sentences with beam 5 and no LMs (0.010 in fp32 and in int8).
|
| 136 |
-
|
| 137 |
-
**transformers, without rkhmer.** The fp32 weights at the repo root run with `transformers` and `torch` alone (`pip install transformers torch`; see Model). This is beam search with k = 5 and no reranking: 0.072 CER / 17% exact on Samdei Type Bench, against 0.058 / 24% with the word LM above.
|
| 138 |
|
| 139 |
```python
|
| 140 |
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
|
@@ -163,108 +82,72 @@ def convert(text: str) -> str:
|
|
| 163 |
print(convert("knhom jong tov phsar")) # αααα»αα
αααα
ααααΆα
|
| 164 |
```
|
| 165 |
|
| 166 |
-
|
| 167 |
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
| d_model / heads | 256 / 4 |
|
| 172 |
-
| Encoder / decoder layers | 6 / 2 (deep encoder, shallow decoder) |
|
| 173 |
-
| FFN width | 1024 |
|
| 174 |
-
| Parameters | 7,149,540 |
|
| 175 |
-
| Vocabulary | 215 input symbols (Latin letters, digits, punctuation, space), 228 output symbols (Khmer code points, space, copied characters), including special tokens |
|
| 176 |
-
| int8 ONNX | 7.0 MiB as released (7.6 MiB in the paper, with the exporter's debug metadata) |
|
| 177 |
-
|
| 178 |
-
**transformers format.** The model folders load with `AutoModelForSeq2SeqLM` and `AutoTokenizer` and need no custom code. They use transformers' Blenderbot classes only because the architecture is the same (a pre-LN encoder-decoder with scaled embeddings, learned positions from 0 and no embedding LayerNorm). The model has nothing else in common with BlenderBot. Blenderbot has one vocabulary for both sides, so the tokenizer stacks the two: ids 0β227 are the output symbols and the input symbols follow. The output layer gives input ids a fixed logit of β10β΄, so they are never generated. For the same reason transformers reports 7.43M parameters: the stacked embedding tables and the output layer carry rows that are never used. `rkhmer` reads the same files and maps them back to its own layout.
|
| 179 |
|
| 180 |
-
|
|
|
|
| 181 |
|
| 182 |
-
|
|
|
|
|
|
|
| 183 |
|
| 184 |
-
|
| 185 |
-
|---|---|---|---|
|
| 186 |
-
| New data | rule-based persona generator applied to 1.41M web sentences and the 2.4k LLM-written chat sentences (6.50M pairs), + Gemma 4 romanizations of real sentences (5.7k Γ3) | chat-heavy generator sample 630k; Gemma 4 romanizations 18.3k Γ4; pairs spelled with the table mined from the Chhunneng word list 290k; chat-like abbreviated copies 140k + 16k Γ2 | casual-register pairs 427k (spoken-register web sentences, stretched letters) |
|
| 187 |
-
| Replayed | β | β | the stage-2 mix |
|
| 188 |
-
| Total pairs | 6.5M | 1.16M | 1.59M |
|
| 189 |
-
| Steps (best checkpoint) | 40k (35k) | 6k (4k) | 3k (2.5k) |
|
| 190 |
-
| Peak lr, warmup | 1e-3, 2k | 2e-4, 300 | 1e-4, 100 |
|
| 191 |
-
| Wall clock | 2.2 h | about 31β36 min | about 10β12 min |
|
| 192 |
|
| 193 |
-
|
| 194 |
-
- AdamW (Ξ² 0.9 / 0.98, weight decay 0.01), cosine decay to 5% of the peak, label smoothing 0.1;
|
| 195 |
-
- about 12k tokens per batch, bf16 autocast;
|
| 196 |
-
- one laptop GPU (Apple M5 Pro).
|
| 197 |
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
|
|
|
|
|
|
|
|
|
|
| 201 |
|
| 202 |
-
**
|
| 203 |
-
- **Stage 1:** [`samdei/samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data), configs `generator_v2` and `llm_romanized`. The pretrained checkpoint is [`samdei/samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained).
|
| 204 |
-
- **Stages 2β3:** [`samdei/samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data). Mini uses `mined_tables_chh`.
|
| 205 |
|
| 206 |
-
|
| 207 |
-
- **Khmer targets.** Always real text, except 2,483 chat sentences written by Gemma 4. The text comes from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT.
|
| 208 |
-
- **Romanized inputs.** These come from:
|
| 209 |
-
- the persona generator, which uses Google's Khmer pronunciation lexicon;
|
| 210 |
-
- a spelling table mined from the Chhunneng word list;
|
| 211 |
-
- Gemma 4 romanizations of real sentences;
|
| 212 |
-
- chat-like abbreviations.
|
| 213 |
-
- **Not used:** no evaluation set, and no data of other romanization systems (in particular, no Akkhara-Sing-Khmer data).
|
| 214 |
|
| 215 |
-
|
| 216 |
|
| 217 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 218 |
|
| 219 |
-
-
|
| 220 |
-
-
|
| 221 |
|
| 222 |
-
|
| 223 |
-
|---|---|---|---|---|
|
| 224 |
-
| **Samdei Type Mini** | 7.1M | 7.6 MiB | 0.058 | 24% |
|
| 225 |
-
| Samdei Type Base | 18.2M | 18.2 MiB | **0.052** | **30%** |
|
| 226 |
-
| Samdei Type Prime (not released) | 18.2M | 18.2 MiB | 0.027 | 61% |
|
| 227 |
-
| Akkhara-mini, whole input | 17.8M | 17.6 MiB | 0.159 | 27% |
|
| 228 |
-
| Akkhara-mini, 6-word pieces | 17.8M | 17.6 MiB | 0.066 | 23% |
|
| 229 |
-
| AkaraAlpha, word by word | 0.08M | β | 0.281 | 2% |
|
| 230 |
|
| 231 |
-
- **
|
| 232 |
-
- **
|
| 233 |
-
- **
|
| 234 |
-
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
- Per keystroke in the live demo, with the word-LM rerank: 5.7 ms p50 / 17.2 ms p95.
|
| 239 |
-
- Not yet measured on a phone.
|
| 240 |
|
| 241 |
## Limitations
|
| 242 |
|
| 243 |
-
-
|
| 244 |
-
-
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
|
| 248 |
-
|
| 249 |
-
- **Spoken-register words that share a spelling with a written word:** `nung` is α αααΉα in chat and αα·α ("and") in prose.
|
| 250 |
-
- **Loanwords typed in English** (`check` for ααα) are copied through as English.
|
| 251 |
-
- **Punctuation.** Mini copies a Latin comma where Khmer uses a space, and never adds a final α that the input lacks.
|
| 252 |
-
- **Heavy abbreviation and dictionary words.** Mini is weaker on input abbreviated more heavily than its training data and on rare dictionary words without context, many of them Pali-Sanskrit (Dict-Words CER 0.282).
|
| 253 |
-
- **Web text.** The Khmer side comes mostly from web text and carries its topics and biases.
|
| 254 |
|
| 255 |
-
## License
|
| 256 |
|
| 257 |
-
|
| 258 |
-
- **
|
| 259 |
-
|
| 260 |
-
- FineWeb-2 (ODC-By 1.0);
|
| 261 |
-
- ALT (CC BY 4.0);
|
| 262 |
-
- our LLM-written chat sentences (CC BY 4.0; written with Gemma 4, Apache-2.0).
|
| 263 |
-
- **Training data:** CC BY-SA 4.0; see the dataset cards. Its sources are:
|
| 264 |
-
- HPLT v2, FineWeb-2, Khmer Wikipedia (CC BY-SA) and ALT;
|
| 265 |
-
- Google's Khmer pronunciation lexicon (CC BY 4.0);
|
| 266 |
-
- the Chhunneng romanized Khmer dataset (MIT, Β© 2025 Chrea Chanchhunneng);
|
| 267 |
-
- Gemma 4 output (Apache-2.0).
|
| 268 |
- **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
|
| 269 |
|
| 270 |
## Citation
|
|
@@ -272,10 +155,8 @@ Each repo's `recipes.json` gives the exact mix.
|
|
| 272 |
```bibtex
|
| 273 |
@misc{samdei-type-2026,
|
| 274 |
title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
|
| 275 |
-
author = {
|
| 276 |
year = {2026},
|
| 277 |
url = {<paper-url>}
|
| 278 |
}
|
| 279 |
```
|
| 280 |
-
|
| 281 |
-
Contact: `<contact>`
|
|
|
|
| 3 |
language:
|
| 4 |
- km
|
| 5 |
library_name: transformers
|
| 6 |
+
pipeline_tag: translation
|
| 7 |
tags:
|
| 8 |
- transliteration
|
| 9 |
- romanization
|
|
|
|
| 27 |
- name: Samdei Type Mini
|
| 28 |
results:
|
| 29 |
- task:
|
| 30 |
+
type: translation
|
| 31 |
name: Romanized Khmer to Khmer script
|
| 32 |
dataset:
|
| 33 |
type: samdei/samdei-type-bench
|
|
|
|
| 44 |
|
| 45 |
# Samdei Type Mini
|
| 46 |
|
| 47 |
+
Samdei Type Mini converts romanized Khmer, as people type it in chat, into Khmer script. It is a 7.1M-parameter character-level encoder-decoder, small enough for a phone keyboard: 7 MiB as int8 ONNX.
|
| 48 |
|
| 49 |
| Input | Output |
|
| 50 |
|---|---|
|
| 51 |
| `knhom jong tov phsar` | αααα»αα
αααα
ααααΆα |
|
| 52 |
| `ot mean wifi te` | α’ααααΆα wifi αα |
|
| 53 |
|
| 54 |
+
## Quick start
|
| 55 |
|
| 56 |
+
With transformers alone (`pip install transformers torch`):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
|
| 58 |
```python
|
| 59 |
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
|
|
|
| 82 |
print(convert("knhom jong tov phsar")) # αααα»αα
αααα
ααααΆα
|
| 83 |
```
|
| 84 |
|
| 85 |
+
This is beam search alone (CER 0.072 on Samdei Type Bench). For the best quality, rerank with the word LM of this repo (0.058), using the [`samdei-type`](https://github.com/Samdei-Labs/samdei-type) package:
|
| 86 |
|
| 87 |
+
```bash
|
| 88 |
+
pip install "samdei-type[deploy] @ git+https://github.com/Samdei-Labs/samdei-type.git"
|
| 89 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
+
```python
|
| 92 |
+
from samdei_type.pipeline import Converter
|
| 93 |
|
| 94 |
+
conv = Converter("samdei/samdei-type-mini", backend="onnx") # int8 ONNX + word-LM rerank, as in the paper
|
| 95 |
+
print(conv.convert("knhom jong tov phsar")) # αααα»αα
αααα
ααααΆα
|
| 96 |
+
```
|
| 97 |
|
| 98 |
+
Or on the command line: `samdei-type "knhom jong tov phsar"`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
|
| 100 |
+
## Files
|
|
|
|
|
|
|
|
|
|
| 101 |
|
| 102 |
+
```
|
| 103 |
+
config.json, generation_config.json, model.safetensors (28.4 MiB), tokenizer*.json fp32 model (transformers)
|
| 104 |
+
onnx/encoder_kv.int8.onnx (5.0 MiB), onnx/decoder_step.int8.onnx (2.0 MiB), onnx/vocab.json int8 ONNX
|
| 105 |
+
lm/wordfull_nowiki_p25.json.gz (1.2 MiB) word bigram LM for reranking
|
| 106 |
+
rerank.json decoding and rerank settings
|
| 107 |
+
```
|
| 108 |
|
| 109 |
+
**On a phone,** the two ONNX graphs are all a runtime needs. `encoder_kv` runs once per input. `decoder_step` decodes one token with a KV cache, and its batch axis holds all beams. [`samdei_type/onnx_infer.py`](https://github.com/Samdei-Labs/samdei-type) is a plain-numpy reference to port from. On one Apple M5 Pro CPU thread, beam 5 takes 4.9 ms per input (p50); with the rerank, a keystroke takes 5.7 ms. It has not been timed on a phone yet.
|
|
|
|
|
|
|
| 110 |
|
| 111 |
+
## Results
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
+
[Samdei Type Bench](https://huggingface.co/datasets/samdei/samdei-type-bench): 2,155 everyday messages and words with human-checked Khmer. The metrics are CER after normalization and exact match.
|
| 114 |
|
| 115 |
+
| System | CER | Exact |
|
| 116 |
+
|---|---|---|
|
| 117 |
+
| **Samdei Type Mini**, reranked | 0.058 | 24% |
|
| 118 |
+
| Samdei Type Mini, beam only (the transformers example above) | 0.072 | 17% |
|
| 119 |
+
| [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base), reranked | 0.052 | 30% |
|
| 120 |
+
| Samdei Type Prime (not released) | 0.027 | 61% |
|
| 121 |
+
| Akkhara-mini, 6-word pieces (0.159 on whole inputs) | 0.066 | 23% |
|
| 122 |
+
| AkaraAlpha | 0.281 | 2% |
|
| 123 |
|
| 124 |
+
- Prime is Base trained further on scenario-designed, LLM-drafted, human-audited data. Most bench items share that scenario design, so its lead is an upper bound.
|
| 125 |
+
- Decode long messages in pieces, as above. On whole inputs, beam-only CER is 0.218.
|
| 126 |
|
| 127 |
+
## Model and training
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
|
| 129 |
+
- **Architecture.** Pre-LN Transformer encoder-decoder: 6 encoder and 2 decoder layers, d_model 256, 4 heads, FFN 1024. It has 215 input symbols (Latin letters, digits, punctuation) and 228 output symbols (Khmer characters, and copied characters).
|
| 130 |
+
- **transformers format.** It loads as `BlenderbotForConditionalGeneration` because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 7.43M parameters.
|
| 131 |
+
- **Training.** Three stages on one laptop GPU (Apple M5 Pro), about 3 hours in all:
|
| 132 |
+
1. pretraining on 6.5M pairs: real Khmer sentences spelled by a rule-based typing generator ([`samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained));
|
| 133 |
+
2. fine-tuning on 1.16M chat-like pairs;
|
| 134 |
+
3. adaptation to casual register (1.59M pairs, with replay).
|
| 135 |
+
- **Data.** The exact training files are in [`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data) and [`samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data), with `recipes.json`. No evaluation data and no other romanization system's data was trained on. Akkhara-Preview, a public 110-row preview, was used only to pick stage 1β2 checkpoints; stage 3 used a closed development set.
|
|
|
|
|
|
|
| 136 |
|
| 137 |
## Limitations
|
| 138 |
|
| 139 |
+
- Every romanized input seen in training is synthetic (rules, mined spelling tables, an LLM). Real typing appears only in the evaluation.
|
| 140 |
+
- Homophones that need more context than a word bigram: ααΎα / ααα (`der`), ααΉα / ααα / ααα (`dek`).
|
| 141 |
+
- Loanwords typed in English (`check` for ααα) stay English. A Latin comma is copied, and a missing final α is never added.
|
| 142 |
+
- Rare dictionary words without context are hard, as is heavier abbreviation than in training.
|
| 143 |
+
- The Khmer side is mostly web text, with its topics and biases.
|
| 144 |
+
- Not for Khmer β Latin, official romanization of names, or output nobody checks.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
|
| 146 |
+
## License
|
| 147 |
|
| 148 |
+
Apache-2.0 (`LICENSE`; keep `NOTICE` when redistributing).
|
| 149 |
+
- **Word LM:** built without Wikipedia, from HPLT v2, FineWeb-2 (ODC-By), ALT (CC BY) and our LLM-written chat sentences.
|
| 150 |
+
- **Training data:** CC BY-SA 4.0, with its sources in the dataset cards.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
- **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
|
| 152 |
|
| 153 |
## Citation
|
|
|
|
| 155 |
```bibtex
|
| 156 |
@misc{samdei-type-2026,
|
| 157 |
title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
|
| 158 |
+
author = {Mengsay Loem},
|
| 159 |
year = {2026},
|
| 160 |
url = {<paper-url>}
|
| 161 |
}
|
| 162 |
```
|
|
|
|
|
|
rerank.json
CHANGED
|
@@ -2,7 +2,7 @@
|
|
| 2 |
"model": "Samdei Type Mini",
|
| 3 |
"beam": 5,
|
| 4 |
"chunk_max_words": 8,
|
| 5 |
-
"chunk_rule": "
|
| 6 |
"rerank_weights": {"char": 0.0, "word": 0.3, "oov": 1.0, "channel": 0.0},
|
| 7 |
"score": "log P(y|x) + char*log P_char(y) + word*log P_word(y) - oov*#OOV(y) + channel*log P_ch(x|y)",
|
| 8 |
"char_lm": null,
|
|
|
|
| 2 |
"model": "Samdei Type Mini",
|
| 3 |
"beam": 5,
|
| 4 |
"chunk_max_words": 8,
|
| 5 |
+
"chunk_rule": "samdei_type.evaluation.chunk_source: at most 8 words per piece, break after any punctuation mark; pieces are decoded and reranked on their own and joined with a space",
|
| 6 |
"rerank_weights": {"char": 0.0, "word": 0.3, "oov": 1.0, "channel": 0.0},
|
| 7 |
"score": "log P(y|x) + char*log P_char(y) + word*log P_word(y) - oov*#OOV(y) + channel*log P_ch(x|y)",
|
| 8 |
"char_lm": null,
|