mengsay commited on
Commit
a3cc27d
Β·
verified Β·
1 Parent(s): 019ccf6

Shorter cards with a transformers quick start; translation task tag; author; samdei-type package

Browse files
Files changed (2) hide show
  1. README.md +58 -186
  2. rerank.json +1 -1
README.md CHANGED
@@ -3,7 +3,7 @@ license: apache-2.0
3
  language:
4
  - km
5
  library_name: transformers
6
- pipeline_tag: text2text-generation
7
  tags:
8
  - transliteration
9
  - romanization
@@ -26,7 +26,7 @@ model-index:
26
  - name: Samdei Type Base
27
  results:
28
  - task:
29
- type: text2text-generation
30
  name: Romanized Khmer to Khmer script
31
  dataset:
32
  type: samdei/samdei-type-bench
@@ -43,97 +43,16 @@ model-index:
43
 
44
  # Samdei Type Base
45
 
46
- Samdei Type Base converts romanized Khmer, as people type it in chat, into Khmer script:
47
 
48
  | Input | Output |
49
  |---|---|
50
  | `knhom jong tov phsar` | αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš |
51
  | `ot mean wifi te` | αž’αžαŸ‹αž˜αžΆαž“ wifi αž‘αŸ |
52
 
53
- It is an 18.2M-parameter character-level Transformer encoder-decoder. It reranks its candidates with two Khmer LMs and a reverse channel model. Base is the most accurate open Samdei Type model, meant for desktop and server use. It comes as a PyTorch checkpoint and as int8 ONNX (18.2 MiB).
54
 
55
- Samdei (αžŸαŸ†αžŠαžΈ, "what you say") is our team, and Samdei Type is the converter. The family has three models:
56
-
57
- | Model | Parameters | Released as | Samdei Type Bench CER / exact |
58
- |---|---|---|---|
59
- | Samdei Type Mini | 7.1M | int8 ONNX + pruned word LM: [`samdei/samdei-type-mini`](https://huggingface.co/samdei/samdei-type-mini) | 0.058 / 24% |
60
- | **Samdei Type Base** (this repo) | 18.2M | PyTorch + int8 ONNX + LMs + channel model | 0.052 / 30% |
61
- | Samdei Type Prime | 18.2M | not released | 0.027 / 61% |
62
-
63
- Samdei Type Prime is Base trained further on scenario-designed, LLM-drafted, human-audited data. It is not released.
64
-
65
- The pretrained (stage-1) checkpoints, for fine-tuning your own converter, are [`samdei/samdei-type-base-pretrained`](https://huggingface.co/samdei/samdei-type-base-pretrained) (this size) and [`samdei/samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained).
66
-
67
- Paper: *Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration* (`<paper-url>`). Code: `<code-repo-url>`.
68
-
69
- ## Files
70
-
71
- ```
72
- config.json configuration and vocabularies (transformers format)
73
- generation_config.json beam search, k = 5
74
- model.safetensors 71.1 MiB weights (fp32)
75
- tokenizer.json character tokenizer
76
- tokenizer_config.json its settings
77
- onnx/encoder_kv.int8.onnx 11.3 MiB input characters -> cross-attention K/V of every decoder layer
78
- onnx/decoder_step.int8.onnx 6.3 MiB one decoder step with a self-attention KV cache
79
- onnx/vocab.json vocabularies and graph shapes
80
- channel/ 28.4 MiB reverse channel model (Khmer -> romanized), 7.1M parameters, used in reranking; the same five files
81
- lm/char5_nowiki.npz 8.9 MiB hashed character 5-gram LM
82
- lm/wordfull_nowiki.json.gz 12.2 MiB word bigram LM over khmercut words (counts, gzipped JSON)
83
- rerank.json decoding and rerank settings used for every number below
84
- ```
85
-
86
- ## Intended use
87
-
88
- - **Converting romanized Khmer chat into Khmer script** on a desktop or server, where tens of MB of LMs and a second model for reranking are affordable.
89
- - **A starting point for fine-tuning:** the weights can be trained further with `scripts/train.py --init <downloaded folder>`.
90
- - **Mixed text:** English words, numbers and brand names are copied through unchanged.
91
-
92
- For a phone keyboard, use [Samdei Type Mini](https://huggingface.co/samdei/samdei-type-mini).
93
-
94
- **Out of scope:**
95
- - the reverse direction (Khmer β†’ Latin);
96
- - official romanization standards;
97
- - any setting where the user cannot see and correct the output (see Limitations).
98
-
99
- ## How to use
100
-
101
- Install the `rkhmer` package from the code repository:
102
-
103
- ```bash
104
- pip install "rkhmer[deploy] @ git+<code-repo-url>"
105
- ```
106
-
107
- **PyTorch, with the full reranker (the setting of every number below):**
108
-
109
- ```python
110
- from huggingface_hub import snapshot_download
111
- from rkhmer.evaluation import chunk_source, translate
112
- from rkhmer.model import load
113
- from rkhmer.rerank import Reranker, load_weights
114
-
115
- d = snapshot_download("samdei/samdei-type-base")
116
- model, sv, tv, _ = load(d, "cpu") # config.json + model.safetensors
117
- rr = Reranker(load_weights(f"{d}/rerank.json"), charlm=f"{d}/lm/char5_nowiki.npz",
118
- wordlm=f"{d}/lm/wordfull_nowiki.json.gz", channel=f"{d}/channel")
119
-
120
- def convert(text: str) -> str:
121
- pieces = chunk_source(text, 8) # at most 8 words per piece, break after punctuation
122
- _, nbest = translate(model, sv, tv, pieces, "cpu", beam=5)
123
- return " ".join(rr.rerank(p, hyps)[0][0] for p, hyps in zip(pieces, nbest))
124
-
125
- print(convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
126
- ```
127
-
128
- **Int8 ONNX** (no torch for the converter itself; the channel model still needs torch):
129
-
130
- ```python
131
- from rkhmer.onnx_infer import OnnxConverter
132
- conv = OnnxConverter(f"{d}/onnx")
133
- hyps = conv.beam("knhom jong tov phsar", k=5) # rerank with rr.rerank(...) as above
134
- ```
135
-
136
- **transformers, without rkhmer** (`pip install transformers torch`; see Model). This is beam search with k = 5 and no reranking: 0.062 CER / 24% exact on Samdei Type Bench, against 0.052 / 30% with the reranker above.
137
 
138
  ```python
139
  from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
@@ -162,122 +81,77 @@ def convert(text: str) -> str:
162
  print(convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
163
  ```
164
 
165
- **Decoding, as in the paper.**
166
- - Beam search with k = 5.
167
- - The input is cut into pieces of at most eight words, with a break after any punctuation mark. Each piece is decoded and reranked on its own, and the outputs are joined with a space.
168
- - Each hypothesis y for input x is scored as `log P(y|x) + 0.3 Β· log P_char(y) + 0.3 Β· log P_word(y) βˆ’ 1.0 Β· #OOV(y) + 0.3 Β· log P_ch(x|y)`. The weights were tuned on a closed development set, and `rerank.json` holds them.
169
- - The channel term checks that a hypothesis explains all of the input. It keeps the LMs from replacing a rare word with a frequent one.
170
 
171
- **No pickles.** The weights are safetensors, and the configuration, vocabularies and word LM are JSON; the char LM is plain numpy arrays. Nothing in this repository runs code when it is loaded. To get only the PyTorch weights: `rkhmer.model.load_pretrained("samdei/samdei-type-base")` (add `subfolder="channel"` for the channel model).
172
-
173
- ## Model
174
-
175
- | | Samdei Type Base | channel model |
176
- |---|---|---|
177
- | Direction | romanized β†’ Khmer | Khmer β†’ romanized |
178
- | Architecture | pre-LN Transformer encoder-decoder, GELU, learned positions up to 256, untied output projection | same |
179
- | d_model / heads | 384 / 6 | 256 / 4 |
180
- | Encoder / decoder layers | 6 / 3 | 6 / 2 |
181
- | FFN width | 1536 | 1024 |
182
- | Parameters | 18,202,980 | 7,146,712 |
183
- | Vocabulary | 216 input, 228 output symbols (incl. special tokens) | 228 input, 216 output |
184
- | int8 ONNX | 18.2 MiB | – |
185
-
186
- Int8 quantization does not change CER on Dev-Sentences with beam 5 and no LMs (0.007 in fp32 and in int8). The cached ONNX decoder reproduces full re-decoding exactly.
187
-
188
- **transformers format.** The model folders load with `AutoModelForSeq2SeqLM` and `AutoTokenizer` and need no custom code. They use transformers' Blenderbot classes only because the architecture is the same (a pre-LN encoder-decoder with scaled embeddings, learned positions from 0 and no embedding LayerNorm). The model has nothing else in common with BlenderBot. Blenderbot has one vocabulary for both sides, so the tokenizer stacks the two: ids 0–227 are the output symbols and the input symbols follow. The output layer gives input ids a fixed logit of βˆ’10⁴, so they are never generated. For the same reason transformers reports 18.63M (7.43M for the channel model) parameters: the stacked embedding tables and the output layer carry rows that are never used. `rkhmer` reads the same files and maps them back to its own layout.
189
- ## Training
190
 
191
- Base was trained in three stages. Each stage started from the best checkpoint of the stage before and replayed its data.
 
192
 
193
- | | Stage 1: pretraining | Stage 2: fine-tuning | Stage 3: adaptation |
194
- |---|---|---|---|
195
- | New data | rule-based persona generator applied to 1.41M web sentences, two seeds (6.29M pairs each), + the 2.4k LLM-written chat sentences spelled the same way, Γ—40 (189k pairs) | chat-heavy generator sample 630k; Gemma 4 romanizations of real sentences 18.3k Γ—4; pairs spelled with tables mined from the Chhunneng word list (70%) and Wikidata names (30%) 290k; chat-like abbreviated copies 140k + 16k Γ—2 | casual-register pairs 427k (spoken-register web sentences, stretched letters) |
196
- | Replayed | – | – | the stage-2 mix |
197
- | Total pairs | 12.8M | 1.16M | 1.59M |
198
- | Steps (best checkpoint) | 60k (60k) | 6k (4k) | 3k (3k) |
199
- | Peak lr, warmup | 7e-4, 3k | 2e-4, 300 | 1e-4, 100 |
200
- | Wall clock | 4.9 h | about 31–36 min | about 10–12 min |
201
 
202
- **Shared settings:**
203
- - AdamW (Ξ² 0.9 / 0.98, weight decay 0.01), cosine decay to 5% of the peak, label smoothing 0.1;
204
- - about 12k tokens per batch, bf16 autocast;
205
- - one laptop GPU (Apple M5 Pro).
206
 
207
- **Checkpoint selection:**
208
- - Stages 1–2: the lowest mean greedy CER on Dev-Sentences and Akkhara-Preview. Akkhara-Preview is the 110-row public preview of Akkhara-Sing-Khmer; it was used for this selection only and never trained on.
209
- - Stage 3: the lowest greedy CER on the first 400 messages of a closed development set.
210
 
211
- **Channel model.** Trained from scratch in the reverse direction on stage-1 data: 25k steps, best checkpoint at 20k, lr 1e-3.
 
 
 
 
 
 
 
212
 
213
- **Rerank LMs.** Built from HPLT v2, FineWeb-2, ALT and the LLM-written chat sentences. Leaving out Wikipedia (CC BY-SA) costs almost nothing.
214
 
215
- **Data.** The exact training files are published:
216
- - **Stage 1:** [`samdei/samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data), configs `generator_v1` and `llm_chat_sentences`. The pretrained checkpoint is [`samdei/samdei-type-base-pretrained`](https://huggingface.co/samdei/samdei-type-base-pretrained).
217
- - **Stages 2–3:** [`samdei/samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data). Base uses `mined_tables_chh_wd`.
218
 
219
- Each repo's `recipes.json` gives the exact mix.
220
- - **Khmer targets.** Always real text, except the 2,483 LLM-written chat sentences. The text comes from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT.
221
- - **Romanized inputs.** These come from:
222
- - the persona generator, which uses Google's Khmer pronunciation lexicon;
223
- - spelling tables mined from public word lists;
224
- - Gemma 4 romanizations of real sentences;
225
- - chat-like abbreviations.
226
- - **Not used:** no evaluation set, and no data of other romanization systems (in particular, no Akkhara-Sing-Khmer data).
227
 
228
- ## Evaluation
 
 
 
 
 
 
 
229
 
230
- **Samdei Type Bench** ([`samdei/samdei-type-bench`](https://huggingface.co/datasets/samdei/samdei-type-bench)) has 2,155 everyday messages and words with human-checked Khmer. 192 of them were typed blind by native speakers. Inputs have a median of 81 characters.
 
 
231
 
232
- - **Metric.** Character error rate (CER) after normalization (`khnormal`, spaces removed, lowercase, coeng-DA merged with coeng-TA, ឲ merged with ឱ) and exact match, on each item's main input and main target.
233
- - **Same code for every system:** `rkhmer.evaluation.score`.
234
 
235
- | System | Parameters | int8 size | CER | Exact |
236
- |---|---|---|---|---|
237
- | Samdei Type Mini | 7.1M | 7.6 MiB | 0.058 | 24% |
238
- | **Samdei Type Base** | 18.2M | 18.2 MiB | **0.052** | **30%** |
239
- | Samdei Type Prime (not released) | 18.2M | 18.2 MiB | 0.027 | 61% |
240
- | Akkhara-mini, whole input | 17.8M | 17.6 MiB | 0.159 | 27% |
241
- | Akkhara-mini, 6-word pieces | 17.8M | 17.6 MiB | 0.066 | 23% |
242
- | AkaraAlpha, word by word | 0.08M | – | 0.281 | 2% |
243
 
244
- - **Prime is partly in-domain.** Its training data shares the scenario design of most Samdei Type Bench items, so its lead is an upper bound.
245
- - **Reranking.** Beam 5 alone gives 0.062 / 24%; the four rerank features together bring it to 0.052 / 30%.
246
- - **Punctuation.** Without punctuation in the target and the output, Base's CER is 0.046.
247
- - **Decode long messages in pieces.** Base was trained on short pieces (inputs of at most 132 characters, median 26). With beam 5 and no LMs, whole-input CER is 0.133 overall and 0.399 on inputs over 132 characters, against 0.062 and 0.070 in pieces.
248
- - **Reference sets** (CER / exact): Dev-Sentences 0.004 / 96%, Chat-Words 0.077 / 84%, Dict-Words 0.246 / 41%, Casual-Synth 0.044 / 69%.
249
- - **Speed.** One M5 Pro CPU thread, int8 ONNX, without reranking:
250
- - greedy: 4.4 ms p50 / 9.1 ms p95 per input;
251
- - beam 5: 9.3 / 19.1 ms.
252
 
253
  ## Limitations
254
 
255
- - **Synthetic training inputs.** Every romanized input Base has seen comes from rules, mined tables or an LLM; real typing appears only in the evaluation.
256
- - **Homophones that need more context than a word bigram gives**, for example:
257
- - ដើរ / αžŠαŸ‚αžš (`der`);
258
- - αžŠαžΉαž€ / αžŠαŸ‚αž€ / αžŠαŸαž€ (`dek`);
259
- - αž’αžΆαž‘αž·αžαŸ’αž™ / ធតីត (`atit`);
260
- - αž”αžΆαž‘ / αž”αžΆαžαŸ‹ at the end of a message.
261
- - **Spoken-register words that share a spelling with a written word:** `nung` (αž αŸ’αž“αžΉαž„ in chat) can come out as αž“αž·αž„ ("and").
262
- - **Loanwords typed in English** (`check` for αž†αŸ‚αž€) are copied through as English.
263
- - **Punctuation.** Base copies a Latin comma where Khmer uses a space, and never adds a final αŸ” that the input lacks.
264
- - **Rare dictionary words without context**, many of them Pali-Sanskrit, remain hard (Dict-Words CER 0.246).
265
- - **Web text.** The Khmer side comes mostly from web text and carries its topics and biases.
266
 
267
- ## License and attribution
268
 
269
- - **Everything in this repo** (weights, ONNX graphs, channel model, LMs, rerank settings): Apache-2.0. The full text is in `LICENSE`. `NOTICE` holds the attributions, and redistributions must keep it.
270
- - **`lm/char5_nowiki.npz` and `lm/wordfull_nowiki.json.gz`:** n-gram counts built without Wikipedia, from:
271
- - HPLT v2 (CC0 packaging; Common Crawl terms apply to the text);
272
- - FineWeb-2 (ODC-By 1.0);
273
- - ALT (CC BY 4.0);
274
- - our LLM-written chat sentences (CC BY 4.0; written with Gemma 4, Apache-2.0).
275
- - **Training data:** CC BY-SA 4.0; see the dataset cards. Its sources are:
276
- - HPLT v2, FineWeb-2, Khmer Wikipedia (CC BY-SA) and ALT;
277
- - Google's Khmer pronunciation lexicon (CC BY 4.0);
278
- - the Chhunneng romanized Khmer dataset (MIT, Β© 2025 Chrea Chanchhunneng);
279
- - Wikidata (CC0);
280
- - Gemma 4 output (Apache-2.0).
281
  - **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
282
 
283
  ## Citation
@@ -285,10 +159,8 @@ Each repo's `recipes.json` gives the exact mix.
285
  ```bibtex
286
  @misc{samdei-type-2026,
287
  title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
288
- author = {<authors>},
289
  year = {2026},
290
  url = {<paper-url>}
291
  }
292
  ```
293
-
294
- Contact: `<contact>`
 
3
  language:
4
  - km
5
  library_name: transformers
6
+ pipeline_tag: translation
7
  tags:
8
  - transliteration
9
  - romanization
 
26
  - name: Samdei Type Base
27
  results:
28
  - task:
29
+ type: translation
30
  name: Romanized Khmer to Khmer script
31
  dataset:
32
  type: samdei/samdei-type-bench
 
43
 
44
  # Samdei Type Base
45
 
46
+ Samdei Type Base converts romanized Khmer, as people type it in chat, into Khmer script. It is an 18.2M-parameter character-level encoder-decoder, the most accurate open Samdei Type model, for desktop and server use. For a phone, use [Samdei Type Mini](https://huggingface.co/samdei/samdei-type-mini).
47
 
48
  | Input | Output |
49
  |---|---|
50
  | `knhom jong tov phsar` | αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš |
51
  | `ot mean wifi te` | αž’αžαŸ‹αž˜αžΆαž“ wifi αž‘αŸ |
52
 
53
+ ## Quick start
54
 
55
+ With transformers alone (`pip install transformers torch`):
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
 
57
  ```python
58
  from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
 
81
  print(convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
82
  ```
83
 
84
+ This is beam search alone (CER 0.062 on Samdei Type Bench). For the best quality, rerank with the two LMs and the channel model of this repo (0.052), using the [`samdei-type`](https://github.com/Samdei-Labs/samdei-type) package:
 
 
 
 
85
 
86
+ ```bash
87
+ pip install "samdei-type @ git+https://github.com/Samdei-Labs/samdei-type.git"
88
+ ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
89
 
90
+ ```python
91
+ from samdei_type.pipeline import Converter
92
 
93
+ conv = Converter("samdei/samdei-type-base") # PyTorch weights + full rerank, as in the paper
94
+ print(conv.convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
95
+ ```
 
 
 
 
 
96
 
97
+ Or on the command line: `samdei-type -m samdei/samdei-type-base "knhom jong tov phsar"`.
 
 
 
98
 
99
+ ## Files
 
 
100
 
101
+ ```
102
+ config.json, generation_config.json, model.safetensors (71.1 MiB), tokenizer*.json fp32 model (transformers)
103
+ onnx/encoder_kv.int8.onnx (11.3 MiB), onnx/decoder_step.int8.onnx (6.3 MiB), onnx/vocab.json int8 ONNX
104
+ channel/ (28.4 MiB) reverse model (Khmer -> romanized) for reranking, same format
105
+ lm/char5_nowiki.npz (8.9 MiB) character 5-gram LM
106
+ lm/wordfull_nowiki.json.gz (12.2 MiB) word bigram LM
107
+ rerank.json decoding and rerank settings
108
+ ```
109
 
110
+ Reranking scores each beam hypothesis y for input x as `log P(y|x) + 0.3 log P_char(y) + 0.3 log P_word(y) βˆ’ 1.0 #OOV(y) + 0.3 log P_ch(x|y)`. The channel term checks that y explains all of the input, so the LMs cannot swap a rare word for a frequent one. Nothing in this repo is a pickle.
111
 
112
+ ## Results
 
 
113
 
114
+ [Samdei Type Bench](https://huggingface.co/datasets/samdei/samdei-type-bench): 2,155 everyday messages and words with human-checked Khmer. The metrics are CER after normalization and exact match.
 
 
 
 
 
 
 
115
 
116
+ | System | CER | Exact |
117
+ |---|---|---|
118
+ | **Samdei Type Base**, reranked | **0.052** | **30%** |
119
+ | Samdei Type Base, beam only (the transformers example above) | 0.062 | 24% |
120
+ | [Samdei Type Mini](https://huggingface.co/samdei/samdei-type-mini), reranked | 0.058 | 24% |
121
+ | Samdei Type Prime (not released) | 0.027 | 61% |
122
+ | Akkhara-mini, 6-word pieces (0.159 on whole inputs) | 0.066 | 23% |
123
+ | AkaraAlpha | 0.281 | 2% |
124
 
125
+ - Prime is Base trained further on scenario-designed, LLM-drafted, human-audited data. Most bench items share that scenario design, so its lead is an upper bound.
126
+ - Without punctuation, Base's CER is 0.046. Decode long messages in pieces, as above. On whole inputs, beam-only CER is 0.133.
127
+ - On one Apple M5 Pro CPU thread with int8 ONNX, beam 5 takes 9.3 ms per input (p50).
128
 
129
+ ## Model and training
 
130
 
131
+ - **Architecture.** Pre-LN Transformer encoder-decoder: 6 encoder and 3 decoder layers, d_model 384, 6 heads, FFN 1536. It has 216 input and 228 output symbols. The channel model is Mini-sized (7.1M) and reads in the reverse direction.
132
+ - **transformers format.** It loads as `BlenderbotForConditionalGeneration` because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 18.63M parameters.
133
+ - **Training.** Three stages on one laptop GPU (Apple M5 Pro), about 6 hours in all:
134
+ 1. pretraining on 12.8M pairs: real Khmer sentences spelled by a rule-based typing generator ([`samdei-type-base-pretrained`](https://huggingface.co/samdei/samdei-type-base-pretrained));
135
+ 2. fine-tuning on 1.16M chat-like pairs;
136
+ 3. adaptation to casual register (1.59M pairs, with replay).
 
 
137
 
138
+ The rerank LMs are built from HPLT v2, FineWeb-2, ALT and our LLM-written chat sentences, without Wikipedia.
139
+ - **Data.** The exact training files are in [`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data) and [`samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data), with `recipes.json`. No evaluation data and no other romanization system's data was trained on. Akkhara-Preview, a public 110-row preview, was used only to pick stage 1–2 checkpoints; stage 3 used a closed development set.
 
 
 
 
 
 
140
 
141
  ## Limitations
142
 
143
+ - Every romanized input seen in training is synthetic (rules, mined spelling tables, an LLM). Real typing appears only in the evaluation.
144
+ - Homophones that need more context than a word bigram: ដើរ / αžŠαŸ‚αžš (`der`), αžŠαžΉαž€ / αžŠαŸ‚αž€ / αžŠαŸαž€ (`dek`).
145
+ - Loanwords typed in English (`check` for αž†αŸ‚αž€) stay English. A Latin comma is copied, and a missing final αŸ” is never added.
146
+ - Rare dictionary words without context are hard, many of them Pali-Sanskrit.
147
+ - The Khmer side is mostly web text, with its topics and biases.
148
+ - Not for Khmer β†’ Latin, official romanization of names, or output nobody checks.
 
 
 
 
 
149
 
150
+ ## License
151
 
152
+ Apache-2.0 (`LICENSE`; keep `NOTICE` when redistributing).
153
+ - **LMs:** built without Wikipedia, from HPLT v2, FineWeb-2 (ODC-By), ALT (CC BY) and our LLM-written chat sentences.
154
+ - **Training data:** CC BY-SA 4.0, with its sources in the dataset cards.
 
 
 
 
 
 
 
 
 
155
  - **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
156
 
157
  ## Citation
 
159
  ```bibtex
160
  @misc{samdei-type-2026,
161
  title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
162
+ author = {Mengsay Loem},
163
  year = {2026},
164
  url = {<paper-url>}
165
  }
166
  ```
 
 
rerank.json CHANGED
@@ -2,7 +2,7 @@
2
  "model": "Samdei Type Base",
3
  "beam": 5,
4
  "chunk_max_words": 8,
5
- "chunk_rule": "rkhmer.evaluation.chunk_source: at most 8 words per piece, break after any punctuation mark; pieces are decoded and reranked on their own and joined with a space",
6
  "rerank_weights": {"char": 0.3, "word": 0.3, "oov": 1.0, "channel": 0.3},
7
  "score": "log P(y|x) + char*log P_char(y) + word*log P_word(y) - oov*#OOV(y) + channel*log P_ch(x|y)",
8
  "char_lm": "lm/char5_nowiki.npz",
 
2
  "model": "Samdei Type Base",
3
  "beam": 5,
4
  "chunk_max_words": 8,
5
+ "chunk_rule": "samdei_type.evaluation.chunk_source: at most 8 words per piece, break after any punctuation mark; pieces are decoded and reranked on their own and joined with a space",
6
  "rerank_weights": {"char": 0.3, "word": 0.3, "oov": 1.0, "channel": 0.3},
7
  "score": "log P(y|x) + char*log P_char(y) + word*log P_word(y) - oov*#OOV(y) + channel*log P_ch(x|y)",
8
  "char_lm": "lm/char5_nowiki.npz",