mengsay commited on
Commit
c824a1a
Β·
verified Β·
1 Parent(s): 3c22801

Shorter cards with a transformers quick start; translation task tag; author; samdei-type package

Browse files
Files changed (2) hide show
  1. README.md +54 -173
  2. rerank.json +1 -1
README.md CHANGED
@@ -3,7 +3,7 @@ license: apache-2.0
3
  language:
4
  - km
5
  library_name: transformers
6
- pipeline_tag: text2text-generation
7
  tags:
8
  - transliteration
9
  - romanization
@@ -27,7 +27,7 @@ model-index:
27
  - name: Samdei Type Mini
28
  results:
29
  - task:
30
- type: text2text-generation
31
  name: Romanized Khmer to Khmer script
32
  dataset:
33
  type: samdei/samdei-type-bench
@@ -44,97 +44,16 @@ model-index:
44
 
45
  # Samdei Type Mini
46
 
47
- Samdei Type Mini converts romanized Khmer, as people type it in chat, into Khmer script:
48
 
49
  | Input | Output |
50
  |---|---|
51
  | `knhom jong tov phsar` | αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš |
52
  | `ot mean wifi te` | αž’αžαŸ‹αž˜αžΆαž“ wifi αž‘αŸ |
53
 
54
- It is a 7.1M-parameter character-level Transformer encoder-decoder. It ships as int8 ONNX (7.0 MiB), with a 1.2 MiB word bigram LM that reranks its candidates, and as fp32 weights that load with transformers. Mini is meant for on-device use, such as a phone keyboard that offers Khmer-script candidates as the user types.
55
 
56
- Samdei (αžŸαŸ†αžŠαžΈ, "what you say") is our team, and Samdei Type is the converter. The family has three models:
57
-
58
- | Model | Parameters | Released as | Samdei Type Bench CER / exact |
59
- |---|---|---|---|
60
- | **Samdei Type Mini** (this repo) | 7.1M | int8 ONNX + pruned word LM + PyTorch | 0.058 / 24% |
61
- | Samdei Type Base | 18.2M | PyTorch + int8 ONNX + LMs + channel model: [`samdei/samdei-type-base`](https://huggingface.co/samdei/samdei-type-base) | 0.052 / 30% |
62
- | Samdei Type Prime | 18.2M | not released | 0.027 / 61% |
63
-
64
- Samdei Type Prime is Base trained further on scenario-designed, LLM-drafted, human-audited data. It is not released.
65
-
66
- The pretrained (stage-1) checkpoints, for fine-tuning your own converter, are [`samdei/samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained) (this size) and [`samdei/samdei-type-base-pretrained`](https://huggingface.co/samdei/samdei-type-base-pretrained).
67
-
68
- Paper: *Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration* (`<paper-url>`). Code: `<code-repo-url>`.
69
-
70
- ## Files
71
-
72
- ```
73
- config.json configuration and vocabularies (transformers format)
74
- generation_config.json beam search, k = 5
75
- model.safetensors 28.4 MiB weights (fp32), for transformers, fine-tuning or the PyTorch reranker
76
- tokenizer.json character tokenizer
77
- tokenizer_config.json its settings
78
- onnx/encoder_kv.int8.onnx 5.0 MiB input characters -> cross-attention K/V of every decoder layer
79
- onnx/decoder_step.int8.onnx 2.0 MiB one decoder step with a self-attention KV cache
80
- onnx/vocab.json vocabularies and graph shapes (layers, heads, head_dim, bos, eos, max_src_len)
81
- lm/wordfull_nowiki_p25.json.gz 1.2 MiB word bigram LM over khmercut words, pruned to bigram count >= 25 (gzipped JSON)
82
- rerank.json decoding and rerank settings used for every number below
83
- ```
84
-
85
- ## Intended use
86
-
87
- - **Inside a keyboard:** turn what a user types in Latin letters into a list of Khmer-script candidates on every keystroke, offline.
88
- - **Reading:** show someone else's romanized message in Khmer script.
89
- - **Mixed text:** English words, numbers and brand names are copied through unchanged.
90
-
91
- **Out of scope:**
92
- - the reverse direction (Khmer β†’ Latin);
93
- - official romanization standards, such as names in documents;
94
- - any setting where the user cannot see and correct the output. The model makes mistakes on homophones and rare words (see Limitations).
95
-
96
- ## How to use
97
-
98
- Install the `rkhmer` package from the code repository:
99
-
100
- ```bash
101
- pip install "rkhmer[deploy] @ git+<code-repo-url>"
102
- ```
103
-
104
- ```python
105
- from huggingface_hub import snapshot_download
106
- from rkhmer.evaluation import chunk_source
107
- from rkhmer.onnx_infer import OnnxConverter
108
- from rkhmer.rerank import Reranker, load_weights
109
-
110
- d = snapshot_download("samdei/samdei-type-mini", allow_patterns=["onnx/*", "lm/*", "rerank.json"])
111
- conv = OnnxConverter(f"{d}/onnx") # int8 graphs + vocab.json, numpy + onnxruntime only
112
- rr = Reranker(load_weights(f"{d}/rerank.json"), wordlm=f"{d}/lm/wordfull_nowiki_p25.json.gz")
113
-
114
- def convert(text: str) -> str:
115
- out = []
116
- for piece in chunk_source(text, 8): # at most 8 words per piece, break after punctuation
117
- hyps = conv.beam(piece, k=5) # [(khmer, logprob), ...], best first
118
- out.append(rr.rerank(piece, hyps)[0][0])
119
- return " ".join(out)
120
-
121
- print(convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
122
- ```
123
-
124
- **Decoding, as in the paper.**
125
- - Beam search with k = 5.
126
- - The input is cut into pieces of at most eight words, with a break after any punctuation mark. Each piece is decoded and reranked on its own, and the outputs are joined with a space.
127
- - Each hypothesis y for input x is scored as `log P(y|x) + 0.3 Β· log P_word(y) βˆ’ 1.0 Β· #OOV(y)`. The weights were tuned on a closed development set, and `rerank.json` holds them.
128
- - Inputs are truncated to 249 characters.
129
-
130
- **On a phone.**
131
- - The two graphs are all a runtime needs.
132
- - `encoder_kv` runs once per input.
133
- - `decoder_step` maps the newest token, its position, the past self-attention K/V and the cross-attention K/V to the logits and the new K/V. It has a dynamic batch axis, so all beams run in one call.
134
- - `rkhmer/onnx_infer.py` is a reference implementation in plain numpy (no torch) to port from.
135
- - The cached decoder reproduces full re-decoding exactly. Int8 quantization does not change CER on Dev-Sentences with beam 5 and no LMs (0.010 in fp32 and in int8).
136
-
137
- **transformers, without rkhmer.** The fp32 weights at the repo root run with `transformers` and `torch` alone (`pip install transformers torch`; see Model). This is beam search with k = 5 and no reranking: 0.072 CER / 17% exact on Samdei Type Bench, against 0.058 / 24% with the word LM above.
138
 
139
  ```python
140
  from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
@@ -163,108 +82,72 @@ def convert(text: str) -> str:
163
  print(convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
164
  ```
165
 
166
- ## Model
167
 
168
- | | Samdei Type Mini |
169
- |---|---|
170
- | Architecture | pre-LN Transformer encoder-decoder, GELU, learned positions up to 256, untied output projection |
171
- | d_model / heads | 256 / 4 |
172
- | Encoder / decoder layers | 6 / 2 (deep encoder, shallow decoder) |
173
- | FFN width | 1024 |
174
- | Parameters | 7,149,540 |
175
- | Vocabulary | 215 input symbols (Latin letters, digits, punctuation, space), 228 output symbols (Khmer code points, space, copied characters), including special tokens |
176
- | int8 ONNX | 7.0 MiB as released (7.6 MiB in the paper, with the exporter's debug metadata) |
177
-
178
- **transformers format.** The model folders load with `AutoModelForSeq2SeqLM` and `AutoTokenizer` and need no custom code. They use transformers' Blenderbot classes only because the architecture is the same (a pre-LN encoder-decoder with scaled embeddings, learned positions from 0 and no embedding LayerNorm). The model has nothing else in common with BlenderBot. Blenderbot has one vocabulary for both sides, so the tokenizer stacks the two: ids 0–227 are the output symbols and the input symbols follow. The output layer gives input ids a fixed logit of βˆ’10⁴, so they are never generated. For the same reason transformers reports 7.43M parameters: the stacked embedding tables and the output layer carry rows that are never used. `rkhmer` reads the same files and maps them back to its own layout.
179
 
180
- ## Training
 
181
 
182
- Mini was trained in three stages. Each stage started from the best checkpoint of the stage before and replayed its data.
 
 
183
 
184
- | | Stage 1: pretraining | Stage 2: fine-tuning | Stage 3: adaptation |
185
- |---|---|---|---|
186
- | New data | rule-based persona generator applied to 1.41M web sentences and the 2.4k LLM-written chat sentences (6.50M pairs), + Gemma 4 romanizations of real sentences (5.7k Γ—3) | chat-heavy generator sample 630k; Gemma 4 romanizations 18.3k Γ—4; pairs spelled with the table mined from the Chhunneng word list 290k; chat-like abbreviated copies 140k + 16k Γ—2 | casual-register pairs 427k (spoken-register web sentences, stretched letters) |
187
- | Replayed | – | – | the stage-2 mix |
188
- | Total pairs | 6.5M | 1.16M | 1.59M |
189
- | Steps (best checkpoint) | 40k (35k) | 6k (4k) | 3k (2.5k) |
190
- | Peak lr, warmup | 1e-3, 2k | 2e-4, 300 | 1e-4, 100 |
191
- | Wall clock | 2.2 h | about 31–36 min | about 10–12 min |
192
 
193
- **Shared settings:**
194
- - AdamW (Ξ² 0.9 / 0.98, weight decay 0.01), cosine decay to 5% of the peak, label smoothing 0.1;
195
- - about 12k tokens per batch, bf16 autocast;
196
- - one laptop GPU (Apple M5 Pro).
197
 
198
- **Checkpoint selection:**
199
- - Stages 1–2: the lowest mean greedy CER on Dev-Sentences and Akkhara-Preview. Akkhara-Preview is the 110-row public preview of Akkhara-Sing-Khmer; it was used for this selection only and never trained on.
200
- - Stage 3: the lowest greedy CER on the first 400 messages of a closed development set.
 
 
 
201
 
202
- **Data.** The exact training files are published:
203
- - **Stage 1:** [`samdei/samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data), configs `generator_v2` and `llm_romanized`. The pretrained checkpoint is [`samdei/samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained).
204
- - **Stages 2–3:** [`samdei/samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data). Mini uses `mined_tables_chh`.
205
 
206
- Each repo's `recipes.json` gives the exact mix.
207
- - **Khmer targets.** Always real text, except 2,483 chat sentences written by Gemma 4. The text comes from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT.
208
- - **Romanized inputs.** These come from:
209
- - the persona generator, which uses Google's Khmer pronunciation lexicon;
210
- - a spelling table mined from the Chhunneng word list;
211
- - Gemma 4 romanizations of real sentences;
212
- - chat-like abbreviations.
213
- - **Not used:** no evaluation set, and no data of other romanization systems (in particular, no Akkhara-Sing-Khmer data).
214
 
215
- ## Evaluation
216
 
217
- **Samdei Type Bench** ([`samdei/samdei-type-bench`](https://huggingface.co/datasets/samdei/samdei-type-bench)) has 2,155 everyday messages and words with human-checked Khmer. 192 of them were typed blind by native speakers. Inputs have a median of 81 characters.
 
 
 
 
 
 
 
218
 
219
- - **Metric.** Character error rate (CER) after normalization (`khnormal`, spaces removed, lowercase, coeng-DA merged with coeng-TA, ឲ merged with ឱ) and exact match, on each item's main input and main target.
220
- - **Same code for every system:** `rkhmer.evaluation.score`.
221
 
222
- | System | Parameters | int8 size | CER | Exact |
223
- |---|---|---|---|---|
224
- | **Samdei Type Mini** | 7.1M | 7.6 MiB | 0.058 | 24% |
225
- | Samdei Type Base | 18.2M | 18.2 MiB | **0.052** | **30%** |
226
- | Samdei Type Prime (not released) | 18.2M | 18.2 MiB | 0.027 | 61% |
227
- | Akkhara-mini, whole input | 17.8M | 17.6 MiB | 0.159 | 27% |
228
- | Akkhara-mini, 6-word pieces | 17.8M | 17.6 MiB | 0.066 | 23% |
229
- | AkaraAlpha, word by word | 0.08M | – | 0.281 | 2% |
230
 
231
- - **Prime is partly in-domain.** Its training data shares the scenario design of most Samdei Type Bench items, so its lead is an upper bound.
232
- - **Reranking.** Beam 5 alone gives 0.072 / 17%; the pruned word LM brings it to 0.058 / 24%.
233
- - **Decode long messages in pieces.** Mini was trained on short pieces (inputs of at most 132 characters, median 26). With beam 5 and no LMs, whole-input CER is 0.218 overall and 0.558 on inputs over 132 characters, against 0.072 and 0.076 in pieces.
234
- - **Reference sets** (CER / exact): Dev-Sentences 0.005 / 95%, Chat-Words 0.081 / 84%, Dict-Words 0.282 / 34%, Casual-Synth 0.055 / 62%.
235
- - **Speed.** One M5 Pro CPU thread, int8 ONNX, without reranking:
236
- - greedy: 2.2 ms p50 / 4.2 ms p95 per input;
237
- - beam 5: 4.9 / 10.0 ms.
238
- - Per keystroke in the live demo, with the word-LM rerank: 5.7 ms p50 / 17.2 ms p95.
239
- - Not yet measured on a phone.
240
 
241
  ## Limitations
242
 
243
- - **Synthetic training inputs.** Every romanized input Mini has seen comes from rules, mined tables or an LLM; real typing appears only in the evaluation.
244
- - **Homophones that need more context than a word bigram gives**, for example:
245
- - ដើរ / αžŠαŸ‚αžš (`der`);
246
- - αžŠαžΉαž€ / αžŠαŸ‚αž€ / αžŠαŸαž€ (`dek`);
247
- - αž’αžΆαž‘αž·αžαŸ’αž™ / ធតីត (`atit`);
248
- - the polite particle αž”αžΆαž‘ against αž”αžΆαžαŸ‹ ("lost") at the end of a message.
249
- - **Spoken-register words that share a spelling with a written word:** `nung` is αž αŸ’αž“αžΉαž„ in chat and αž“αž·αž„ ("and") in prose.
250
- - **Loanwords typed in English** (`check` for αž†αŸ‚αž€) are copied through as English.
251
- - **Punctuation.** Mini copies a Latin comma where Khmer uses a space, and never adds a final αŸ” that the input lacks.
252
- - **Heavy abbreviation and dictionary words.** Mini is weaker on input abbreviated more heavily than its training data and on rare dictionary words without context, many of them Pali-Sanskrit (Dict-Words CER 0.282).
253
- - **Web text.** The Khmer side comes mostly from web text and carries its topics and biases.
254
 
255
- ## License and attribution
256
 
257
- - **Everything in this repo** (ONNX graphs, vocabulary, rerank settings, word LM): Apache-2.0. The full text is in `LICENSE`. `NOTICE` holds the attributions, and redistributions must keep it.
258
- - **`lm/wordfull_nowiki_p25.json.gz`:** word n-gram counts built without Wikipedia, from:
259
- - HPLT v2 (CC0 packaging; Common Crawl terms apply to the text);
260
- - FineWeb-2 (ODC-By 1.0);
261
- - ALT (CC BY 4.0);
262
- - our LLM-written chat sentences (CC BY 4.0; written with Gemma 4, Apache-2.0).
263
- - **Training data:** CC BY-SA 4.0; see the dataset cards. Its sources are:
264
- - HPLT v2, FineWeb-2, Khmer Wikipedia (CC BY-SA) and ALT;
265
- - Google's Khmer pronunciation lexicon (CC BY 4.0);
266
- - the Chhunneng romanized Khmer dataset (MIT, Β© 2025 Chrea Chanchhunneng);
267
- - Gemma 4 output (Apache-2.0).
268
  - **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
269
 
270
  ## Citation
@@ -272,10 +155,8 @@ Each repo's `recipes.json` gives the exact mix.
272
  ```bibtex
273
  @misc{samdei-type-2026,
274
  title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
275
- author = {<authors>},
276
  year = {2026},
277
  url = {<paper-url>}
278
  }
279
  ```
280
-
281
- Contact: `<contact>`
 
3
  language:
4
  - km
5
  library_name: transformers
6
+ pipeline_tag: translation
7
  tags:
8
  - transliteration
9
  - romanization
 
27
  - name: Samdei Type Mini
28
  results:
29
  - task:
30
+ type: translation
31
  name: Romanized Khmer to Khmer script
32
  dataset:
33
  type: samdei/samdei-type-bench
 
44
 
45
  # Samdei Type Mini
46
 
47
+ Samdei Type Mini converts romanized Khmer, as people type it in chat, into Khmer script. It is a 7.1M-parameter character-level encoder-decoder, small enough for a phone keyboard: 7 MiB as int8 ONNX.
48
 
49
  | Input | Output |
50
  |---|---|
51
  | `knhom jong tov phsar` | αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš |
52
  | `ot mean wifi te` | αž’αžαŸ‹αž˜αžΆαž“ wifi αž‘αŸ |
53
 
54
+ ## Quick start
55
 
56
+ With transformers alone (`pip install transformers torch`):
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57
 
58
  ```python
59
  from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
 
82
  print(convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
83
  ```
84
 
85
+ This is beam search alone (CER 0.072 on Samdei Type Bench). For the best quality, rerank with the word LM of this repo (0.058), using the [`samdei-type`](https://github.com/Samdei-Labs/samdei-type) package:
86
 
87
+ ```bash
88
+ pip install "samdei-type[deploy] @ git+https://github.com/Samdei-Labs/samdei-type.git"
89
+ ```
 
 
 
 
 
 
 
 
90
 
91
+ ```python
92
+ from samdei_type.pipeline import Converter
93
 
94
+ conv = Converter("samdei/samdei-type-mini", backend="onnx") # int8 ONNX + word-LM rerank, as in the paper
95
+ print(conv.convert("knhom jong tov phsar")) # αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš
96
+ ```
97
 
98
+ Or on the command line: `samdei-type "knhom jong tov phsar"`.
 
 
 
 
 
 
 
99
 
100
+ ## Files
 
 
 
101
 
102
+ ```
103
+ config.json, generation_config.json, model.safetensors (28.4 MiB), tokenizer*.json fp32 model (transformers)
104
+ onnx/encoder_kv.int8.onnx (5.0 MiB), onnx/decoder_step.int8.onnx (2.0 MiB), onnx/vocab.json int8 ONNX
105
+ lm/wordfull_nowiki_p25.json.gz (1.2 MiB) word bigram LM for reranking
106
+ rerank.json decoding and rerank settings
107
+ ```
108
 
109
+ **On a phone,** the two ONNX graphs are all a runtime needs. `encoder_kv` runs once per input. `decoder_step` decodes one token with a KV cache, and its batch axis holds all beams. [`samdei_type/onnx_infer.py`](https://github.com/Samdei-Labs/samdei-type) is a plain-numpy reference to port from. On one Apple M5 Pro CPU thread, beam 5 takes 4.9 ms per input (p50); with the rerank, a keystroke takes 5.7 ms. It has not been timed on a phone yet.
 
 
110
 
111
+ ## Results
 
 
 
 
 
 
 
112
 
113
+ [Samdei Type Bench](https://huggingface.co/datasets/samdei/samdei-type-bench): 2,155 everyday messages and words with human-checked Khmer. The metrics are CER after normalization and exact match.
114
 
115
+ | System | CER | Exact |
116
+ |---|---|---|
117
+ | **Samdei Type Mini**, reranked | 0.058 | 24% |
118
+ | Samdei Type Mini, beam only (the transformers example above) | 0.072 | 17% |
119
+ | [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base), reranked | 0.052 | 30% |
120
+ | Samdei Type Prime (not released) | 0.027 | 61% |
121
+ | Akkhara-mini, 6-word pieces (0.159 on whole inputs) | 0.066 | 23% |
122
+ | AkaraAlpha | 0.281 | 2% |
123
 
124
+ - Prime is Base trained further on scenario-designed, LLM-drafted, human-audited data. Most bench items share that scenario design, so its lead is an upper bound.
125
+ - Decode long messages in pieces, as above. On whole inputs, beam-only CER is 0.218.
126
 
127
+ ## Model and training
 
 
 
 
 
 
 
128
 
129
+ - **Architecture.** Pre-LN Transformer encoder-decoder: 6 encoder and 2 decoder layers, d_model 256, 4 heads, FFN 1024. It has 215 input symbols (Latin letters, digits, punctuation) and 228 output symbols (Khmer characters, and copied characters).
130
+ - **transformers format.** It loads as `BlenderbotForConditionalGeneration` because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 7.43M parameters.
131
+ - **Training.** Three stages on one laptop GPU (Apple M5 Pro), about 3 hours in all:
132
+ 1. pretraining on 6.5M pairs: real Khmer sentences spelled by a rule-based typing generator ([`samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained));
133
+ 2. fine-tuning on 1.16M chat-like pairs;
134
+ 3. adaptation to casual register (1.59M pairs, with replay).
135
+ - **Data.** The exact training files are in [`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data) and [`samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data), with `recipes.json`. No evaluation data and no other romanization system's data was trained on. Akkhara-Preview, a public 110-row preview, was used only to pick stage 1–2 checkpoints; stage 3 used a closed development set.
 
 
136
 
137
  ## Limitations
138
 
139
+ - Every romanized input seen in training is synthetic (rules, mined spelling tables, an LLM). Real typing appears only in the evaluation.
140
+ - Homophones that need more context than a word bigram: ដើរ / αžŠαŸ‚αžš (`der`), αžŠαžΉαž€ / αžŠαŸ‚αž€ / αžŠαŸαž€ (`dek`).
141
+ - Loanwords typed in English (`check` for αž†αŸ‚αž€) stay English. A Latin comma is copied, and a missing final αŸ” is never added.
142
+ - Rare dictionary words without context are hard, as is heavier abbreviation than in training.
143
+ - The Khmer side is mostly web text, with its topics and biases.
144
+ - Not for Khmer β†’ Latin, official romanization of names, or output nobody checks.
 
 
 
 
 
145
 
146
+ ## License
147
 
148
+ Apache-2.0 (`LICENSE`; keep `NOTICE` when redistributing).
149
+ - **Word LM:** built without Wikipedia, from HPLT v2, FineWeb-2 (ODC-By), ALT (CC BY) and our LLM-written chat sentences.
150
+ - **Training data:** CC BY-SA 4.0, with its sources in the dataset cards.
 
 
 
 
 
 
 
 
151
  - **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
152
 
153
  ## Citation
 
155
  ```bibtex
156
  @misc{samdei-type-2026,
157
  title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
158
+ author = {Mengsay Loem},
159
  year = {2026},
160
  url = {<paper-url>}
161
  }
162
  ```
 
 
rerank.json CHANGED
@@ -2,7 +2,7 @@
2
  "model": "Samdei Type Mini",
3
  "beam": 5,
4
  "chunk_max_words": 8,
5
- "chunk_rule": "rkhmer.evaluation.chunk_source: at most 8 words per piece, break after any punctuation mark; pieces are decoded and reranked on their own and joined with a space",
6
  "rerank_weights": {"char": 0.0, "word": 0.3, "oov": 1.0, "channel": 0.0},
7
  "score": "log P(y|x) + char*log P_char(y) + word*log P_word(y) - oov*#OOV(y) + channel*log P_ch(x|y)",
8
  "char_lm": null,
 
2
  "model": "Samdei Type Mini",
3
  "beam": 5,
4
  "chunk_max_words": 8,
5
+ "chunk_rule": "samdei_type.evaluation.chunk_source: at most 8 words per piece, break after any punctuation mark; pieces are decoded and reranked on their own and joined with a space",
6
  "rerank_weights": {"char": 0.0, "word": 0.3, "oov": 1.0, "channel": 0.0},
7
  "score": "log P(y|x) + char*log P_char(y) + word*log P_word(y) - oov*#OOV(y) + channel*log P_ch(x|y)",
8
  "char_lm": null,