mengsay commited on
Commit
de04dfc
Β·
verified Β·
1 Parent(s): 37f0549

Shorter cards with a transformers quick start; translation task tag; author; samdei-type package

Browse files
Files changed (1) hide show
  1. README.md +59 -117
README.md CHANGED
@@ -3,7 +3,7 @@ license: apache-2.0
3
  language:
4
  - km
5
  library_name: transformers
6
- pipeline_tag: text2text-generation
7
  tags:
8
  - transliteration
9
  - romanization
@@ -21,156 +21,98 @@ metrics:
21
 
22
  # Samdei Type Base (pretrained)
23
 
24
- This is the **pretrained (stage-1) checkpoint** of Samdei Type Base, a romanized-Khmer β†’ Khmer-script converter with 18.2M parameters. It is the starting point that [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base) was fine-tuned from. The Mini-size checkpoint is [`samdei/samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained).
25
 
26
- Pretraining teaches a model Khmer words and spelling, and to copy English words and numbers through, from millions of real Khmer sentences spelled by a rule-based typing generator. It does **not** yet teach how people actually type in chat. That is the job of the two short stages that follow (fine-tuning and adaptation, a few tens of minutes each on a laptop).
27
 
28
- **Use this checkpoint to fine-tune your own converter.** Examples:
29
- - a different romanization habit or scheme;
30
- - a domain;
31
- - data you collected.
32
 
33
- **To convert text, use [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base)**, which is better on chat input.
34
-
35
- Samdei (αžŸαŸ†αžŠαžΈ, "what you say") is our team, and Samdei Type is the converter. Paper: `<paper-url>`. Code: `<code-repo-url>`.
36
-
37
- ## Files
38
-
39
- ```
40
- config.json configuration and vocabularies (transformers format)
41
- generation_config.json beam search, k = 5
42
- model.safetensors 71.1 MiB weights (fp32)
43
- tokenizer.json character tokenizer
44
- tokenizer_config.json its settings
45
- ```
46
-
47
- ## How to use
48
-
49
- Install the `rkhmer` package from the code repository:
50
-
51
- ```bash
52
- pip install "rkhmer @ git+<code-repo-url>"
53
- ```
54
-
55
- **Fine-tune** with the training script of the code repository, as the released models were. For example, stage 2 of Base, with the data of [`samdei/samdei-type-finetuning-data`](https://huggingface.co/datasets/samdei/samdei-type-finetuning-data) exported to TSV, or with your own `latin<TAB>khmer` file:
56
-
57
- ```bash
58
- hf download samdei/samdei-type-base-pretrained --local-dir samdei-base-pretrained
59
- python scripts/train.py --init samdei-base-pretrained --train my_pairs.tsv \
60
- --dev_sets my_dev.tsv --select my_dev.tsv \
61
- --steps 6000 --warmup 300 --lr 2e-4 --eval_every 1000 --out runs/my_finetune
62
- ```
63
-
64
- The model configuration comes from the checkpoint. `best.pt` is picked on the greedy CER of `--select`. The settings above are those of stage 2: AdamW, cosine decay to 5% of the peak, label smoothing 0.1, about 12k tokens per batch, bf16 autocast. Replaying some pretraining-style data alongside new data keeps the model's general Khmer.
65
-
66
- `train.py` writes PyTorch checkpoints (`best.pt`, a pickle). To share a fine-tuned model safely, convert it to the format of this repository:
67
-
68
- ```python
69
- from rkhmer.model import load, save_pretrained
70
- save_pretrained("my_model", *load("runs/my_finetune/best.pt")[:3]) # my_model/: the five files of this repository
71
- ```
72
-
73
- **Convert** text directly (for comparison):
74
-
75
- ```python
76
- from rkhmer.evaluation import chunk_source, translate
77
- from rkhmer.model import load_pretrained
78
-
79
- model, sv, tv, _ = load_pretrained("samdei/samdei-type-base-pretrained")
80
- pieces = chunk_source("knhom jong tov phsar", 8)
81
- best, _ = translate(model, sv, tv, pieces, "cpu", beam=5)
82
- print(" ".join(best))
83
- ```
84
-
85
- With transformers instead (see Model):
86
 
87
  ```python
88
  from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
 
89
  tok = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained")
90
  model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained")
91
  batch = tok(["knhom jong tov phsar"], return_tensors="pt")
92
- print(tok.batch_decode(model.generate(**batch), skip_special_tokens=True)) # beam search, k = 5
93
  ```
94
 
95
- The [Samdei Type Base card](https://huggingface.co/samdei/samdei-type-base) shows how to cut long inputs into pieces first.
96
-
97
- **No pickles.** The weights are safetensors and the configuration is JSON, so loading runs no code from this repository. `load_pretrained` downloads only `config.json` and `model.safetensors`.
98
 
99
- ## Model
100
 
101
- | | Samdei Type Base (pretrained) |
102
- |---|---|
103
- | Architecture | pre-LN Transformer encoder-decoder, GELU, learned positions up to 256, untied output projection |
104
- | d_model / heads | 384 / 6 |
105
- | Encoder / decoder layers | 6 / 3 |
106
- | FFN width | 1536 |
107
- | Parameters | 18,202,980 |
108
- | Vocabulary (incl. special tokens) | 216 input, 228 output |
109
 
110
- **transformers format.** The model folder loads with `AutoModelForSeq2SeqLM` and `AutoTokenizer` and needs no custom code. It uses transformers' Blenderbot classes only because the architecture is the same (a pre-LN encoder-decoder with scaled embeddings, learned positions from 0 and no embedding LayerNorm). The model has nothing else in common with BlenderBot. Blenderbot has one vocabulary for both sides, so the tokenizer stacks the two: ids 0–227 are the output symbols and the input symbols follow. The output layer gives input ids a fixed logit of βˆ’10⁴, so they are never generated. For the same reason transformers reports 18.63M parameters: the stacked embedding tables and the output layer carry rows that are never used. `rkhmer` reads the same files and maps them back to its own layout.
111
-
112
- ## Training
113
-
114
- | | Stage 1: pretraining |
115
- |---|---|
116
- | Data | the first generator version on 1.41M web sentences, two seeds (6.29M pairs each), + 2.4k LLM-written chat sentences spelled the same way, Γ—40 (189k pairs) |
117
- | Total pairs | 12.8M |
118
- | Steps (best checkpoint) | 60k (60k) |
119
- | Peak lr, warmup | 7e-4, 3k |
120
- | Wall clock | 4.9 h |
121
 
122
- - **Generator versions.** Base was pretrained with the first generator version. The second version adds English chat words, place names and punctuation mirroring; Base's later stages use it.
123
- - **Shared settings:**
124
- - AdamW (Ξ² 0.9 / 0.98, weight decay 0.01), cosine decay to 5% of the peak, label smoothing 0.1;
125
- - about 12k tokens per batch, bf16 autocast;
126
- - one laptop GPU (Apple M5 Pro).
127
- - **Checkpoint selection.** The lowest mean greedy CER on Dev-Sentences and Akkhara-Preview. Akkhara-Preview is the 110-row public preview of Akkhara-Sing-Khmer; it was used for this selection only and never trained on.
128
- - **Data.** The exact files are in [`samdei/samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data), configs `generator_v1` and `llm_chat_sentences`. Its `recipes.json` holds the exact mix.
129
- - **Khmer targets:** HPLT v2, FineWeb-2, Khmer Wikipedia and ALT, plus 2,483 chat sentences written by Gemma 4.
130
- - **Romanized inputs:** the persona generator, with Google's Khmer pronunciation lexicon, and Gemma 4 romanizations.
131
- - **Not used:** no evaluation data and no other romanization system's data.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
132
 
133
- ## Evaluation
134
 
135
- **Samdei Type Bench** ([`samdei/samdei-type-bench`](https://huggingface.co/datasets/samdei/samdei-type-bench)) has 2,155 everyday messages and words with human-checked Khmer.
136
 
137
- - **Setting:** beam 5, no LMs, inputs decoded in pieces of at most eight words.
138
- - **Metric:** CER after normalization, and exact match.
139
 
140
  | Checkpoint | CER | Exact |
141
  |---|---|---|
142
- | this checkpoint (pretrained) | 0.097 | 15% |
143
- | Samdei Type Base (after stages 2–3) | 0.062 | 24% |
 
 
 
144
 
145
- Fine-tuning brings the largest single gain after pretraining. Part of the gain is convention: this checkpoint keeps many place names in Latin script, as its pretraining data copies them, while Samdei Type Bench writes them in Khmer script. Reranking with LMs, as Samdei Type Base does, lowers the error further (0.052).
 
 
 
 
146
 
147
  ## Limitations
148
 
149
- - **Romanizations follow the generator's personas,** not real typing. This checkpoint handles careful spelling far better than heavy chat shortcuts.
150
- - **Short inputs only in training:** at most 132 characters, median 26. Decode longer input in pieces.
151
- - **Web text.** The Khmer side comes mostly from web text and carries its topics and biases.
152
 
153
- ## License and attribution
154
 
155
- - **Weights:** Apache-2.0. The full text is in `LICENSE`. `NOTICE` holds the attributions, and redistributions must keep it.
156
- - **Training data** (CC BY-SA 4.0, [`samdei/samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data)). Its sources are:
157
- - HPLT v2 (CC0 packaging; Common Crawl terms apply to the text);
158
- - FineWeb-2 (ODC-By 1.0);
159
- - Khmer Wikipedia (CC BY-SA 4.0);
160
- - ALT (CC BY 4.0);
161
- - Google's Khmer pronunciation lexicon (CC BY 4.0);
162
- - Gemma 4 output (Apache-2.0).
163
- - **Tools:** `khmercut` (Apache-2.0) and SIL `khnormal` (MIT).
164
 
165
  ## Citation
166
 
167
  ```bibtex
168
  @misc{samdei-type-2026,
169
  title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
170
- author = {<authors>},
171
  year = {2026},
172
  url = {<paper-url>}
173
  }
174
  ```
175
-
176
- Contact: `<contact>`
 
3
  language:
4
  - km
5
  library_name: transformers
6
+ pipeline_tag: translation
7
  tags:
8
  - transliteration
9
  - romanization
 
21
 
22
  # Samdei Type Base (pretrained)
23
 
24
+ This is the pretrained (stage-1) checkpoint of [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base), a 18.2M-parameter character-level encoder-decoder that converts romanized Khmer into Khmer script. Pretraining taught it Khmer words and spelling from millions of real Khmer sentences spelled by a rule-based typing generator. It has not yet learned how people actually type in chat.
25
 
26
+ **Use it to fine-tune your own converter:** for another typing habit, a domain, or data you collected. To convert chat as it is, use [Samdei Type Base](https://huggingface.co/samdei/samdei-type-base). The Mini-size checkpoint is [`samdei-type-mini-pretrained`](https://huggingface.co/samdei/samdei-type-mini-pretrained).
27
 
28
+ ## Quick start
 
 
 
29
 
30
+ With transformers alone (`pip install transformers torch`):
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
31
 
32
  ```python
33
  from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
34
+
35
  tok = AutoTokenizer.from_pretrained("samdei/samdei-type-base-pretrained")
36
  model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base-pretrained")
37
  batch = tok(["knhom jong tov phsar"], return_tensors="pt")
38
+ print(tok.batch_decode(model.generate(**batch), skip_special_tokens=True)) # ['αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš']
39
  ```
40
 
41
+ Inputs should be short pieces. The [Samdei Type Base card](https://huggingface.co/samdei/samdei-type-base) shows how to cut a long message into pieces of at most eight words.
 
 
42
 
43
+ ## Fine-tune
44
 
45
+ The folder is a standard transformers model, so any seq2seq training loop works. Two details matter:
46
+ - **Labels.** Output symbols are stored in the tokenizer as `"\ue000"` + character.
47
+ - **`dropout=0.0`.** Blenderbot's `dropout` also drops the embeddings, which this model was never trained with. With it on, held-out loss rises during fine-tuning; with it off, the loss falls. Attention and FFN dropout (0.1) stay on.
 
 
 
 
 
48
 
49
+ ```python
50
+ import random
51
+ import torch
52
+ from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
 
 
 
 
 
 
 
53
 
54
+ repo = "samdei/samdei-type-base-pretrained"
55
+ tok = AutoTokenizer.from_pretrained(repo)
56
+ model = AutoModelForSeq2SeqLM.from_pretrained(repo, dropout=0.0)
57
+ vocab = tok.get_vocab()
58
+
59
+ def labels(khmer: list[str]) -> torch.Tensor:
60
+ ids = [[vocab.get("\ue000" + c, vocab["<unk>"]) for c in y] + [tok.eos_token_id] for y in khmer]
61
+ n = max(map(len, ids))
62
+ return torch.tensor([x + [-100] * (n - len(x)) for x in ids]) # -100: padding, ignored by the loss
63
+
64
+ pairs = [("knhom jong tov phsar", "αžαŸ’αž‰αž»αŸ†αž…αž„αŸ‹αž‘αŸ…αž•αŸ’αžŸαžΆαžš")] # your (romanized, Khmer) pairs, short pieces
65
+ opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.01)
66
+ model.train()
67
+ for step in range(1000):
68
+ batch = random.sample(pairs, min(64, len(pairs)))
69
+ x = tok([s for s, _ in batch], padding=True, truncation=True, return_tensors="pt")
70
+ loss = model(**x, labels=labels([y for _, y in batch])).loss
71
+ loss.backward()
72
+ opt.step()
73
+ opt.zero_grad()
74
+
75
+ model.save_pretrained("my-converter")
76
+ tok.save_pretrained("my-converter")
77
+ ```
78
 
79
+ The released model was fine-tuned from this checkpoint with AdamW at 2e-4, 300 warmup steps and cosine decay, label smoothing 0.1 and about 12k tokens per batch, for 6k steps. Mixing in some pretraining-style pairs ([`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data)) keeps the model's general Khmer. The saved folder loads with transformers and with [`samdei-type`](https://github.com/Samdei-Labs/samdei-type). To rerank, add a `rerank.json` and its LM files, as in the released model.
80
 
81
+ ## Results
82
 
83
+ [Samdei Type Bench](https://huggingface.co/datasets/samdei/samdei-type-bench), beam 5, inputs in pieces of at most eight words:
 
84
 
85
  | Checkpoint | CER | Exact |
86
  |---|---|---|
87
+ | this checkpoint | 0.097 | 15% |
88
+ | Samdei Type Base (fine-tuned from it) | 0.062 | 24% |
89
+ | Samdei Type Base, with LM reranking | 0.052 | 30% |
90
+
91
+ Fine-tuning brings the largest single gain. Part of the gain from fine-tuning is convention: this checkpoint keeps many place names in Latin script, as its pretraining data does, while the bench writes them in Khmer script.
92
 
93
+ ## Model and training
94
+
95
+ - **Architecture.** Pre-LN Transformer encoder-decoder: 6 encoder and 3 decoder layers, d_model 384, 6 heads, FFN 1536; 216 input and 228 output symbols. It loads as `BlenderbotForConditionalGeneration` because the architecture is the same; it has nothing else in common with BlenderBot. Both vocabularies share one id space (output symbols first), so transformers counts 18.63M parameters.
96
+ - **Data.** 12.8M pairs: the first version of the typing generator applied twice (two seeds) to 1.41M web sentences, plus 2.4k LLM-written chat sentences spelled the same way (Γ—40). The files are in [`samdei-type-pretraining-data`](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data), configs `generator_v1` and `llm_chat_sentences`, with the exact mix in `recipes.json`. The Khmer side is real text from HPLT v2, FineWeb-2, Khmer Wikipedia and ALT, plus the chat sentences.
97
+ - **Training.** 60k steps (best at 60k), peak lr 7e-4, 4.9 hours, on one laptop GPU (Apple M5 Pro). The checkpoint was picked on Dev-Sentences and Akkhara-Preview, a public 110-row preview that was never trained on. No evaluation data was trained on.
98
 
99
  ## Limitations
100
 
101
+ - Its romanized inputs follow the generator's typing personas, not real chat. It handles careful spelling far better than heavy shortcuts.
102
+ - It was trained on short inputs only (at most 132 characters), so decode longer text in pieces.
103
+ - The Khmer side is mostly web text, with its topics and biases.
104
 
105
+ ## License
106
 
107
+ Apache-2.0 (`LICENSE`; keep `NOTICE` when redistributing). The training data is CC BY-SA 4.0, with its sources in the [dataset card](https://huggingface.co/datasets/samdei/samdei-type-pretraining-data).
 
 
 
 
 
 
 
 
108
 
109
  ## Citation
110
 
111
  ```bibtex
112
  @misc{samdei-type-2026,
113
  title = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
114
+ author = {Mengsay Loem},
115
  year = {2026},
116
  url = {<paper-url>}
117
  }
118
  ```