Translation
Keras
Khmer
bigru-additive-attention-seq2seq
transliteration
khmer
romanization
seq2seq
gru
attention
Eval Results (legacy)
Instructions to use Netra-Lab/AkaraAlpha with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Keras
How to use Netra-Lab/AkaraAlpha with Keras:
# !pip install -U keras tensorflow huggingface_hub # Keras needs TensorFlow installed to read "hf://" paths, so the tensorflow backend is selected here; # "jax" and "torch" also work for computation once TensorFlow is installed. import os os.environ["KERAS_BACKEND"] = "tensorflow" import keras model = keras.saving.load_model("hf://Netra-Lab/AkaraAlpha") - Notebooks
- Google Colab
- Kaggle
Release AkaraAlpha
Browse filesKeras model, tokenizers (pickle and JSON), the Khmer dictionary, config and model card.
- .gitattributes +2 -0
- README.md +182 -0
- assets/architecture.png +3 -0
- bi_gru_attention_khmer_transliterator.keras +3 -0
- config.json +29 -0
- gru_transliteration_assets.pkl +3 -0
- khmer_dictionary.txt +0 -0
- tokenizers.json +123 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
assets/architecture.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
bi_gru_attention_khmer_transliterator.keras filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,182 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- km
|
| 5 |
+
library_name: keras
|
| 6 |
+
pipeline_tag: text2text-generation
|
| 7 |
+
tags:
|
| 8 |
+
- transliteration
|
| 9 |
+
- khmer
|
| 10 |
+
- romanization
|
| 11 |
+
- seq2seq
|
| 12 |
+
- gru
|
| 13 |
+
- attention
|
| 14 |
+
- keras
|
| 15 |
+
datasets:
|
| 16 |
+
- Chhunneng/khmer-text-transliteration
|
| 17 |
+
metrics:
|
| 18 |
+
- cer
|
| 19 |
+
- accuracy
|
| 20 |
+
model-index:
|
| 21 |
+
- name: AkaraAlpha
|
| 22 |
+
results:
|
| 23 |
+
- task:
|
| 24 |
+
type: text2text-generation
|
| 25 |
+
name: Romanized Khmer to Khmer transliteration
|
| 26 |
+
dataset:
|
| 27 |
+
type: Chhunneng/khmer-text-transliteration
|
| 28 |
+
name: Khmer Text Transliteration Dataset (20% held out)
|
| 29 |
+
metrics:
|
| 30 |
+
- type: cer
|
| 31 |
+
value: 15.07
|
| 32 |
+
name: CER (%), model alone
|
| 33 |
+
- type: accuracy
|
| 34 |
+
value: 73.6
|
| 35 |
+
name: Top-1 word accuracy (%), beam 5 with dictionary
|
| 36 |
+
---
|
| 37 |
+
|
| 38 |
+
# AkaraAlpha
|
| 39 |
+
|
| 40 |
+
AkaraAlpha turns Khmer typed in the Latin alphabet back into Khmer script: `brodae` โ แแแแแ. It is a character-level encoder-decoder with **78,705 parameters**. A bidirectional GRU reads the romanized word, and a GRU with additive attention writes it in Khmer, one character at a time. A Khmer dictionary of 38,743 words then checks its suggestions.
|
| 41 |
+
|
| 42 |
+
- **Accuracy.** The model alone has a character error rate of 15.07% on held-out words, lower than every baseline trained on the same data, including a Transformer with three times as many parameters.
|
| 43 |
+
- **Dictionary.** Checking five beam candidates against the dictionary makes the first suggestion right 73.6% of the time, against 44.4% for the model's single best guess.
|
| 44 |
+
- **Speed.** The whole system takes 94 ms per word with five candidates.
|
| 45 |
+
|
| 46 |
+
The code is on [GitHub](https://github.com/netra-ai-lab/Romanized-Khmer-to-Khmer-Transliteration) and the package is [`netra-transliterate`](https://pypi.org/project/netra-transliterate/) on PyPI, both under the MIT licence.
|
| 47 |
+
|
| 48 |
+
## Architecture
|
| 49 |
+
|
| 50 |
+
<p align="center"><img src="https://huggingface.co/Netra-Lab/AkaraAlpha/resolve/main/assets/architecture.png" width="100%" alt="AkaraAlpha turning brodae into Khmer: a bidirectional GRU encoder reading upwards and a GRU decoder with additive attention reading downwards"></p>
|
| 51 |
+
|
| 52 |
+
*AkaraAlpha turning `brodae` into แแแแแ.* The encoder reads upwards: each letter becomes a 32-dimensional embedding, a bidirectional GRU reads the letters in both directions, and a projection brings each state to 64 dimensions, the keys and values for attention. The decoder reads downwards. At step 4, with แ แ แ written, its GRU output is the query for additive attention, and a softmax layer over 81 ids gives แ with probability 0.997. The chosen character joins the input and the loop repeats until the end marker.
|
| 53 |
+
|
| 54 |
+
| | |
|
| 55 |
+
| --- | --- |
|
| 56 |
+
| Input | Romanized word, up to 25 characters: 26 Latin letters |
|
| 57 |
+
| Output | Khmer word, up to 24 characters: 77 Khmer characters |
|
| 58 |
+
| Character embeddings | 32 dimensions, separate for input and output |
|
| 59 |
+
| Encoder | Bidirectional GRU, 64 units each way, then a projection to 64 |
|
| 60 |
+
| Decoder | GRU, 64 units, starting from zeros |
|
| 61 |
+
| Attention | Additive (Keras `AdditiveAttention`, learned scale); keys and values are the projected encoder states |
|
| 62 |
+
| Output layer | Softmax over 81 ids, from the GRU output and the context |
|
| 63 |
+
| Parameters | 78,705 (encoder 46,784, decoder 31,921) |
|
| 64 |
+
| Dictionary | 38,743 Khmer words |
|
| 65 |
+
|
| 66 |
+
The decoder doesn't start from the encoder's final state, so everything it knows about the input reaches it through attention.
|
| 67 |
+
|
| 68 |
+
## Results
|
| 69 |
+
|
| 70 |
+
Character error rate of the models alone on the 5,714 held-out words. Lower is better.
|
| 71 |
+
|
| 72 |
+
| Model | Parameters | CER (%) |
|
| 73 |
+
| --- | ---: | ---: |
|
| 74 |
+
| **AkaraAlpha** | 78,705 | **15.07** |
|
| 75 |
+
| Transformer | 248,337 | 17.78 |
|
| 76 |
+
| Attention BiLSTM | 96,753 | 18.26 |
|
| 77 |
+
| LSTM | 58,417 | 31.78 |
|
| 78 |
+
| GRU | 46,385 | 51.64 |
|
| 79 |
+
| RNN | 42,609 | 102.41 |
|
| 80 |
+
|
| 81 |
+
Word-level accuracy and latency of the full system, with the dictionary check. Top-1 is the share of words where the first suggestion is exactly right; Top-K the share where the right word is anywhere in the list.
|
| 82 |
+
|
| 83 |
+
| Decoding | Top-1 | Top-K | Latency per word |
|
| 84 |
+
| --- | ---: | ---: | ---: |
|
| 85 |
+
| Greedy | 44.44% | 44.44% | 21.92 ms |
|
| 86 |
+
| Beam, k = 3 | 70.06% | 78.72% | 66.03 ms |
|
| 87 |
+
| **Beam, k = 5 (recommended)** | **73.60%** | **83.41%** | **94.15 ms** |
|
| 88 |
+
| Beam, k = 7 | 74.87% | 85.60% | 122.10 ms |
|
| 89 |
+
| Beam, k = 10 | 75.85% | 87.37% | 167.21 ms |
|
| 90 |
+
|
| 91 |
+
## Usage
|
| 92 |
+
|
| 93 |
+
### With the package
|
| 94 |
+
|
| 95 |
+
The package includes beam search and the dictionary check:
|
| 96 |
+
|
| 97 |
+
```bash
|
| 98 |
+
pip install netra-transliterate
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
```python
|
| 102 |
+
from khmer_transliterator import transliterate, transliterate_with_dict
|
| 103 |
+
|
| 104 |
+
transliterate("brodae") # 'แแแแแ'
|
| 105 |
+
transliterate_with_dict("brodae", n=5) # dictionary-checked suggestions
|
| 106 |
+
```
|
| 107 |
+
|
| 108 |
+
`netra-transliterate --serve` opens a typing interface in the browser that suggests Khmer words as you type romanized ones.
|
| 109 |
+
|
| 110 |
+
### From this repository
|
| 111 |
+
|
| 112 |
+
The Keras model and a plain JSON copy of its tokenizers are enough for greedy decoding (TensorFlow 2.17 or later):
|
| 113 |
+
|
| 114 |
+
```python
|
| 115 |
+
import json, re
|
| 116 |
+
import numpy as np
|
| 117 |
+
import keras
|
| 118 |
+
from huggingface_hub import snapshot_download
|
| 119 |
+
|
| 120 |
+
path = snapshot_download("Netra-Lab/AkaraAlpha")
|
| 121 |
+
model = keras.saving.load_model(f"{path}/bi_gru_attention_khmer_transliterator.keras")
|
| 122 |
+
tok = json.load(open(f"{path}/tokenizers.json", encoding="utf-8"))
|
| 123 |
+
latin, khmer = tok["latin"], tok["khmer"]
|
| 124 |
+
id_to_khmer = {i: c for c, i in khmer["word_index"].items()}
|
| 125 |
+
start, end = khmer["word_index"]["\t"], khmer["word_index"]["\n"]
|
| 126 |
+
|
| 127 |
+
def transliterate(word: str) -> str:
|
| 128 |
+
letters = re.sub("[^a-z]", "", word.lower())[: latin["max_len"]]
|
| 129 |
+
src = [latin["word_index"].get(c, latin["word_index"]["<unk>"]) for c in letters]
|
| 130 |
+
src = np.array([src + [0] * (latin["max_len"] - len(src))])
|
| 131 |
+
out = [start]
|
| 132 |
+
for _ in range(khmer["max_len"] + 1):
|
| 133 |
+
probs = model.predict([src, np.array([out])], verbose=0)[0, -1]
|
| 134 |
+
nxt = int(probs.argmax())
|
| 135 |
+
if nxt == end:
|
| 136 |
+
break
|
| 137 |
+
out.append(nxt)
|
| 138 |
+
return "".join(id_to_khmer[i] for i in out[1:])
|
| 139 |
+
|
| 140 |
+
print(transliterate("brodae")) # แแแแแ
|
| 141 |
+
```
|
| 142 |
+
|
| 143 |
+
## Files
|
| 144 |
+
|
| 145 |
+
| File | What it is |
|
| 146 |
+
| --- | --- |
|
| 147 |
+
| `bi_gru_attention_khmer_transliterator.keras` | The trained model (Keras 3) |
|
| 148 |
+
| `tokenizers.json` | Character-to-id maps for both scripts, maximum lengths and special tokens |
|
| 149 |
+
| `gru_transliteration_assets.pkl` | The same tokenizers as pickled Keras `Tokenizer` objects, as the package loads them. Only unpickle files you trust |
|
| 150 |
+
| `khmer_dictionary.txt` | The Khmer word list used by the dictionary check, one word per line |
|
| 151 |
+
| `config.json` | Architecture and decoding settings |
|
| 152 |
+
|
| 153 |
+
## Training
|
| 154 |
+
|
| 155 |
+
- **Data.** The [Khmer Text Transliteration Dataset](https://github.com/Chhunneng/khmer-text-transliteration) (Chhunneng, 2023): 28,569 single words in romanized form and in Khmer script. The last 20% (5,714 words) was held out; the model trained on the other 22,855.
|
| 156 |
+
- **Objective.** Teacher forcing with sparse categorical cross-entropy.
|
| 157 |
+
- **Optimiser.** Adam at 0.001, batch size 64, dropout 0.3 on the inputs and recurrent states of both GRUs.
|
| 158 |
+
- **Schedule.** 50 epochs. The learning rate was halved when the validation loss stopped improving for three epochs, which happened once, after epoch 42. The weights are from epoch 48, which had the lowest validation loss (0.1034).
|
| 159 |
+
|
| 160 |
+
## Limitations
|
| 161 |
+
|
| 162 |
+
- **Words, not sentences.** The model transliterates one word at a time, so it can't use the surrounding sentence to choose between two words with the same romanized spelling.
|
| 163 |
+
- **No separate test set.** The held-out words were also used to adjust the learning rate and choose the epoch, so the scores may be slightly optimistic.
|
| 164 |
+
- **One way of romanizing.** All the training data comes from one dataset. People who spell Khmer differently in Latin letters may get worse results.
|
| 165 |
+
- **The dictionary has edges.** A name or new word that isn't in the dictionary may be "corrected" into a real word within two edits.
|
| 166 |
+
- **Latency depends on hardware.** The timings were measured on one machine; read them as the relative cost of each setting.
|
| 167 |
+
|
| 168 |
+
## Citation
|
| 169 |
+
|
| 170 |
+
```bibtex
|
| 171 |
+
@software{akaraalpha2026,
|
| 172 |
+
title = {AkaraAlpha: Romanized Khmer to Khmer Script Transliteration},
|
| 173 |
+
author = {Nhem, Darayut},
|
| 174 |
+
year = {2026},
|
| 175 |
+
publisher = {Netra Lab},
|
| 176 |
+
url = {https://huggingface.co/Netra-Lab/AkaraAlpha}
|
| 177 |
+
}
|
| 178 |
+
```
|
| 179 |
+
|
| 180 |
+
## Licence
|
| 181 |
+
|
| 182 |
+
MIT.
|
assets/architecture.png
ADDED
|
Git LFS Details
|
bi_gru_attention_khmer_transliterator.keras
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5f6b1af75679af1a3d56198a9035c4e70e973c99c354f0db750f5cfa13809680
|
| 3 |
+
size 1014442
|
config.json
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "bigru-additive-attention-seq2seq",
|
| 3 |
+
"framework": "keras",
|
| 4 |
+
"keras_file": "bi_gru_attention_khmer_transliterator.keras",
|
| 5 |
+
"embedding_dim": 32,
|
| 6 |
+
"encoder": {
|
| 7 |
+
"type": "bidirectional_gru",
|
| 8 |
+
"units_per_direction": 64
|
| 9 |
+
},
|
| 10 |
+
"encoder_projection": 64,
|
| 11 |
+
"decoder": {
|
| 12 |
+
"type": "gru",
|
| 13 |
+
"units": 64,
|
| 14 |
+
"initial_state": "zeros"
|
| 15 |
+
},
|
| 16 |
+
"attention": {
|
| 17 |
+
"type": "additive",
|
| 18 |
+
"use_scale": true
|
| 19 |
+
},
|
| 20 |
+
"latin_vocab_size": 28,
|
| 21 |
+
"khmer_vocab_size": 81,
|
| 22 |
+
"max_latin_len": 25,
|
| 23 |
+
"max_khmer_len": 24,
|
| 24 |
+
"parameters": 78705,
|
| 25 |
+
"dictionary_file": "khmer_dictionary.txt",
|
| 26 |
+
"dictionary_words": 38743,
|
| 27 |
+
"recommended_beam_width": 5,
|
| 28 |
+
"max_edit_distance": 2
|
| 29 |
+
}
|
gru_transliteration_assets.pkl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a163c7264d1a68d3d78fc9777f0f82b9bb695796faf33f63b5c16bed2204b6d9
|
| 3 |
+
size 3538
|
khmer_dictionary.txt
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizers.json
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"latin": {
|
| 3 |
+
"word_index": {
|
| 4 |
+
"<unk>": 1,
|
| 5 |
+
"a": 2,
|
| 6 |
+
"o": 3,
|
| 7 |
+
"h": 4,
|
| 8 |
+
"e": 5,
|
| 9 |
+
"n": 6,
|
| 10 |
+
"r": 7,
|
| 11 |
+
"k": 8,
|
| 12 |
+
"t": 9,
|
| 13 |
+
"u": 10,
|
| 14 |
+
"s": 11,
|
| 15 |
+
"m": 12,
|
| 16 |
+
"b": 13,
|
| 17 |
+
"i": 14,
|
| 18 |
+
"c": 15,
|
| 19 |
+
"p": 16,
|
| 20 |
+
"l": 17,
|
| 21 |
+
"g": 18,
|
| 22 |
+
"y": 19,
|
| 23 |
+
"v": 20,
|
| 24 |
+
"d": 21,
|
| 25 |
+
"j": 22,
|
| 26 |
+
"z": 23,
|
| 27 |
+
"f": 24,
|
| 28 |
+
"w": 25,
|
| 29 |
+
"x": 26,
|
| 30 |
+
"q": 27
|
| 31 |
+
},
|
| 32 |
+
"max_len": 25
|
| 33 |
+
},
|
| 34 |
+
"khmer": {
|
| 35 |
+
"word_index": {
|
| 36 |
+
"<unk>": 1,
|
| 37 |
+
"แ": 2,
|
| 38 |
+
"แถ": 3,
|
| 39 |
+
"แ": 4,
|
| 40 |
+
"แ": 5,
|
| 41 |
+
"แ": 6,
|
| 42 |
+
"แ": 7,
|
| 43 |
+
"แ": 8,
|
| 44 |
+
"แ": 9,
|
| 45 |
+
"แ": 10,
|
| 46 |
+
"แ": 11,
|
| 47 |
+
"แท": 12,
|
| 48 |
+
"แ": 13,
|
| 49 |
+
"แข": 14,
|
| 50 |
+
"แ
": 15,
|
| 51 |
+
"แป": 16,
|
| 52 |
+
"แ": 17,
|
| 53 |
+
"แ": 18,
|
| 54 |
+
"แ": 19,
|
| 55 |
+
"แ": 20,
|
| 56 |
+
"แ": 21,
|
| 57 |
+
"แ": 22,
|
| 58 |
+
"แ ": 23,
|
| 59 |
+
"แ": 24,
|
| 60 |
+
"แ": 25,
|
| 61 |
+
"แ": 26,
|
| 62 |
+
"แ": 27,
|
| 63 |
+
"แธ": 28,
|
| 64 |
+
"แ": 29,
|
| 65 |
+
"แ": 30,
|
| 66 |
+
"แผ": 31,
|
| 67 |
+
"แ": 32,
|
| 68 |
+
"แ": 33,
|
| 69 |
+
"แ": 34,
|
| 70 |
+
"แ": 35,
|
| 71 |
+
"แ": 36,
|
| 72 |
+
"แพ": 37,
|
| 73 |
+
"แ": 38,
|
| 74 |
+
"แ": 39,
|
| 75 |
+
"แน": 40,
|
| 76 |
+
"แฝ": 41,
|
| 77 |
+
"แก": 42,
|
| 78 |
+
"แ": 43,
|
| 79 |
+
"แ": 44,
|
| 80 |
+
"แ": 45,
|
| 81 |
+
"แ": 46,
|
| 82 |
+
"แ": 47,
|
| 83 |
+
"แ": 48,
|
| 84 |
+
"แ": 49,
|
| 85 |
+
"แ": 50,
|
| 86 |
+
"แ": 51,
|
| 87 |
+
"แ": 52,
|
| 88 |
+
"แบ": 53,
|
| 89 |
+
"แ": 54,
|
| 90 |
+
"แ": 55,
|
| 91 |
+
"แง": 56,
|
| 92 |
+
"แ
": 57,
|
| 93 |
+
"แ": 58,
|
| 94 |
+
"แ": 59,
|
| 95 |
+
"แ": 60,
|
| 96 |
+
"แฟ": 61,
|
| 97 |
+
"แฅ": 62,
|
| 98 |
+
"แฑ": 63,
|
| 99 |
+
"แซ": 64,
|
| 100 |
+
"แฏ": 65,
|
| 101 |
+
"แ": 66,
|
| 102 |
+
"แ": 67,
|
| 103 |
+
"แ": 68,
|
| 104 |
+
"แช": 69,
|
| 105 |
+
"แฉ": 70,
|
| 106 |
+
"แฒ": 71,
|
| 107 |
+
"แฆ": 72,
|
| 108 |
+
"แฎ": 73,
|
| 109 |
+
"แฐ": 74,
|
| 110 |
+
"แณ": 75,
|
| 111 |
+
"แ": 76,
|
| 112 |
+
"แญ": 77,
|
| 113 |
+
"แฌ": 78,
|
| 114 |
+
"\t": 79,
|
| 115 |
+
"\n": 80
|
| 116 |
+
},
|
| 117 |
+
"max_len": 24
|
| 118 |
+
},
|
| 119 |
+
"padding_id": 0,
|
| 120 |
+
"start_token": "\t",
|
| 121 |
+
"end_token": "\n",
|
| 122 |
+
"oov_token": "<unk>"
|
| 123 |
+
}
|