Translation
Keras
Khmer
bigru-additive-attention-seq2seq
transliteration
khmer
romanization
seq2seq
gru
attention
Eval Results (legacy)
Instructions to use Netra-Lab/AkaraAlpha with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Keras
How to use Netra-Lab/AkaraAlpha with Keras:
# !pip install -U keras tensorflow huggingface_hub # Keras needs TensorFlow installed to read "hf://" paths, so the tensorflow backend is selected here; # "jax" and "torch" also work for computation once TensorFlow is installed. import os os.environ["KERAS_BACKEND"] = "tensorflow" import keras model = keras.saving.load_model("hf://Netra-Lab/AkaraAlpha") - Notebooks
- Google Colab
- Kaggle
Fix the card metadata and add the pipeline figure
Browse filespipeline_tag is now translation (text2text-generation is not an official tag); the dataset entry, which pointed at a dataset that is not on the Hub, is removed. Adds a Dictionary check section with the full-pipeline figure and a note on ties.
- .gitattributes +1 -0
- README.md +12 -5
- assets/pipeline.png +3 -0
.gitattributes
CHANGED
|
@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
assets/architecture.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
bi_gru_attention_khmer_transliterator.keras filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
assets/architecture.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
bi_gru_attention_khmer_transliterator.keras filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
assets/pipeline.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -3,7 +3,7 @@ license: mit
|
|
| 3 |
language:
|
| 4 |
- km
|
| 5 |
library_name: keras
|
| 6 |
-
pipeline_tag:
|
| 7 |
tags:
|
| 8 |
- transliteration
|
| 9 |
- khmer
|
|
@@ -12,8 +12,6 @@ tags:
|
|
| 12 |
- gru
|
| 13 |
- attention
|
| 14 |
- keras
|
| 15 |
-
datasets:
|
| 16 |
-
- Chhunneng/khmer-text-transliteration
|
| 17 |
metrics:
|
| 18 |
- cer
|
| 19 |
- accuracy
|
|
@@ -21,10 +19,10 @@ model-index:
|
|
| 21 |
- name: AkaraAlpha
|
| 22 |
results:
|
| 23 |
- task:
|
| 24 |
-
type:
|
| 25 |
name: Romanized Khmer to Khmer transliteration
|
| 26 |
dataset:
|
| 27 |
-
type:
|
| 28 |
name: Khmer Text Transliteration Dataset (20% held out)
|
| 29 |
metrics:
|
| 30 |
- type: cer
|
|
@@ -65,6 +63,14 @@ The code is on [GitHub](https://github.com/netra-ai-lab/Romanized-Khmer-to-Khmer
|
|
| 65 |
|
| 66 |
The decoder doesn't start from the encoder's final state, so everything it knows about the input reaches it through attention.
|
| 67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
## Results
|
| 69 |
|
| 70 |
Character error rate of the models alone on the 5,714 held-out words. Lower is better.
|
|
@@ -162,6 +168,7 @@ print(transliterate("brodae")) # ααααα
|
|
| 162 |
- **Words, not sentences.** The model transliterates one word at a time, so it can't use the surrounding sentence to choose between two words with the same romanized spelling.
|
| 163 |
- **No separate test set.** The held-out words were also used to adjust the learning rate and choose the epoch, so the scores may be slightly optimistic.
|
| 164 |
- **One way of romanizing.** All the training data comes from one dataset. People who spell Khmer differently in Latin letters may get worse results.
|
|
|
|
| 165 |
- **The dictionary has edges.** A name or new word that isn't in the dictionary may be "corrected" into a real word within two edits.
|
| 166 |
- **Latency depends on hardware.** The timings were measured on one machine; read them as the relative cost of each setting.
|
| 167 |
|
|
|
|
| 3 |
language:
|
| 4 |
- km
|
| 5 |
library_name: keras
|
| 6 |
+
pipeline_tag: translation
|
| 7 |
tags:
|
| 8 |
- transliteration
|
| 9 |
- khmer
|
|
|
|
| 12 |
- gru
|
| 13 |
- attention
|
| 14 |
- keras
|
|
|
|
|
|
|
| 15 |
metrics:
|
| 16 |
- cer
|
| 17 |
- accuracy
|
|
|
|
| 19 |
- name: AkaraAlpha
|
| 20 |
results:
|
| 21 |
- task:
|
| 22 |
+
type: translation
|
| 23 |
name: Romanized Khmer to Khmer transliteration
|
| 24 |
dataset:
|
| 25 |
+
type: khmer-text-transliteration
|
| 26 |
name: Khmer Text Transliteration Dataset (20% held out)
|
| 27 |
metrics:
|
| 28 |
- type: cer
|
|
|
|
| 63 |
|
| 64 |
The decoder doesn't start from the encoder's final state, so everything it knows about the input reaches it through attention.
|
| 65 |
|
| 66 |
+
## Dictionary check
|
| 67 |
+
|
| 68 |
+
The model proposes *k* candidates by beam search, and each is normalised with [khmerspell](https://pypi.org/project/khmerspell/). Candidates that are dictionary words come first. Each remaining candidate is replaced by the closest dictionary word within two edits (Levenshtein distance), unless that word is already in the list. If no candidate is near a dictionary word, the raw candidates are returned.
|
| 69 |
+
|
| 70 |
+
<p align="center"><img src="https://huggingface.co/Netra-Lab/AkaraAlpha/resolve/main/assets/pipeline.png" width="100%" alt="The full pipeline on konhchreng: preprocessing, encoder and decoder, three beam candidates, the dictionary check, Levenshtein correction and the merged suggestions"></p>
|
| 71 |
+
|
| 72 |
+
*The full system on `konhchreng`, a held-out word, with k = 3.* Left, reading upwards: the word is cleaned, turned into letter ids and padded, and the model proposes three candidates by beam search. Right: candidates that are dictionary words are kept, and the others are replaced by their nearest dictionary word within two edits; a replacement that is already kept is dropped (dashed). The model ranked the correct word, αααα
αααα, second; the dictionary puts it first.
|
| 73 |
+
|
| 74 |
## Results
|
| 75 |
|
| 76 |
Character error rate of the models alone on the 5,714 held-out words. Lower is better.
|
|
|
|
| 168 |
- **Words, not sentences.** The model transliterates one word at a time, so it can't use the surrounding sentence to choose between two words with the same romanized spelling.
|
| 169 |
- **No separate test set.** The held-out words were also used to adjust the learning rate and choose the epoch, so the scores may be slightly optimistic.
|
| 170 |
- **One way of romanizing.** All the training data comes from one dataset. People who spell Khmer differently in Latin letters may get worse results.
|
| 171 |
+
- **Ties.** When several dictionary words are equally close to a candidate, the package keeps whichever it meets first, and that order can change between runs, so later suggestions can differ from run to run.
|
| 172 |
- **The dictionary has edges.** A name or new word that isn't in the dictionary may be "corrected" into a real word within two edits.
|
| 173 |
- **Latency depends on hardware.** The timings were measured on one machine; read them as the relative cost of each setting.
|
| 174 |
|
assets/pipeline.png
ADDED
|
Git LFS Details
|