shilluk_mt / README.md
dayomtechnologies's picture
Upload folder using huggingface_hub
599bb35 verified
|
Raw History Blame Contribute Delete
1.44 kB
---
tags:
- translation
- machine-translation
- shilluk
- low-resource
---
# Shilluk ↔ English translation (trained from scratch)
Two encoder-decoder Transformers trained from scratch (no pretrained weights) on a Shilluk–English Bible parallel corpus.
| Direction | File | Test-set scores |
|---|---|---|
| Shilluk → English | `shilluk2en.pt` | BLEU 14.4 / chrF 34.0 / chrF++ 33.4 (test split, 1507 sentences, beam 5) |
| English → Shilluk | `en2shilluk.pt` | BLEU 11.0 / chrF 32.2 / chrF++ 30.8 (test split, 1507 sentences, beam 5) |
* Shared SentencePiece tokenizer: `spm_shared.model`
* Architecture/decoding code: `src/model.py`, `src/decode.py`, `tokenizer/tokenizer_utils.py`
## Usage
```python
import torch
from src.model import load_model_from_checkpoint
from src.decode import translate
from tokenizer.tokenizer_utils import SPTokenizer
tok = SPTokenizer("spm_shared.model")
model, meta = load_model_from_checkpoint("en2shilluk.pt", "cpu")
print(translate(model, tok, "In the beginning God created the heaven and the earth.", "cpu", beam_size=5))
```
(Put `spm_shared.model` at `tokenizer/spm_shared.model` or pass its path as above.)
## Limitations
Trained on Bible text only: expect good results on Bible-style sentences and unreliable output on everyday or modern vocabulary. Scores are on held-out chapters of the same corpus, so they overestimate quality on other domains.