shilla_base: direct Luganda ⇄ Ateso translation (pretrained base)
shilla_base is a 89M-parameter encoder-decoder Transformer that translates Luganda → Ateso and
Ateso → Luganda directly, with one model and no English on either side. The direction is set by a prefix token on the
source (<2teo> / <2lug>; the tag names the target language).
This is a pretrained base, not a finished translator. It was trained only on machine-translated pairs: both the Luganda side and the Ateso side of every training pair are output of one NLLB-based model (see Training). It has not been fine-tuned on human-translated data. Full test-set scores have not been run yet (see Evaluation).
Quick start
pip install torch sentencepiece safetensors huggingface_hub
import sys
from huggingface_hub import snapshot_download
folder = snapshot_download("amDANIEL2024/shilla_base") # add token="hf_..." if the repo is still private
sys.path.insert(0, folder)
from shilla import Translator
t = Translator(folder) # uses the GPU if there is one
ateso = t.translate("Okuzimba bbugwe kwatandika mu Gatonnya 2000.", "lug2teo")
print(ateso)
print(t.translate(ateso, "teo2lug")) # back again
Directions are "lug2teo" and "teo2lug". translate takes a string or a list and accepts beam (default 4).
This is a custom PyTorch model, not a transformers model, so AutoModel will not load it.
Example outputs
Not yet added. Examples will be taken unedited from the SALT test set, mistakes included, once the evaluation has been run.
Model
| Architecture | pre-LN Transformer encoder-decoder, 6 + 6 layers, d_model 640, 8 heads, FFN 2560, GELU |
| Parameters | 89M (input embedding, decoder embedding and output projection are tied) |
| Tokenizer | shared SentencePiece unigram, 32,000 pieces, case preserved, digits split (spm.model); trained on Luganda and Ateso text only |
| Context | 256 positions (sinusoidal); training pairs were limited to 128 pieces per side |
| Weights | model.safetensors (float32) |
Training
- Data: the Luganda–Ateso pairs of tulati, used in both directions. In tulati every non-English column is a machine translation of one English column, made by Sunbird/translate-nllb-1.3b-salt; no human-written Luganda or Ateso is in it. The English source column was not used for this model. This card does not report how good that translator is for Ateso.
- Filtering (counts from this run): of 1,534,699 tulati rows, 127,024 were dropped because they were flagged empty, degenerate, looks-like-English or length-outlier in either language (flagged in Luganda: 34,011, in Ateso: 104,732). Then 1,026 pairs with identical sides (removed) and 483 exact duplicates were removed, leaving 1,406,166 pairs. Decontamination against SALT dev/test removed 2 pair(s), leaving 1,406,164; 2,000 of those were held out for validation. No further pair-level filter was used.
- Tokenizer: trained on tulati plus the text of the SALT train split (Luganda and Ateso only). The model weights were trained on tulati only.
- Schedule: 5 epochs, 17,380 optimizer steps (about 25k target tokens per step), fp16 on one Kaggle T4, label smoothing 0.1, AdamW, linear warmup then inverse-sqrt decay (peak LR 3e-4).
Evaluation
Complete SALT dev/test scores (BLEU, chrF, chrF++): not run yet. SALT test has not been used for anything.
What has been measured so far, at the end of pretraining (step 17,380):
| Set | Direction | Loss | Quick chrF++ (greedy, first 128 sentences) |
|---|---|---|---|
| tulati validation (held-out machine-translated pairs) | lug→teo | 1.470 | 54.49 |
| tulati validation (held-out machine-translated pairs) | teo→lug | 1.796 | 48.45 |
| SALT dev (human translations) | lug→teo | 3.059 | 44.31 |
| SALT dev (human translations) | teo→lug | 2.642 | 42.01 |
The quick chrF++ is a noisy check on a small subset with greedy decoding; it is not comparable to full-set scores. SALT dev was only observed here, not used to choose anything.
Copy baseline on the complete SALT dev set (output the source sentence unchanged), sacreBLEU 2.5.1:
| Direction | BLEU | chrF | chrF++ |
|---|---|---|---|
| lug→teo | 0.16 | 15.84 | 13.47 |
| teo→lug | 0.16 | 15.27 | 13.11 |
Note that SALT's Luganda and Ateso sentences are independent human translations of the same English sentence, so a reference in one language is not a translation of the reference in the other.
Limitations
- Both sides of every training pair are machine output, so the model inherits that translator's errors and style. Real human-written text (dialect, spelling, code-switching) may be handled worse than benchmark numbers suggest.
- Meaning can be dropped or changed, especially for numbers, names and rare words. Do not use it for medical, legal or other high-stakes content without human review.
- Sentence-level model: inputs longer than about 128 pieces are truncated or degrade.
- Outputs have not been reviewed by an Ateso reader.
Files
model.safetensors weights · config.json architecture and training summary · spm.model tokenizer ·
shilla.py standalone inference code · README.md this card.
License
Released under the MIT License (as are tulati, the translation model that generated the training data, and SALT).
Author
Amanyire Daniel (amDANIEL2024) · domusdeiug@gmail.com · https://domusdeiug.com
- Downloads last month
- -