shilla_base: direct Luganda ⇄ Ateso translation (pretrained base)

shilla_base is a 89M-parameter encoder-decoder Transformer that translates Luganda → Ateso and Ateso → Luganda directly, with one model and no English on either side. The direction is set by a prefix token on the source (<2teo> / <2lug>; the tag names the target language).

This is a pretrained base, not a finished translator. It was trained only on machine-translated pairs: both the Luganda side and the Ateso side of every training pair are output of one NLLB-based model (see Training). It has not been fine-tuned on human-translated data. Full test-set scores have not been run yet (see Evaluation).

Quick start

pip install torch sentencepiece safetensors huggingface_hub
import sys
from huggingface_hub import snapshot_download

folder = snapshot_download("amDANIEL2024/shilla_base")   # add token="hf_..." if the repo is still private
sys.path.insert(0, folder)
from shilla import Translator

t = Translator(folder)                                   # uses the GPU if there is one
ateso = t.translate("Okuzimba bbugwe kwatandika mu Gatonnya 2000.", "lug2teo")
print(ateso)
print(t.translate(ateso, "teo2lug"))                     # back again

Directions are "lug2teo" and "teo2lug". translate takes a string or a list and accepts beam (default 4). This is a custom PyTorch model, not a transformers model, so AutoModel will not load it.

Example outputs

Not yet added. Examples will be taken unedited from the SALT test set, mistakes included, once the evaluation has been run.

Model

Architecture pre-LN Transformer encoder-decoder, 6 + 6 layers, d_model 640, 8 heads, FFN 2560, GELU
Parameters 89M (input embedding, decoder embedding and output projection are tied)
Tokenizer shared SentencePiece unigram, 32,000 pieces, case preserved, digits split (spm.model); trained on Luganda and Ateso text only
Context 256 positions (sinusoidal); training pairs were limited to 128 pieces per side
Weights model.safetensors (float32)

Training

  • Data: the Luganda–Ateso pairs of tulati, used in both directions. In tulati every non-English column is a machine translation of one English column, made by Sunbird/translate-nllb-1.3b-salt; no human-written Luganda or Ateso is in it. The English source column was not used for this model. This card does not report how good that translator is for Ateso.
  • Filtering (counts from this run): of 1,534,699 tulati rows, 127,024 were dropped because they were flagged empty, degenerate, looks-like-English or length-outlier in either language (flagged in Luganda: 34,011, in Ateso: 104,732). Then 1,026 pairs with identical sides (removed) and 483 exact duplicates were removed, leaving 1,406,166 pairs. Decontamination against SALT dev/test removed 2 pair(s), leaving 1,406,164; 2,000 of those were held out for validation. No further pair-level filter was used.
  • Tokenizer: trained on tulati plus the text of the SALT train split (Luganda and Ateso only). The model weights were trained on tulati only.
  • Schedule: 5 epochs, 17,380 optimizer steps (about 25k target tokens per step), fp16 on one Kaggle T4, label smoothing 0.1, AdamW, linear warmup then inverse-sqrt decay (peak LR 3e-4).

Evaluation

Complete SALT dev/test scores (BLEU, chrF, chrF++): not run yet. SALT test has not been used for anything.

What has been measured so far, at the end of pretraining (step 17,380):

Set Direction Loss Quick chrF++ (greedy, first 128 sentences)
tulati validation (held-out machine-translated pairs) lug→teo 1.470 54.49
tulati validation (held-out machine-translated pairs) teo→lug 1.796 48.45
SALT dev (human translations) lug→teo 3.059 44.31
SALT dev (human translations) teo→lug 2.642 42.01

The quick chrF++ is a noisy check on a small subset with greedy decoding; it is not comparable to full-set scores. SALT dev was only observed here, not used to choose anything.

Copy baseline on the complete SALT dev set (output the source sentence unchanged), sacreBLEU 2.5.1:

Direction BLEU chrF chrF++
lug→teo 0.16 15.84 13.47
teo→lug 0.16 15.27 13.11

Note that SALT's Luganda and Ateso sentences are independent human translations of the same English sentence, so a reference in one language is not a translation of the reference in the other.

Limitations

  • Both sides of every training pair are machine output, so the model inherits that translator's errors and style. Real human-written text (dialect, spelling, code-switching) may be handled worse than benchmark numbers suggest.
  • Meaning can be dropped or changed, especially for numbers, names and rare words. Do not use it for medical, legal or other high-stakes content without human review.
  • Sentence-level model: inputs longer than about 128 pieces are truncated or degrade.
  • Outputs have not been reviewed by an Ateso reader.

Files

model.safetensors weights · config.json architecture and training summary · spm.model tokenizer · shilla.py standalone inference code · README.md this card.

License

Released under the MIT License (as are tulati, the translation model that generated the training data, and SALT).

Author

Amanyire Daniel (amDANIEL2024) · domusdeiug@gmail.com · https://domusdeiug.com

Downloads last month
-
Safetensors
Model size
89.4M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for amDANIEL2024/shilla_base

Finetunes
1 model

Dataset used to train amDANIEL2024/shilla_base