SMOL: Professionally translated parallel data for 115 under-represented languages
Paper • 2502.12301 • Published • 1
BOHOK pilot checkpoint
This repository contains a from-scratch byte-level T5 trained by the BOHOK research platform. It is not IndicTrans2, not NLLB, and not a production translator.
What this checkpoint is
| Architecture | T5 encoder-decoder, byte tokenizer (ByT5Tokenizer) |
| Parameters | about 3.8 million |
| Initialization | random |
| Training data | Google SMOL CC-BY-4.0 sentence pairs only |
| Languages in training | Kokborok (Latin), Garo (Latin), Mundari (Devanagari), Santali (Ol Chiki), Bodo (Devanagari), Mizo (Latin), Meitei (Meitei Mayek) |
| Steps | 3,396 on CPU, seed 42 |
| Best validation loss | 2.12 |
| Hardware | 4 CPU cores, no GPU |
The source string is prefixed with the language id (kokborok: , and the same pattern for the other ids). Decoding in the evaluation was greedy.
What this checkpoint is not
- It was not fine-tuned from IndicTrans2, Indic-Translate, NLLB, ByT5-small, or OPUS-MT.
- It does not cover Gondi, Bhili, Kui, Khasi, Nyishi, Karbi, Nagamese, Tagin, or Bengali-script Kokborok.
- Bengali and Marathi baselines in the BOHOK app are separate Helsinki-NLP OPUS-MT models. Their weights are not in this repository.
- A 40-step LoRA run from OPUS-MT Bengali–English is a different experiment and is not this checkpoint.
Results measured for this checkpoint
Held-out SMOL sentences, up to 40 per language, SacreBLEU 2.6.0. COMET was not run.
| Language | BLEU | chrF++ |
|---|---|---|
| Bodo | 0.1680 | 6.5293 |
| Garo | 0.0974 | 5.3811 |
| Kokborok | 0.0698 | 4.8828 |
| Meitei | 0.1678 | 6.2282 |
| Mizo | 0.0914 | 5.2372 |
| Mundari | 0.0839 | 5.9302 |
| Santali | 0.1605 | 6.1771 |
Outputs degenerate into repeated function words. Do not use them for communication, education, health, or government text.
For context only, unmodified OPUS-MT on a different 40-sentence OPUS-100 sample scored Bengali BLEU 22.58 / chrF++ 44.22 and Marathi BLEU 27.32 / chrF++ 39.29. Those numbers are not scores of this checkpoint. No Google Translate comparison exists.
Data and licence
Training pairs are the *_*_en SMOL files listed in the BOHOK dataset registry. English is the original side (is_src_orig=false). Reverse files were not added. GATITOS was not used. Samanantar (CC-BY-NC-4.0) was not used.
The checkpoint is released under CC-BY-4.0 because that is the SMOL licence. Cite Caswell et al., arXiv:2502.12301, and Sudhamoy DebBarma for the Kokborok translations. BOHOK code is Apache-2.0 and lives with the Space.
Intended use
Reproduce the negative CPU pilot and compare future fine-tunes against it. Research on data curation and evaluation harnesses.
Out of scope
Deploying this file as a community translation service.
Evaluation method
Document-grouped split, seed 42, English targets that also appear in test removed from train and validation. Metrics: BLEU and chrF++. References were not used as training text.
Limitations
Undertrained random initialization, translationese from English-original bitext, no pretrained vocabulary, no external benchmark (IN22, FLORES, and WMT files were not in the training or test pools because they were gated or not downloaded).
Ethical notes
A low automatic score is not a judgement of the languages. It is a judgement of this model. Speakers should not be asked to correct a stream of repeated English function words as if it were a serious draft.