midwestcyr/nllb-200-distilled-600m-rus-ruskaromani-lora

NLLB Russian ↔ Ruska Romani (LoRA Adapter)

LoRA adapter fine-tuned on top of facebook/nllb-200-distilled-600M for Russian ↔ Ruska Romani translation.

Ruska Romani is not natively supported by NLLB-200. A custom language code, ruk_Cyrl, was added to the tokenizer to represent it. This repo contains only the LoRA adapter weights — you need the base NLLB model plus this adapter's tokenizer (which includes the added ruk_Cyrl token) to use it.

If you'd rather not deal with PEFT/merging yourself, see the companion merged, ready-to-use model: midwestcyr/nllb-200-distilled-600m-rus-ruskaromani.

Model details

• Base model: facebook/nllb-200-distilled-600M
• Fine-tuning method: LoRA (via 🤗 PEFT), targeting attention and feed-forward projections (q_proj, k_proj, v_proj, out_proj, fc1, fc2)
• New language code: ruk_Cyrl (Ruska Romani, Cyrillic script) — added to the tokenizer vocabulary and warm-started from the embedding of rom_Latn (Vlax Romani, the closest existing NLLB language) rather than a random vector
• Translation direction trained: rus_Cyrl → ruk_Cyrl only. The reverse direction was not trained and is not expected to work well.
• Training data: parallel Russian / Ruska Romani sentence pairs extracted from a JSON corpus, split 95% train / 5% validation
• Evaluation metric: SacreBLEU on the held-out validation split

Data

Ruska Romani is the dialect of the Romani language attributed to Ruska Roma, the largest subgroup of Romani people in Russia.

Training and evaluation data consists of parallel Russian / Ruska Romani sentence pairs from the Russian National Corpus. The data curation process is described in The Parallel Corpus of Russian and Ruska Romani Languages. Please cite that paper in any publication using this model or the underlying corpus.

Training

The model was fine-tuned using the Hugging Face Seq2SeqTrainer with LoRA adapters on parallel Russian ↔ Ruska Romani sentence pairs.

Training configuration

Parameter	Value
Epochs	8
Per-device train batch size	4
Per-device evaluation batch size	4
Gradient accumulation steps	2
Effective training batch size	8
Learning rate	3e-4
Mixed precision	FP16
Evaluation strategy	Every epoch
Checkpoint saving	Every epoch
Logging frequency	Every 50 steps
Generation during evaluation	Enabled
Maximum generation length	MAX_LEN
Best model selection	Highest validation SacreBLEU
Trainer	Hugging Face Seq2SeqTrainer

Training results

Metric	Value
Total epochs completed	8
Total optimization steps	7,432
Best validation SacreBLEU	22.93
Best model epoch	8

The best-performing checkpoint was automatically restored at the end of training (load_best_model_at_end=True) using the validation SacreBLEU score as the model selection criterion.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel
import torch

BASE_MODEL = "facebook/nllb-200-distilled-600M"
ADAPTER_REPO = "midwestcyr/nllb-200-distilled-600m-rus-ruskaromani-lora"

# load base model
base_model = AutoModelForSeq2SeqLM.from_pretrained(BASE_MODEL)

# load tokenizer from the ADAPTER repo, not the base model —
# it includes the new ruk_Cyrl token and resized vocab
tokenizer = AutoTokenizer.from_pretrained(ADAPTER_REPO)

# resize the base model's embeddings to match the tokenizer
# BEFORE loading the adapter
base_model.resize_token_embeddings(len(tokenizer))

# load the LoRA adapter on top
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO)

device = torch.device("cuda" if torch.cuda.is_available() else
                       "mps" if torch.backends.mps.is_available() else "cpu")
model = model.to(device)
model.eval()

new_lang_id = tokenizer.convert_tokens_to_ids("ruk_Cyrl")

def translate(text, max_length=128, num_beams=5):
    tokenizer.src_lang = "rus_Cyrl"
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=max_length).to(device)
    generated = model.generate(
        **inputs,
        forced_bos_token_id=new_lang_id,
        max_length=max_length,
        num_beams=num_beams,
    )
    return tokenizer.batch_decode(generated, skip_special_tokens=True)[0]

print(translate("Привет, как дела?"))

Merging the adapter yourself

merged = model.merge_and_unload()
merged.save_pretrained("nllb-ru-ruskaromani-merged")
tokenizer.save_pretrained("nllb-ru-ruskaromani-merged")

Limitations

• Low-resource pair: dataset size is modest; translation quality on out-of-domain or long/complex sentences may be inconsistent.
• Custom language code: ruk_Cyrl is not a standard NLLB/FLORES code. If you use this adapter with a different base checkpoint or tokenizer, you must add the same token yourself before loading the adapter.

Citation / acknowledgment

Ruska Romani and Russian Parallel Corpus

@inproceedings{koncha-etal-2024-parallel,
    title = "The Parallel Corpus of {R}ussian and Ruska {R}omani Languages",
    author = "Koncha, Kirill  and
      Kukanova, Abina  and
      Tatiana, Kazakova  and
      Rozovskaya, Gloria",
    editor = "Serikov, Oleg  and
      Voloshina, Ekaterina  and
      Postnikova, Anna  and
      Muradoglu, Saliha  and
      Le Ferrand, Eric  and
      Klyachko, Elena  and
      Vylomova, Ekaterina  and
      Shavrina, Tatiana  and
      Tyers, Francis",
    booktitle = "Proceedings of the Third Workshop on NLP Applications to Field Linguistics",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.fieldmatters-1.1/",
    doi = "10.18653/v1/2024.fieldmatters-1.1",
    pages = "1--5"
}

Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model

@inproceedings{koishekenov-etal-2023-memory,
    title = "Memory-efficient {NLLB}-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model",
    author = "Koishekenov, Yeskendir  and
      Berard, Alexandre  and
      Nikoulina, Vassilina",
    editor = "Rogers, Anna  and
      Boyd-Graber, Jordan  and
      Okazaki, Naoaki",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.198/",
    doi = "10.18653/v1/2023.acl-long.198",
    pages = "3567--3585",
    abstract = "The recently released NLLB-200 is a set of multilingual Neural Machine Translation models that cover 202 languages. The largest model is based on a Mixture of Experts architecture and achieves SoTA results across many language pairs. It contains 54.5B parameters and requires at least four 32GB GPUs just for inference. In this work, we propose a pruning method that enables the removal of up to 80{\%} of experts without further finetuning and with a negligible loss in translation quality, which makes it feasible to run the model on a single 32GB GPU. Further analysis suggests that our pruning metrics can identify language-specific experts."
}