wujang/qwen2_5_1_5b_balanda_qlora

Qwen2.5-1.5B Balanda (Njo Viri) QLoRA Adapter

A QLoRA fine-tune of Qwen/Qwen2.5-1.5B-Instruct on the Balanda language (also called Njo Viri), spoken by the Balanda people of South Sudan and the Democratic Republic of the Congo. Njo Viri is an undocumented language with no prior presence on the HuggingFace Hub.

Status

v1 — experimental. Trained primarily for Balanda → English translation, which performs well. Other directions and tasks are included but undertrained.

Task	Test N	chrF++	BLEU
Balanda → English	267	92.9	86.3
English → Balanda	5	22.9	8.6

English → Balanda scores reflect severe data imbalance in v1 (only 5 test examples, ~40 training examples). A v2 adapter with mirrored translation pairs is planned.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

ADAPTER = "wujang/qwen2_5_1_5b_balanda_qlora"
BASE = "Qwen/Qwen2.5-1.5B-Instruct"

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,
)

tok = AutoTokenizer.from_pretrained(ADAPTER)
base = AutoModelForCausalLM.from_pretrained(BASE, quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(base, ADAPTER).eval()

messages = [
    {"role": "system", "content": "You are a Balanda (Njo Viri) language expert. Follow the user's instruction precisely."},
    {"role": "user", "content": "Translate from Balanda to English\n\nNi timande kee ni ja nja gba."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())
# → "He works but does not eat."

Training

• Base model: Qwen2.5-1.5B-Instruct (Apache-2.0)
• Method: QLoRA, 4-bit nf4 double quantization
• LoRA: r=16, α=32, dropout=0.05, target = all linear modules (1.18% trainable params)
• Trainer: TRL SFTTrainer with chat-template formatting
• Epochs: 3, effective batch size 16 (2 × 8 grad accum), lr 2e-4 cosine, warmup 3%
• Hardware: Single T4 (Colab free tier), bf16, ~84 min total
• Max seq length: 1024
• Seed: 42

Loss curve:

Epoch	Train loss	Val loss
1	0.433	0.478
2	0.336	0.412
3	0.292	0.412

Dataset

Instruction-tuning JSONL derived from hand-curated Njo Viri linguistic data:

• Translation pairs (primary: Balanda → English)
• Grammar rule explanations (conjunctions, tense/aspect, pronouns, modals, etc.)
• Sentence identification and correction tasks
• Alphabet and number system

Source: github.com/Bawilla/Njo-Viri

Totals: 4,745 train / 349 validation / 352 test, stratified by source file and instruction type.

Known limitations

• Primarily one-directional (Bal → En). English → Balanda generation is unreliable in v1.
• Limited vocabulary coverage; rare lexical items may not translate.
• No dialectal variants modeled.
• Grammar-explanation outputs are templated and may not generalize beyond seen structures.

Citation

If you use this adapter or the underlying dataset for research on Njo Viri or low-resource African language modeling, please cite:

@misc{ujang2026njoviri,
  author = {Ujang, Jamson Batista Ulau},
  title  = {Njo Viri (Balanda) instruction-tuning dataset and QLoRA adapter},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/wujang/qwen2_5_1_5b_balanda_qlora}},
}

Author

Built by wujang. Dataset authored through direct elicitation and analysis of Njo Viri as a living language.