urdu-286m-instruct

Instruction-tuned version of urdu-286m. It answers questions in Urdu, in the chat format, from a base model that was pretrained from random initialisation on 2.85B tokens of Urdu text.

It has no reliable knowledge. That is stated here rather than buried, because the benchmark below says so.

What changed from the base model

base instruct
Behaviour continues text in news register answers a question in the chat format
Validation bpb 0.6113 on raw text 0.4050 on held-out instruction data

The two bpb numbers are not comparable. The base model's is next-token loss on raw documents, the instruct model's is loss on held-out instruction pairs, which are shorter and easier.

Training data, all public, nothing generated

Source Rows used Licence
large-traversaal/urdu-instruct 49,275 CC-BY-SA-4.0
CohereLabs/aya_collection_language_split, urdu, filtered to 80-900 chars 27,909 Apache-2.0
large-traversaal/commonsenseqa_urdu_final, four options kept 9,737 unstated upstream
large-traversaal/openbookqa_urdu_final 4,957 unstated upstream

Total 90,878 rows, about 5.7M tokens. A 5,000-row holdout from the same sources is used for validation and is never trained on.

Aya's Urdu split is mostly translated long-context question answering, median 2,219 characters per row, so only the short-answer band is used. CommonsenseQA ships five options, so four are subsampled with the correct answer kept, which makes every multiple choice item four-option.

Training

Instruction tuning only, from the published base checkpoint at step 4,200.

Setting Value
Steps 101, one pass
Batch 65,536 tokens
Sequence length 1,024
Learning rate 0.8 of the pretraining value
Validation bpb 0.4050

Two passes over four passes were tested and rejected: with four passes the validation bpb rose from 0.3488 to 0.4077 while training loss fell, which is overfitting on 5.7M tokens.

What it does and does not do

Works: answers in Urdu, stays on topic, produces a paragraph when asked for one, keeps the answer short when the question is short. 19 of 19 sampled prompts returned real Urdu text, median 123 characters.

Does not work: facts. It says Pakistan beat India in 2011 in Lahore, it says it will snow in Lahore. It cannot hold a long conversation, because the training data is single-turn. Multiple choice answers collapse to a letter prior, because the model has no knowledge to retrieve, so the letter it picks follows corpus frequency rather than the question.

The honest summary: this is a 286M parameter model that learned the shape of an Urdu assistant, without the knowledge that would make the answers true.

Samples

Six prompts from a sheet of 33 across ten task types. Answers are verbatim, including the wrong ones.

Definitions work. Prompt جمہوریت کیا ہے؟

جمہوریت ایک ایسی حکومت یا ادارہ ہے جس میں ہر شخص کو اپنے خیالات کا حق حاصل ہے، بشمول سیاسی، عسکری اور سماجی۔

Prompt فوٹو سنتھیسس کیا ہے؟

فوٹو سنتھیسس سورج کی روشنی کو توڑ کر پانی میں موجود کاربن ڈائی آکسائیڈ کو جذب کرکے زمین کو آکسیجن فراہم کرتا ہے۔

Some facts land. Prompt پاکستان کا قومی پھول کیا ہے؟

پاکستان کا قومی پھول چنبیلی ہے۔

Structured writing is the strongest thing here. Prompt پرنسپل کو چھٹی کی درخواست لکھیں۔

محترم پرنسپل صاحب، میں اپنی بیماری کی وجہ سے پچھلے دو ہفتوں سے چھٹی پر جا رہا ہوں۔ برا کرم میری درخواست پر غور فرمائیں۔ شکریہ۔

Arithmetic shows the limit precisely. The format is perfect, the sum is not. Prompt 2 اور 2 کا مجموعہ کتنا ہے؟

Reasoning: 2 کا مجموعہ: 2 + 2 = 4 2 کا مجموعہ: 4 + 2 = 7 Answer: 7

And on a two-turn exchange it gives the wrong city, then corrects itself when challenged. First turn پاکستان کا سب سے بڑا شہر کون سا ہے؟

پاکستان کا سب سے بڑا شہر لاہور ہے۔

Then کراچی ہے یا لاہور؟

کراچی پاکستان کا سب سے بڑا شہر ہے۔

The full sheet is in samples.md in this repository: 33 prompts, 39 answers including the follow-ups, none empty, median 107 characters.

How to use

The weights are in the nanochat checkpoint layout, so they load with the same code as the base model:

from nanochat.checkpoint_manager import load_model

# NANOCHAT_BASE_DIR points at this repository's directory
model, tokenizer, meta = load_model("sft", device, phase="eval",
                                    model_tag="urdu_286m_instruct")
ids, _ = tokenizer.render_conversation(
    {"messages": [{"role": "user", "content": "پاکستان کا سب سے بڑا شہر کون سا ہے؟"}]})
ids.append(tokenizer.encode_special("<|assistant_start|>"))

The <|assistant_start|> token matters. render_conversation stops after the user turn, and generation has to be seeded with the assistant token or the model continues from a position training never covered.

There is no GGUF. nanochat's architecture uses value embeddings, per-layer residual lambdas and a logit cap, and no llama.cpp implementation exists for it.

Licence

Weights: CC-BY-NC-4.0. Free for research, evaluation and personal use, with attribution. Commercial use requires a licence, available on request at me@awahab.dev.

The training data carries its own terms and they sit underneath this licence: urdu-instruct is CC-BY-SA-4.0, Aya is Apache-2.0, and the two MCQ translations state no licence, so their terms are inherited from CommonsenseQA and OpenBookQA upstream.

Citation

@misc{urdu286m_instruct,
  title  = {urdu-286m-instruct: an Urdu instruction-tuned model trained from scratch},
  author = {Abdul Wahab},
  year   = {2026},
  note   = {https://huggingface.co/awahab111/urdu-286m-instruct}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for awahab111/urdu-286m-instruct

Finetuned
(1)
this model

Datasets used to train awahab111/urdu-286m-instruct