urdu-126m: a 125.9M parameter Urdu language model trained from scratch
A decoder-only language model pretrained from random initialisation on Urdu text, with a 32,768-entry Urdu BPE tokenizer trained on the same corpus. No pretrained weights are used at any stage. It is a base model, so it continues text. It does not follow instructions.
This is not a transformers model. It uses the
nanochat architecture, which includes value
embeddings, a smear gate, per-layer residual lambdas and a logit soft cap. No
llama.cpp or GGUF implementation of that architecture exists, so run it with
nanochat's code, shown below.
Model family
| Model | Depth | Transformer matrices | Total params | Tokens | Val bpb | UrduMMLU |
|---|---|---|---|---|---|---|
| urdu-126m | 8 | 25.2M | 126M | 0.84B | 0.6953 | 24.84% |
| urdu-286m | 12 | 84.9M | 286M | 2.20B | 0.6113 | 24.58% |
Both models use the same corpus, the same tokenizer and the same recipe. Only the size changed. The larger model compresses held-out Urdu 12% better, and that is the whole scaling result.
Architecture
| Layers | 8 |
| d_model | 512 |
| Query heads / KV heads | 4 / 4, head dim 128 |
| MLP | 4x expansion, ReLU squared |
| Positions | RoPE |
| Normalisation | RMSNorm, pre-norm |
| Attention | sliding window, pattern SSSL (quarter context on S layers) |
| Context | 2,048 tokens |
| Vocab | 32,768 |
Where the parameters live:
| Component | Parameters |
|---|---|
| Transformer matrices (attention + MLP) | 25.2M |
| Unembedding head | 16.8M |
| Token embedding | 16.8M |
| Value embeddings | 67.1M |
| Total stored | 125.9M |
More than half of this model is lookup tables rather than computation. The number to use when comparing sizes or reasoning about scaling is 42.0M, the matrices plus the head, because that is what does arithmetic on every token.
Tokenizer: the reason this project exists
Urdu is served badly by tokenizers built for English. Measured on 3,000 held-out Wikipedia documents (11.4M characters), the same text through two tokenizers:
| Tokenizer | Tokens per word | Bytes per token |
|---|---|---|
| GPT-2 BPE | 5.95 | 1.42 |
| This model's 32k Urdu BPE | 1.25 | 6.76 |
4.76x fewer tokens for identical text, so training and inference cost 4.76x less
per document. The trade is real and runs the other way: this tokenizer spends 3
tokens on education where GPT-2 spends 1. Cheap for Urdu, expensive for English,
by design. A bilingual model would need a mixed vocabulary instead.
Training data
| Sources | FineWeb-2 urd_Arab, Wikipedia 20231101.ur |
| Documents before filtering | 4.86M web, 200,154 articles |
| Documents after cleaning | 4,048,757 |
| Tokens | 2.85B in the corpus, 0.84B used for this model, one pass, no repetition |
| Normalisation | Arabic yeh/kaf/heh folded to Urdu code points, diacritics and bidi controls stripped, digits normalised, URLs and markup removed |
| Quality gates | language score, duplicate cluster size, length bounds, script ratio, symbol ratio, repetition guard |
| Deduplication | exact, BLAKE2b, across both sources |
| Validation split | Wikipedia only, excluded from training, so val bpb measures cross-source generalisation |
| Licence | FineWeb-2 is ODC-By 1.0, Wikipedia text is CC BY-SA |
Training recipe
| Hardware | one L40S on Modal, bf16 |
| Steps | 3,200 |
| Batch | 262,144 tokens, gradient accumulation |
| Wall clock | 59 min |
| Compute cost | $1.92 |
| Optimiser | Muon for matrices (lr 0.02, momentum 0.95), AdamW for embeddings (0.3) and scalars |
| Weight decay | 0.28 |
| Warmup / decay | 40 steps warmup, 65% warmdown, final LR 5% of peak |
| Checkpoints | every 200 steps, committed to a persistent volume |
| Final validation bpb | 0.6953 |
Results
Validation bits per byte on the held-out Wikipedia split. Bits per byte is tokenizer-independent, which is why it is the metric here and not perplexity.
| Step | bpb |
|---|---|
| 0 | 2.2648 |
| 250 | 0.9639 |
| 500 | 0.8535 |
| 1000 | 0.8089 |
| 1500 | 0.7783 |
| 2000 | 0.751 |
| 2500 | 0.722 |
| 3000 | 0.6965 |
| 3200 | 0.6953 |
Two caveats worth stating. Bits per byte compares models on the same text, not across languages, because UTF-8 Urdu spends two bytes per character and each byte therefore carries less information than an English byte. And a falling bpb means better compression of Urdu, not better reasoning.
Benchmarks: the size comparison
Five probes, both models, the same items, all built from the held-out split so neither trained on them. The paired test is McNemar's, which counts the items where only one model was right.
| Metric | urdu-126m | urdu-286m | Gap | Test |
|---|---|---|---|---|
| HellaSwag-ur, 4-way continuation | 57.05% | 60.60% | +3.55 pts | McNemar, p = 9.9e-06 |
| Final-word top-1 | 24.25% | 25.45% | +1.20 pts | McNemar, p = 0.0025 |
| Repeated 4-gram share, lower is better | 7.95% | 3.64% | -4.31 pts | descriptive |
| Distinct-2, higher is better | 0.813 | 0.881 | +0.068 | descriptive |
| Bits per byte, whole held-out split | 0.6953 | 0.6113 | -12.1% | - |
Bits per byte by document length, where the smaller model falls further behind on short documents:
| Document size | urdu-126m | urdu-286m | Relative gain |
|---|---|---|---|
| under 2KB | 0.9984 | 0.8159 | 18.3% |
| 2 to 5KB | 0.8339 | 0.7152 | 14.2% |
| 5 to 10KB | 0.7526 | 0.6668 | 11.4% |
| over 10KB | 0.6870 | 0.6147 | 10.5% |
HellaSwag-ur is built rather than borrowed: the context is the first three to five sentences of a held-out page, the true option is the sentence that actually follows, and the three distractors come from other documents. Each option is scored by its mean token log-likelihood given the context, which is exactly what a base model is trained to produce. Chance is 25%.
What the gap looks like in text. Prompted with مصنوعی ذہانت, the smaller model
writes "Artificialicial Intelligence", a mis-spelling, while the larger one writes it
correctly and adds the byline convention تحریر: علی احمدی. Fifteen prompts across
registers, same seed, are side by side in SAMPLES_COMPARE.md in this repo.
What these numbers do not show: neither model has knowledge, and this does not make UrduMMLU meaningful for either of them. The benchmark can see the size difference because it tests the training objective, not because either model learned facts.
UrduMMLU
This model scores 24.84% on UrduMMLU's 26,431 questions, against a 25.0% random baseline. That number is not a measure of knowledge, and the reason is measurable.
The letter-prior analysis above, plus two further scoring protocols, all land at chance. Rescoring by option content removes the letter bias and does not help.
| Protocol | urdu-126m | urdu-286m | Chance |
|---|---|---|---|
| Answer letter probability, n = 26,431 | 24.84% | 24.58% | 25.0% |
| Option likelihood, n = 26,431 | 24.54% | 22.71% | 25.0% |
| Option likelihood, n = 5,000 subset | 25.78% | 25.14% | 25.0% |
| Option PMI, n = 5,000 subset | 24.34% | 24.22% | 25.0% |
XNLI-ur, the standard cross-lingual NLI benchmark, 5,010 test pairs, both models: 33.33% against a 33.33% class prior, with 100% of predictions on a single label.
Raw option likelihood can land below chance because mean token likelihood prefers a generic phrase over a specific correct answer. PMI subtracts each option's own prior and does not rescue it either. At this scale there is no measurable school knowledge, and no multiple-choice protocol recovers any. A real knowledge number needs instruction tuning first.
Benchmark leakage, measured
Each UrduMMLU question contributes a 60-character fingerprint, normalised like the corpus, and the corpus was searched for those spans.
| File | Documents | Documents containing a fingerprint |
|---|---|---|
| Wikipedia | 165,039 | 1 |
| FineWeb-2 Urdu | 4,048,757 | 140 |
166 questions, 0.63% of the benchmark. The leaked spans are classical poetry that the web reproduces, and the affected subdomains, Urdu literature, Urdu grammar, Urdu language and general knowledge, all score at the chance line. Storing the text did not buy the ability to answer questions about it.
Samples
Seeded, temperature 0.8, top-k 50. The full sheet is in samples_126m.md.
Prompt کرکٹ میچ میں:
پاکستان کو شکست، آسٹریلیا نے سیریز میں برتری حاصل کرلی سڈنی: ٹی ٹوئنٹی ورلڈکپ کے اہم میچ میں آسٹریلیا پاکستان کو 10 وکٹوں سے شکست دے کر ورلڈکپ سے باہر ہوگیا، آسٹریلیا نے پاکستان کا 189 رنز کا ہدف 19ویں اوور میں صرف 3 وکٹوں کے نقصان پر حاصل کرلیا۔
The news register, dateline, overs and toss idiom are learned. The logic is not, because a team cannot win by ten wickets and leave the tournament in one sentence.
Prompt مصنوعی ذہانت:
(AI) کے ارد گرد بہت سے غلط فہمیاں ہیں، اور یہ کہنا درست نہیں ہوگا کہ آپ کو مصنوعی ذہانت کی ضرورت ہے۔ AI کی حقیقت میں ایک اور وسیع پیمانے پر تصویر ہے۔
How to run
git clone https://github.com/karpathy/nanochat
hf download awahab111/urdu-126m --local-dir ./urdu-126m
export NANOCHAT_BASE_DIR=$PWD/urdu-126m
python -m scripts.chat_cli # asks for prompts, streams Urdu
Repository layout mirrors what nanochat expects:
base_checkpoints/urdu_126m/model_003200.pt
base_checkpoints/urdu_126m/meta_003200.json
tokenizer/tokenizer.pkl
tokenizer/token_bytes.pt
Load it directly if you prefer:
from nanochat.checkpoint_manager import load_model
model, tokenizer, meta = load_model("base", device, phase="eval",
model_tag="urdu_126m")
Checkpoint size on disk is 321 MB. Measured 19.8 tokens/sec for the 286M model on a CPU-only box, so a GPU is optional for demos.
Limitations
It produces grammatical, register-correct Urdu, and it invents facts, numbers and causes without warning. It repeats itself and can contradict itself inside one paragraph. It is a research and portfolio artifact. Nothing it generates is advice or a factual source.
Corpus composition shows through. The Urdu web inside FineWeb-2 is heavily Indian
Urdu news, so prompted with تعلیم کی اہمیت (the importance of education) this
model produces Indian political parties and schemes.
There is no instruction tuning in these weights. There is no SFT stage, no chat template beyond the base token format, and no safety tuning of any kind.
Reproduce
The full pipeline, roughly 15 minutes of CPU work and about $13 of GPU time:
python scripts/build_fineweb.py --workers 3 # 4.86M web docs -> clean JSONL
python scripts/build_wiki_hf.py # 200k articles -> clean JSONL
python scripts/decontaminate.py # benchmark fingerprints -> skip list
python scripts/make_shards.py # dedup, hash-shard, val split
python -m scripts.tok_train --vocab-size 32768 # 32k Urdu BPE
python -m modal run modal_train.py --depth 8 --ratio 20 --tag urdu_126m
python -m modal run modal_train.py::evaluate --tag urdu_126m
License
Weights: CC-BY-NC-4.0. Free for research, evaluation and personal use, with attribution. Commercial use requires a licence, available on request at me@awahab.dev.
Training data carries its own terms underneath: FineWeb-2 is ODC-By 1.0, the Wikipedia extract is CC BY-SA 4.0, and the instruction data used for any fine-tuned sibling is listed on that model's card.
Citation
@misc{urdu_126m,
title = {urdu-126m: a 125.9M parameter Urdu language model trained from scratch},
author = {Abdul Wahab},
year = {2026},
url = {https://huggingface.co/awahab111/urdu-126m}
}