urdu-126m: a 125.9M parameter Urdu language model trained from scratch

A decoder-only language model pretrained from random initialisation on Urdu text, with a 32,768-entry Urdu BPE tokenizer trained on the same corpus. No pretrained weights are used at any stage. It is a base model, so it continues text. It does not follow instructions.

This is not a transformers model. It uses the nanochat architecture, which includes value embeddings, a smear gate, per-layer residual lambdas and a logit soft cap. No llama.cpp or GGUF implementation of that architecture exists, so run it with nanochat's code, shown below.

Model family

Model Depth Transformer matrices Total params Tokens Val bpb UrduMMLU
urdu-126m 8 25.2M 126M 0.84B 0.6953 24.84%
urdu-286m 12 84.9M 286M 2.20B 0.6113 24.58%

Both models use the same corpus, the same tokenizer and the same recipe. Only the size changed. The larger model compresses held-out Urdu 12% better, and that is the whole scaling result.

Architecture

Layers 8
d_model 512
Query heads / KV heads 4 / 4, head dim 128
MLP 4x expansion, ReLU squared
Positions RoPE
Normalisation RMSNorm, pre-norm
Attention sliding window, pattern SSSL (quarter context on S layers)
Context 2,048 tokens
Vocab 32,768

Where the parameters live:

Component Parameters
Transformer matrices (attention + MLP) 25.2M
Unembedding head 16.8M
Token embedding 16.8M
Value embeddings 67.1M
Total stored 125.9M

More than half of this model is lookup tables rather than computation. The number to use when comparing sizes or reasoning about scaling is 42.0M, the matrices plus the head, because that is what does arithmetic on every token.

Tokenizer: the reason this project exists

Urdu is served badly by tokenizers built for English. Measured on 3,000 held-out Wikipedia documents (11.4M characters), the same text through two tokenizers:

Tokenizer Tokens per word Bytes per token
GPT-2 BPE 5.95 1.42
This model's 32k Urdu BPE 1.25 6.76

4.76x fewer tokens for identical text, so training and inference cost 4.76x less per document. The trade is real and runs the other way: this tokenizer spends 3 tokens on education where GPT-2 spends 1. Cheap for Urdu, expensive for English, by design. A bilingual model would need a mixed vocabulary instead.

Training data

Sources FineWeb-2 urd_Arab, Wikipedia 20231101.ur
Documents before filtering 4.86M web, 200,154 articles
Documents after cleaning 4,048,757
Tokens 2.85B in the corpus, 0.84B used for this model, one pass, no repetition
Normalisation Arabic yeh/kaf/heh folded to Urdu code points, diacritics and bidi controls stripped, digits normalised, URLs and markup removed
Quality gates language score, duplicate cluster size, length bounds, script ratio, symbol ratio, repetition guard
Deduplication exact, BLAKE2b, across both sources
Validation split Wikipedia only, excluded from training, so val bpb measures cross-source generalisation
Licence FineWeb-2 is ODC-By 1.0, Wikipedia text is CC BY-SA

Training recipe

Hardware one L40S on Modal, bf16
Steps 3,200
Batch 262,144 tokens, gradient accumulation
Wall clock 59 min
Compute cost $1.92
Optimiser Muon for matrices (lr 0.02, momentum 0.95), AdamW for embeddings (0.3) and scalars
Weight decay 0.28
Warmup / decay 40 steps warmup, 65% warmdown, final LR 5% of peak
Checkpoints every 200 steps, committed to a persistent volume
Final validation bpb 0.6953

Results

Validation bits per byte on the held-out Wikipedia split. Bits per byte is tokenizer-independent, which is why it is the metric here and not perplexity.

Step bpb
0 2.2648
250 0.9639
500 0.8535
1000 0.8089
1500 0.7783
2000 0.751
2500 0.722
3000 0.6965
3200 0.6953

Two caveats worth stating. Bits per byte compares models on the same text, not across languages, because UTF-8 Urdu spends two bytes per character and each byte therefore carries less information than an English byte. And a falling bpb means better compression of Urdu, not better reasoning.

Benchmarks: the size comparison

Five probes, both models, the same items, all built from the held-out split so neither trained on them. The paired test is McNemar's, which counts the items where only one model was right.

Metric urdu-126m urdu-286m Gap Test
HellaSwag-ur, 4-way continuation 57.05% 60.60% +3.55 pts McNemar, p = 9.9e-06
Final-word top-1 24.25% 25.45% +1.20 pts McNemar, p = 0.0025
Repeated 4-gram share, lower is better 7.95% 3.64% -4.31 pts descriptive
Distinct-2, higher is better 0.813 0.881 +0.068 descriptive
Bits per byte, whole held-out split 0.6953 0.6113 -12.1% -

Bits per byte by document length, where the smaller model falls further behind on short documents:

Document size urdu-126m urdu-286m Relative gain
under 2KB 0.9984 0.8159 18.3%
2 to 5KB 0.8339 0.7152 14.2%
5 to 10KB 0.7526 0.6668 11.4%
over 10KB 0.6870 0.6147 10.5%

HellaSwag-ur is built rather than borrowed: the context is the first three to five sentences of a held-out page, the true option is the sentence that actually follows, and the three distractors come from other documents. Each option is scored by its mean token log-likelihood given the context, which is exactly what a base model is trained to produce. Chance is 25%.

What the gap looks like in text. Prompted with مصنوعی ذہانت, the smaller model writes "Artificialicial Intelligence", a mis-spelling, while the larger one writes it correctly and adds the byline convention تحریر: علی احمدی. Fifteen prompts across registers, same seed, are side by side in SAMPLES_COMPARE.md in this repo.

What these numbers do not show: neither model has knowledge, and this does not make UrduMMLU meaningful for either of them. The benchmark can see the size difference because it tests the training objective, not because either model learned facts.

UrduMMLU

This model scores 24.84% on UrduMMLU's 26,431 questions, against a 25.0% random baseline. That number is not a measure of knowledge, and the reason is measurable.

The letter-prior analysis above, plus two further scoring protocols, all land at chance. Rescoring by option content removes the letter bias and does not help.

Protocol urdu-126m urdu-286m Chance
Answer letter probability, n = 26,431 24.84% 24.58% 25.0%
Option likelihood, n = 26,431 24.54% 22.71% 25.0%
Option likelihood, n = 5,000 subset 25.78% 25.14% 25.0%
Option PMI, n = 5,000 subset 24.34% 24.22% 25.0%

XNLI-ur, the standard cross-lingual NLI benchmark, 5,010 test pairs, both models: 33.33% against a 33.33% class prior, with 100% of predictions on a single label.

Raw option likelihood can land below chance because mean token likelihood prefers a generic phrase over a specific correct answer. PMI subtracts each option's own prior and does not rescue it either. At this scale there is no measurable school knowledge, and no multiple-choice protocol recovers any. A real knowledge number needs instruction tuning first.

Benchmark leakage, measured

Each UrduMMLU question contributes a 60-character fingerprint, normalised like the corpus, and the corpus was searched for those spans.

File Documents Documents containing a fingerprint
Wikipedia 165,039 1
FineWeb-2 Urdu 4,048,757 140

166 questions, 0.63% of the benchmark. The leaked spans are classical poetry that the web reproduces, and the affected subdomains, Urdu literature, Urdu grammar, Urdu language and general knowledge, all score at the chance line. Storing the text did not buy the ability to answer questions about it.

Samples

Seeded, temperature 0.8, top-k 50. The full sheet is in samples_126m.md.

Prompt کرکٹ میچ میں:

پاکستان کو شکست، آسٹریلیا نے سیریز میں برتری حاصل کرلی سڈنی: ٹی ٹوئنٹی ورلڈکپ کے اہم میچ میں آسٹریلیا پاکستان کو 10 وکٹوں سے شکست دے کر ورلڈکپ سے باہر ہوگیا، آسٹریلیا نے پاکستان کا 189 رنز کا ہدف 19ویں اوور میں صرف 3 وکٹوں کے نقصان پر حاصل کرلیا۔

The news register, dateline, overs and toss idiom are learned. The logic is not, because a team cannot win by ten wickets and leave the tournament in one sentence.

Prompt مصنوعی ذہانت:

(AI) کے ارد گرد بہت سے غلط فہمیاں ہیں، اور یہ کہنا درست نہیں ہوگا کہ آپ کو مصنوعی ذہانت کی ضرورت ہے۔ AI کی حقیقت میں ایک اور وسیع پیمانے پر تصویر ہے۔

How to run

git clone https://github.com/karpathy/nanochat
hf download awahab111/urdu-126m --local-dir ./urdu-126m
export NANOCHAT_BASE_DIR=$PWD/urdu-126m
python -m scripts.chat_cli    # asks for prompts, streams Urdu

Repository layout mirrors what nanochat expects:

base_checkpoints/urdu_126m/model_003200.pt
base_checkpoints/urdu_126m/meta_003200.json
tokenizer/tokenizer.pkl
tokenizer/token_bytes.pt

Load it directly if you prefer:

from nanochat.checkpoint_manager import load_model
model, tokenizer, meta = load_model("base", device, phase="eval",
                                    model_tag="urdu_126m")

Checkpoint size on disk is 321 MB. Measured 19.8 tokens/sec for the 286M model on a CPU-only box, so a GPU is optional for demos.

Limitations

It produces grammatical, register-correct Urdu, and it invents facts, numbers and causes without warning. It repeats itself and can contradict itself inside one paragraph. It is a research and portfolio artifact. Nothing it generates is advice or a factual source.

Corpus composition shows through. The Urdu web inside FineWeb-2 is heavily Indian Urdu news, so prompted with تعلیم کی اہمیت (the importance of education) this model produces Indian political parties and schemes.

There is no instruction tuning in these weights. There is no SFT stage, no chat template beyond the base token format, and no safety tuning of any kind.

Reproduce

The full pipeline, roughly 15 minutes of CPU work and about $13 of GPU time:

python scripts/build_fineweb.py --workers 3     # 4.86M web docs -> clean JSONL
python scripts/build_wiki_hf.py                  # 200k articles -> clean JSONL
python scripts/decontaminate.py                  # benchmark fingerprints -> skip list
python scripts/make_shards.py                    # dedup, hash-shard, val split
python -m scripts.tok_train --vocab-size 32768   # 32k Urdu BPE
python -m modal run modal_train.py --depth 8 --ratio 20 --tag urdu_126m
python -m modal run modal_train.py::evaluate --tag urdu_126m

License

Weights: CC-BY-NC-4.0. Free for research, evaluation and personal use, with attribution. Commercial use requires a licence, available on request at me@awahab.dev.

Training data carries its own terms underneath: FineWeb-2 is ODC-By 1.0, the Wikipedia extract is CC BY-SA 4.0, and the instruction data used for any fine-tuned sibling is listed on that model's card.

Citation

@misc{urdu_126m,
  title  = {urdu-126m: a 125.9M parameter Urdu language model trained from scratch},
  author = {Abdul Wahab},
  year   = {2026},
  url    = {https://huggingface.co/awahab111/urdu-126m}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train awahab111/urdu-126m