PakMosaic

PakMosaic is a Pakistan multilingual foundation encoder research project. This Hub page is the public landing card. Trained weights are not published here until a run finishes.

Family

model	params	when it is a serious train
PakMosaic-Micro	26,638,208 measured	engineering / debug
PakMosaic-Small	~67M	only after ≥2B verified tokens
PakMosaic-Base-132M	~132.5M	only after ≥5B verified tokens

Encoder-only MLM: pre-norm, RoPE, GeGLU, SDPA, tied embeddings, sequence length 512.

Tokenizer

• vocab 32,000 · identity normalization
• artifact hash 0f55e47920211588e6985172cd71338c8aff6226fe912c9668055b44bacc7ce7
• vocab hash 911860f93e6c2d26939a57621516fdb4a140a299a9a286f4d24f8571aa425f1c
• frozen Urdu fertility 1.250 vs XLM-R 1.382 is a tokenizer metric, not model SOTA

Public data on the Hub

PakMosaic-Wikimedia-v0.2 — Wikimedia / CC BY-SA only.

Not on this Hub product: FineWeb2, HPLT 2.0, Common Voice. Those stay with their original licenses and stewards.

Languages

66 identities are registered. That is not training coverage. Serious scale training uses Pashto, Urdu, Punjabi Shahmukhi, Saraiki, Sindhi, Kashmiri, and Gujarati. Frozen native review today is Urdu-only (pilot_frozen).

Benchmarks

SOTA_VERIFIED stays false until the PakBench sota-gate passes predefined multi-language, multi-task conditions. No fabricated Base scores.

Intended use

Research on Pakistan-language representation learning and reproduction of the Wikimedia subset.

Out of scope

Production deployment, surveillance, claiming 66-language training, or calling this model SOTA.

Reproducibility

Every local run writes runs/<run_id>/manifest.json. Training aborts on tokenizer-hash mismatch, holdout contamination, or NaN loss.

Citation

Cite the PakMosaic repository version until a paper exists.

Acknowledgements

Wikipedia volunteer editors, native reviewers.