Add PakMosaic-Small research-preview weights, training summary, and diagnostic evals

README.md

@@ -0,0 +1,140 @@
+---
+library_name: pytorch
+license: apache-2.0
+pipeline_tag: fill-mask
+language:
+- ur
+- ps
+- pa
+- sd
+- skr
+- ks
+- gu
+tags:
+- pakmosaic
+- pakistan
+- multilingual
+- encoder
+- research-preview
+- sota-target
+- fill-mask
+---
+
+# PakMosaic-Small
+
+**Pakistan NLP Suite** · published by [Proxima AI](https://huggingface.co/ProximaAI)
+
+**Status:** research preview · **SOTA-target** · not a factual SOTA claim  
+`SOTA_VERIFIED = false`
+
+This is the finished **~67M** PakMosaic encoder after a **2B-token** serious train on the clean scale mix. It is not PakMosaic-Base-132M (that run is still in progress). It is not a published SOTA model.
+
+## Training result
+
+| item | value |
+| --- | --- |
+| run | `pakmosaic-small_20260925T090924Z` |
+| last step | 2,904,769 |
+| tokens seen | 2,000,000,068 / 2,000,000,000 |
+| last-500 mean MLM loss | 1.958 |
+| last-500 mean masked accuracy | 0.640 |
+| hardware | NVIDIA GeForce RTX 4060 8GB (solo) |
+| objective | masked language modeling, mask prob 0.15 |
+| seq length | 512 |
+| tokenizer | PakMosaic-Tokenizer-32k (`hf_byte_bpe_32k_identity`) |
+
+Last-batch loss jumps by language. The 500-step mean is the number to quote.
+
+## Diagnostic evals (not PakBench)
+
+These are engineering sanity checks from `python -m paknlp_registry encoder evaluate`. They are **not** frozen benchmark scores.
+
+| check | result |
+| --- | ---: |
+| Same Urdu sentence vs spacing/punctuation noise (cosine) | 0.760 |
+| Near paraphrase pair (cosine) | 0.900 |
+| Unrelated pair (cosine) | 0.689 |
+| Urdu vs Pashto mean-pool centroid cosine | 0.890 |
+
+Near > far is the expected direction. Centroid cosine is **not** a language-ID score.
+
+SOTA gate (required PakBench evidence): **failed closed**. Missing frozen evaluation, contamination certificate, multi-language task results, macros, and contemporary baselines. Do not write SOTA on this card until that gate returns true.
+
+## Architecture
+
+Encoder-only MLM: pre-norm, RoPE, GeGLU, full attention, tied embeddings.
+
+| | |
+| --- | ---: |
+| hidden size | 512 |
+| layers | 12 |
+| heads | 8 |
+| intermediate | 2048 |
+| vocab | 32,000 |
+
+This is a custom `PakEncoderForMLM` checkpoint (`model.safetensors` + `config.json`). It is **not** a drop-in `AutoModel` BERT.
+
+## How to load
+
+```python
+import json
+from pathlib import Path
+import torch
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+
+# Install the Pakistan NLP Suite package, then:
+from paknlp_registry.encoder.config import EncoderConfig, SamplingConfig, MaskingConfig, TrainConfig
+from paknlp_registry.encoder.model import PakEncoderForMLM
+
+repo = "ProximaAI/PakMosaic-Small"
+cfg_path = hf_hub_download(repo, "config.json")
+raw = json.loads(Path(cfg_path).read_text(encoding="utf-8"))
+raw.pop("algebraic_parameters", None)
+if isinstance(raw.get("sampling"), dict):
+    raw["sampling"] = SamplingConfig(**raw["sampling"])
+if isinstance(raw.get("masking"), dict):
+    raw["masking"] = MaskingConfig(**raw["masking"])
+if isinstance(raw.get("train"), dict):
+    raw["train"] = TrainConfig(**raw["train"])
+model = PakEncoderForMLM(EncoderConfig(**raw))
+weights = load_file(hf_hub_download(repo, "model.safetensors"))
+model.load_state_dict(weights)
+model.eval()
+```
+
+Tokenizer artifact: `hf_byte_bpe_32k_identity` (hash `0f55e47920211588e6985172cd71338c8aff6226fe912c9668055b44bacc7ce7`). Frozen Urdu fertility 1.250 vs XLM-R 1.382 is a **tokenizer** metric, not model SOTA.
+
+## Data
+
+Training mix is Wikimedia plus third-party web crawls used under their original terms. **Only Wikimedia / CC BY-SA is redistributed on the Hub:** [PakMosaic-Wikimedia-v0.2](https://huggingface.co/datasets/ProximaAI/PakMosaic-Wikimedia-v0.2).
+
+FineWeb2, HPLT 2.0, and Common Voice are **not** re-uploaded here. Get those from the original steward.
+
+Languages actually sampled in this run: Pashto, Urdu, Punjabi Shahmukhi, Saraiki, Sindhi, Kashmiri, Gujarati. 66 registered identities are not 66 trained languages.
+
+## Intended use
+
+Research on Pakistan-language representation learning and reproduction of the public Wikimedia subset.
+
+## Out of scope
+
+Production deployment, surveillance, claiming 66-language training, or calling this model SOTA.
+
+## Family
+
+| model | params | this release |
+| --- | ---: | --- |
+| PakMosaic-Micro | ~26.6M | earlier engineering run |
+| **PakMosaic-Small** | **~67M** | **this repo, 2B tokens** |
+| PakMosaic-Base-132M | ~132.5M | training, not published |
+
+Project landing: [ProximaAI/PakMosaic](https://huggingface.co/ProximaAI/PakMosaic)
+
+## Citation
+
+Cite the Pakistan NLP Suite / PakMosaic repository version until a paper exists.
+
+## Acknowledgements
+
+Wikipedia volunteer editors, native reviewers, and [Proxima AI](https://huggingface.co/ProximaAI).

config.json

@@ -0,0 +1,75 @@
+{
+  "name": "pakmosaic_small",
+  "hidden_size": 512,
+  "num_hidden_layers": 12,
+  "num_attention_heads": 8,
+  "intermediate_size": 2048,
+  "vocab_size": 32000,
+  "max_position_embeddings": 512,
+  "rope_theta": 10000.0,
+  "attention_dropout": 0.0,
+  "hidden_dropout": 0.0,
+  "ffn_activation": "geglu",
+  "attention_kind": "full",
+  "local_window": 128,
+  "global_every": 3,
+  "use_bias": false,
+  "tie_word_embeddings": true,
+  "pad_token_id": 0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "unk_token_id": 3,
+  "mask_token_id": 4,
+  "layer_norm_eps": 1e-05,
+  "initializer_range": 0.02,
+  "tokenizer_id": "hf_byte_bpe_32k_identity",
+  "public_name": "PakMosaic-Small",
+  "script_signal": "none",
+  "sampling": {
+    "kind": "temperature",
+    "temperature": 3.0,
+    "anneal_start_temperature": 5.0,
+    "anneal_end_temperature": 1.0,
+    "min_language_prob": 0.01,
+    "languages": [
+      "pus_Arab",
+      "urd_Arab",
+      "pnb_Arab",
+      "skr_Arab",
+      "snd_Arab",
+      "kas_Arab",
+      "guj_Gujr"
+    ]
+  },
+  "masking": {
+    "kind": "fixed",
+    "mask_prob": 0.15,
+    "scheduled_peak": 0.3,
+    "replace_prob": 0.8,
+    "random_prob": 0.1,
+    "keep_prob": 0.1
+  },
+  "train": {
+    "max_seq_len": 512,
+    "smoke_seq_len": 128,
+    "token_budget": 2000000000,
+    "micro_batch_size": 2,
+    "grad_accum": 8,
+    "learning_rate": 0.0003,
+    "weight_decay": 0.01,
+    "warmup_steps": 50,
+    "max_grad_norm": 1.0,
+    "seed": 42,
+    "log_every": 10,
+    "checkpoint_every": 20000,
+    "keep_last_checkpoints": 2,
+    "require_expensive_confirm": true
+  },
+  "algebraic_parameters": {
+    "embeddings": 16384000,
+    "layers": 50343936,
+    "mlm_head": 263168,
+    "total": 66991616,
+    "trainable": 66991616
+  }
+}

evals.json

@@ -0,0 +1,83 @@
+{
+  "public_name": "PakMosaic-Small",
+  "run_id": "pakmosaic-small_20260925T090924Z",
+  "step": 2904769,
+  "tokens_seen_checkpoint": 2000000068,
+  "parameter_count": {
+    "total": 67037952,
+    "trainable": 67037952
+  },
+  "training": {
+    "step": 2904769,
+    "tokens_seen": 2000000068,
+    "token_budget": 2000000000,
+    "last_loss": 3.107429265975952,
+    "last_masked_accuracy": 0.4583333432674408,
+    "mean_loss_last_500": 1.9579948586970568,
+    "mean_masked_accuracy_last_500": 0.6402293934822082,
+    "n": 500
+  },
+  "diagnostic_eval": {
+    "checkpoint": "/home/mahwiz/ai-workspace/mahwiz/pakmosaic/runs/pakmosaic-small_20260925T090924Z/final.pt",
+    "step": 2904769,
+    "perturbation": {
+      "label": "DIAGNOSTIC_NOT_PAKBENCH",
+      "kind": "same_sentence_perturbation",
+      "cosine": 0.7603878974914551,
+      "note": "Engineering sanity only. Not a semantic benchmark."
+    },
+    "urdu_pairs": {
+      "label": "DIAGNOSTIC_NOT_PAKBENCH",
+      "kind": "urdu_pair_sanity",
+      "pairs": [
+        {
+          "tag": "near",
+          "cosine": 0.9000482559204102
+        },
+        {
+          "tag": "far",
+          "cosine": 0.6890260577201843
+        }
+      ]
+    },
+    "language_centroids": {
+      "label": "DIAGNOSTIC_NOT_PAKBENCH",
+      "kind": "language_clustering_diagnostic",
+      "pairwise_cosine": {
+        "urd_Arab__pus_Arab": 0.8904119729995728
+      },
+      "note": "Untrained or barely-trained centroids are not language-ID scores."
+    },
+    "label": "DIAGNOSTIC_NOT_PAKBENCH"
+  },
+  "sota_gate": {
+    "SOTA_VERIFIED": false,
+    "checkpoint": "/home/mahwiz/ai-workspace/mahwiz/pakmosaic/runs/pakmosaic-small_20260925T090924Z/final.pt",
+    "missing_evidence": [
+      "frozen_evaluation",
+      "contamination_zero",
+      "multiple_pakistan_languages",
+      "contemporary_baselines",
+      "task_level_results",
+      "macro",
+      "low_resource_macro",
+      "worst_language",
+      "not_tokenizer_fertility_only"
+    ],
+    "public_label": "research preview",
+    "note": "A tokenizer fertility win is not model SOTA. Do not write SOTA on a model card until this gate returns true."
+  },
+  "SOTA_VERIFIED": false,
+  "label": "research preview",
+  "note": "Diagnostic evals are not PakBench. Weights are inference safetensors only.",
+  "safetensors": {
+    "path": "/home/mahwiz/ai-workspace/mahwiz/pakmosaic/release/hf_small_staging/model.safetensors",
+    "sha256": "209c1aa99ff759ffdad62f38467ff4a8945c7a94397a7440ab815ba9e47529b7"
+  },
+  "not_uploaded": [
+    "FineWeb2",
+    "HPLT",
+    "Common Voice",
+    "optimizer .pt"
+  ]
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:209c1aa99ff759ffdad62f38467ff4a8945c7a94397a7440ab815ba9e47529b7
+size 333698800

tokenizer/meta.json

@@ -0,0 +1,44 @@
+{
+  "id": "hf_byte_bpe_32k_identity",
+  "status": "trained",
+  "normalization": "identity",
+  "normalization_note": "identity leaves original code points unchanged; nfc recomposes canonical equivalents",
+  "training_manifest_hash": "1abe78dd7c9dc9f93ac65bce02fb0c5d8ea501c65c12d4f3d660024935174319",
+  "n_train_lines": 24683,
+  "config": {
+    "id": "hf_byte_bpe_32k_identity",
+    "family": "hf_byte_bpe",
+    "vocab_size": 32000,
+    "normalization": "identity",
+    "seed": 42
+  },
+  "embedding_cost": {
+    "vocab_size": 32000,
+    "parameters": {
+      "384": 12288000,
+      "768": 24576000,
+      "1024": 32768000
+    },
+    "fp32_bytes": {
+      "384": 49152000,
+      "768": 98304000,
+      "1024": 131072000
+    },
+    "bf16_bytes": {
+      "384": 24576000,
+      "768": 49152000,
+      "1024": 65536000
+    }
+  },
+  "artifact_bytes": 2884675,
+  "reload_reproduced_first_example": true,
+  "vocab_collapsed": false,
+  "vocab_collapse_note": null,
+  "family": "hf_byte_bpe",
+  "model_path": "tokenizer\\artifacts\\open_release\\hf_byte_bpe_32k_identity\\tokenizer.json",
+  "requested_vocab_size": 32000,
+  "actual_vocab_size": 32000,
+  "artifact_hash": "0f55e47920211588e6985172cd71338c8aff6226fe912c9668055b44bacc7ce7",
+  "byte_fallback": true,
+  "seed": 42
+}

training_summary.json

@@ -0,0 +1,10 @@
+{
+  "step": 2904769,
+  "tokens_seen": 2000000068,
+  "token_budget": 2000000000,
+  "last_loss": 3.107429265975952,
+  "last_masked_accuracy": 0.4583333432674408,
+  "mean_loss_last_500": 1.9579948586970568,
+  "mean_masked_accuracy_last_500": 0.6402293934822082,
+  "n": 500
+}