| ---
|
| library_name: transformers
|
| license: apache-2.0
|
| pipeline_tag: fill-mask
|
| language:
|
| - ur
|
| - ps
|
| - pa
|
| - sd
|
| - skr
|
| - ks
|
| - gu
|
| tags:
|
| - pakmosaic
|
| - pakistan
|
| - multilingual
|
| - encoder
|
| - research-preview
|
| - sota-target
|
| - fill-mask
|
| ---
|
|
|
| # PakMosaic-Small
|
|
|
| This is the finished ~67M PakMosaic encoder after a **2B-token** serious train on the clean scale mix.
|
|
|
| ## Architecture
|
|
|
| Encoder-only MLM: pre-norm, RoPE, GeGLU, full attention, tied embeddings.
|
|
|
| | | |
|
| | --- | ---: |
|
| | hidden size | 512 |
|
| | layers | 12 |
|
| | heads | 8 |
|
| | intermediate | 2048 |
|
| | vocab | 32,000 |
|
|
|
|
|
| ## How to load
|
|
|
| ```python
|
| from transformers import AutoModelForMaskedLM, AutoTokenizer, pipeline
|
|
|
| repo = "ProximaAI/PakMosaic-Small"
|
| tok = AutoTokenizer.from_pretrained(repo)
|
| model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True)
|
|
|
| fill = pipeline("fill-mask", model=model, tokenizer=tok)
|
| print(fill("یہ ایک <mask> ہے۔"))
|
| ```
|
|
|
| `AutoModel` also works (`trust_remote_code=True`) and returns encoder hidden states.
|
|
|
|
|
| ## Data
|
|
|
| Training mix is Wikimedia plus third-party web crawls used under their original terms. **Only Wikimedia / CC BY-SA is redistributed on the Hub:** [PakMosaic-Wikimedia-v0.2](https://huggingface.co/datasets/ProximaAI/PakMosaic-Wikimedia-v0.2).
|
|
|
|
|
| ## Citation
|
|
|
|
|
| ## Acknowledgements
|
|
|
| Wikipedia volunteer editors, native reviewers, and [Proxima AI](https://huggingface.co/ProximaAI).
|
|
|
