Upload folder using huggingface_hub

README.md

@@ -0,0 +1,63 @@
+---
+license: apache-2.0
+datasets:
+- HuggingFaceFW/fineweb
+- HuggingFaceFW/fineweb-2
+language:
+- en
+- uk
+- zh
+- hi
+- da
+- tr
+- fi
+- es
+- ko
+- ar
+- th
+- el
+- bn
+- vi
+- he
+- fa
+tags:
+- pixel
+- mae
+- vit
+- tokenization-free
+---
+# MPIXEL
+
+MPIXEL, introduced in [Scaling Multilingual Pretraining of Pixel Language Models](https://arxiv.org/abs/2610.29681), is a family of multilingual pixel language models, which operate on visually rendered text. Our main models, `mpixel-base` (86M parameters) and `mpixel-large` (307M parameters), are pretrained on sixteen languages that span ten writing systems. MPIXEL uses adaptive rendering: it renders text as grapheme bigrams by default, Chinese, Japanese and Korean text one character per patch, and Arabic and Hebrew text word by word.
+
+## Basic usage
+To use our models, please first install our [`mpixel` package](https://github.com/ilkerkesen/mpixel). Here is an example of loading a pretrained model and encoding input text:
+
+```python
+from transformers import AutoModel
+import mpixel.models.auto_register  # registers MPIXEL with the Auto classes
+from mpixel.processing import MPIXELProcessor
+processor = MPIXELProcessor.from_pretrained("Team-PIXEL/mpixel-large")
+model = AutoModel.from_pretrained("Team-PIXEL/mpixel-large")
+inputs = processor(["Hello world!", "The quick brown fox jumps over the lazy dog."])
+outputs = model(**inputs)
+print(outputs.last_hidden_state.shape)  # torch.Size([2, 513, 768])
+```
+
+For further information, please take a look at our repository: [https://github.com/ilkerkesen/mpixel](https://github.com/ilkerkesen/mpixel).
+
+## Citation
+
+If you find MPIXEL beneficial for your research, please cite it,
+
+```
+@misc{kesen2026scalingmultilingualpretrainingpixel,
+      title={Scaling Multilingual Pretraining of Pixel Language Models}, 
+      author={Ilker Kesen and Jenny Kunz and Kasra Malihi and Desmond Elliott},
+      year={2026},
+      eprint={2610.29681},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2610.29681}, 
+}
+```

config.json

@@ -0,0 +1,33 @@
+{
+  "architectures": [
+    "MPIXELForPreTraining"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "decoder_hidden_size": 512,
+  "decoder_intermediate_size": 2048,
+  "decoder_num_attention_heads": 16,
+  "decoder_num_hidden_layers": 8,
+  "dtype": "float32",
+  "head_init": "pytorch",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-12,
+  "mask_ratio": 0.25,
+  "max_num_patches": 512,
+  "model_type": "mpixel",
+  "norm_pix_loss": true,
+  "num_attention_heads": 16,
+  "num_channels": 3,
+  "num_hidden_layers": 24,
+  "patch_embedding_init": null,
+  "patch_size": 16,
+  "pooler_add_layer_norm": true,
+  "pooling_mode": "mean",
+  "qkv_bias": true,
+  "reference_compile": false,
+  "rope_theta": 160000.0,
+  "transformers_version": "4.57.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7acccd2baf890911dd334d2a7d64a2d1817df14d3ef9cc825273e5b76e41f732
+size 1317019656

preprocessor_config.json

@@ -0,0 +1,5 @@
+{
+  "patch_size": 16,
+  "max_num_patches": 512,
+  "font_size": 12
+}