| --- |
| license: apache-2.0 |
| datasets: |
| - HuggingFaceFW/fineweb |
| language: |
| - en |
| tags: |
| - pixel |
| - mae |
| - vit |
| - tokenization-free |
| --- |
| # MPIXEL |
|
|
| MPIXEL, introduced in [Scaling Multilingual Pretraining of Pixel Language Models](https://arxiv.org/abs/2610.29681), is a family of multilingual pixel language models, which operate on visually rendered text. Our main models, `mpixel-base` (86M parameters) and `mpixel-large` (307M parameters), are pretrained on sixteen languages that span ten writing systems. MPIXEL uses adaptive rendering: it renders text as grapheme bigrams by default, Chinese, Japanese and Korean text one character per patch, and Arabic and Hebrew text word by word. |
|
|
| This model is `pixel-base-adaptive`, which is monolingually pretrained on English. We pretrained this model to eliminate the effect of confounding factors such as data source and rendering. |
|
|
| ## Basic usage |
| To use our models, please first install our [`mpixel` package](https://github.com/ilkerkesen/mpixel). Here is an example of loading a pretrained model and encoding input text: |
|
|
| ```python |
| from transformers import AutoModel |
| import mpixel.models.auto_register # registers MPIXEL with the Auto classes |
| from mpixel.processing import MPIXELProcessor |
| |
| processor = MPIXELProcessor.from_pretrained("Team-PIXEL/pixel-base-adaptive") |
| model = AutoModel.from_pretrained("Team-PIXEL/pixel-base-adaptive") |
| |
| inputs = processor(["Hello world!", "The quick brown fox jumps over the lazy dog."]) |
| outputs = model(**inputs) |
| |
| print(outputs.last_hidden_state.shape) # torch.Size([2, 513, 768]) |
| ``` |
|
|
| For further information, please take a look at our repository: [https://github.com/ilkerkesen/mpixel](https://github.com/ilkerkesen/mpixel). |
|
|
| ## Citation |
|
|
| If you find MPIXEL beneficial for your research, please cite it, |
|
|
| ``` |
| @misc{kesen2026scalingmultilingualpretrainingpixel, |
| title={Scaling Multilingual Pretraining of Pixel Language Models}, |
| author={Ilker Kesen and Jenny Kunz and Kasra Malihi and Desmond Elliott}, |
| year={2026}, |
| eprint={2610.29681}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2610.29681}, |
| } |
| ``` |
|
|
