---
license: other
license_name: mistral-ecosystem-open-weight-license
license_link: LICENSE
pipeline_tag: image-feature-extraction
library_name: transformers
tags:
- vision
base_model:
- mistralai/Mistral-Large-4-1T-A52B
---

# PixtralViT-Large-4

This repository packages the Tower and Projector from
[Mistral Large 4](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B).

## Contents

| File | Tensors | What it holds |
|---|---|---|
| `model.safetensors` | 272 | Pixtral vision encoder, from the parent's `vision_encoder.*` |
| `projector.safetensors` | 4 | `pre_mm_projector_norm`, `patch_merger` and `vision_language_adapter` |
| `modeling_pixtral_large4.py` | | `PixtralVisionModel` subclass following the parent's numeric order |
| `projector.py`, `projector_config.json` | | Projector shapes and loader |
| `config.json`, `preprocessor_config.json` | | Vision configuration and `PixtralImageProcessor` settings |
| `examples/inference.py` | | Single-image extraction example |
| `export_manifest.json` | | Source pin, shard SHA256 and tensor name map |
| `LICENSE`, `LICENSE-CODE`, `NOTICE` | | Weight license, code license and attribution |

## Architecture

| Component | Details |
|---|---|
| Tower | 30 blocks, 2048 hidden, 16 heads of 128, SwiGLU 6144, RMSNorm (eps 1e-5), 14 px patches, axial 2D RoPE, no class token, no final norm |
| Token compression | 2x2 patch merge inside the Projector |
| Projector | `RMSNorm(2048)`, 2x2 patch merge to 8192, `Linear(8192, 2048)`, `Linear(2048, 6144)`, GELU, `Linear(6144, 6144)`, no biases |

At 448 pixels the bench reports 1,024 Tower tokens of width 2048, 256 merged tokens
of width 2048 and 256 projected tokens of width 6144.

## Usage

See [`examples/inference.py`](examples/inference.py) for image feature extraction.

## Validation

The release tests compare all 276 tensors with the pinned parent checkpoint using
`torch.equal`. Fixed-image Tower, merged and projected outputs also match the parent
implementation bit-for-bit in FP32 and BF16 at 448 and 1540 pixels.

## Reproduction

The [export script](https://github.com/immanuel-peter/vision-tower-bench/blob/main/scripts/export_mistral_large4_vision.py)
range-reads the 276 vision tensors from the shard the pinned index names, renames
them to HF Pixtral names and reorders each `wq` and `wk` head from interleaved
rotary pairs to rotate-half order. It writes the original BF16 tensors. The script
pins the parent revision.

## Credits

Mistral AI released the Mistral Large 4 weights and the mistral-inference reference code.

## License

The weights keep the [Mistral Ecosystem Open-Weight License](LICENSE) from the
source model. The Python files are MIT; see [LICENSE-CODE](LICENSE-CODE).