# MLP-Mixer, ResMLP and gMLP

**Category:** MLP · **Code:**
[`timm/models/mlp_mixer.py`](https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/mlp_mixer.py) ·
**Architectures:** 15 · **Pretrained weights:** 19 (14 with ImageNet-1k results)

All-MLP architectures that alternate token mixing across patches with channel mixing: MLP-Mixer, ResMLP and gMLP (gating
with a spatial projection). The module docstring notes the gMLP implementation is not verified.

## Papers

* [MLP-Mixer: An all-MLP Architecture for Vision](https://arxiv.org/abs/2105.01601)
* [ResMLP: Feedforward networks for image classification with data-efficient training](https://arxiv.org/abs/2105.03404)
* [Pay Attention to MLPs](https://arxiv.org/abs/2105.08050)
* [ImageNet-21K Pretraining for the Masses](https://arxiv.org/abs/2104.10972)
* [Emerging Properties in Self-Supervised Vision Transformers](https://arxiv.org/abs/2104.14294)

## Usage

See the [Quickstart](../quickstart) for loading pretrained weights, image preprocessing and fine-tuning, and [Feature
Extraction](../feature_extraction) for using the models as backbones.

```py
>>> import timm
>>> model = timm.create_model('gmixer_24_224.ra3_in1k', pretrained=True)
>>> data_config = timm.data.resolve_model_data_config(model)
```

## Model API support

| Feature | `MlpMixer` |
|---|---|
| [`features_only=True` feature maps](../model_api#intermediate-features) | ✓ (strides 16, 16, 16) |
| [`forward_intermediates()`](../model_api#intermediate-features) | ✓ |
| [`prune_intermediate_layers()`](../model_api#intermediate-features) | ✓ |
| [Any input size, no changes needed](../model_api#input-size) | ✗ |
| [`set_input_size()`](../model_api#input-size) | ✗ |
| [`dynamic_img_size=True`](../model_api#input-size) | ✗ |
| [`set_grad_checkpointing()`](../model_api#training-hooks) | ✓ |
| [Layer-wise LR decay (`group_matcher()`)](../model_api#training-hooks) | ✓ |
| [`reset_classifier()`](../model_api#classifier) | ✓ |

✓ supported, ✓ (square) works for square sizes only, ✗ not supported, – not needed (the model already accepts any input
size). Checked by running each feature on a randomly initialized model (`MlpMixer` with `gmlp_ti16_224` at 224), other
architectures of the same class are expected to behave the same. See the [Model API](../model_api) page for details.

## Architectures

| Architecture | Params (M) | GMACs | Input size | Pretrained weights |
|---|---:|---:|---:|---:|
| `gmixer_12_224` | 12.70 | 2.67 | 224 | 0 |
| `gmixer_24_224` | 24.72 | 5.28 | 224 | 1 |
| `gmlp_b16_224` | 73.08 | 15.78 | 224 | 0 |
| `gmlp_s16_224` | 19.42 | 4.42 | 224 | 1 |
| `gmlp_ti16_224` | 5.87 | 1.34 | 224 | 0 |
| `mixer_b16_224` | 59.88 | 12.62 | 224 | 4 |
| `mixer_b32_224` | 60.29 | 3.24 | 224 | 0 |
| `mixer_l16_224` | 208.20 | 44.60 | 224 | 2 |
| `mixer_l32_224` | 206.94 | 11.27 | 224 | 0 |
| `mixer_s16_224` | 18.53 | 3.79 | 224 | 0 |
| `mixer_s32_224` | 19.10 | 1.00 | 224 | 0 |
| `resmlp_12_224` | 15.35 | 3.01 | 224 | 3 |
| `resmlp_24_224` | 30.02 | 5.96 | 224 | 3 |
| `resmlp_36_224` | 44.69 | 8.91 | 224 | 2 |
| `resmlp_big_24_224` | 129.14 | 100.23 | 224 | 3 |

GMACs and parameter counts are for the default input size and classifier of each architecture, from the [benchmark
results](https://github.com/huggingface/pytorch-image-models/tree/main/results).

## Pretrained weights

Top-1 is ImageNet-1k validation accuracy from
[`results-imagenet.csv`](https://github.com/huggingface/pytorch-image-models/blob/main/results/results-imagenet.csv) at
the listed eval size, - if the weight has no ImageNet-1k result (e.g. pretraining or image-text weights). Classes is the
size of the pretrained classifier, none for weights without one.

Weights are grouped by source: the paper or origin given in the pretrained config, else a Hub organization other than
`timm`, else the first part of the pretrained tag (`architecture.tag`). Sources with a single weight are listed under
other.

### `fb` tags

| Weight | Input size | Top-1 (%) | Eval size | Classes | License | Hub |
|---|---:|---:|---:|---:|---|---|
| `resmlp_12_224.fb_dino` | 224 | - | - | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_12_224.fb_dino) |
| `resmlp_12_224.fb_distilled_in1k` | 224 | 77.948 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_12_224.fb_distilled_in1k) |
| `resmlp_12_224.fb_in1k` | 224 | 76.660 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_12_224.fb_in1k) |
| `resmlp_24_224.fb_dino` | 224 | - | - | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_24_224.fb_dino) |
| `resmlp_24_224.fb_distilled_in1k` | 224 | 80.766 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_24_224.fb_distilled_in1k) |
| `resmlp_24_224.fb_in1k` | 224 | 79.380 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_24_224.fb_in1k) |
| `resmlp_36_224.fb_distilled_in1k` | 224 | 81.150 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_36_224.fb_distilled_in1k) |
| `resmlp_36_224.fb_in1k` | 224 | 79.772 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_36_224.fb_in1k) |
| `resmlp_big_24_224.fb_distilled_in1k` | 224 | 83.594 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_big_24_224.fb_distilled_in1k) |
| `resmlp_big_24_224.fb_in1k` | 224 | 81.034 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_big_24_224.fb_in1k) |
| `resmlp_big_24_224.fb_in22k_ft_in1k` | 224 | 84.414 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/resmlp_big_24_224.fb_in22k_ft_in1k) |

### `goog` tags

| Weight | Input size | Top-1 (%) | Eval size | Classes | License | Hub |
|---|---:|---:|---:|---:|---|---|
| `mixer_b16_224.goog_in21k` | 224 | - | - | 21843 | apache-2.0 | [timm](https://huggingface.co/timm/mixer_b16_224.goog_in21k) |
| `mixer_b16_224.goog_in21k_ft_in1k` | 224 | 76.616 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/mixer_b16_224.goog_in21k_ft_in1k) |
| `mixer_l16_224.goog_in21k` | 224 | - | - | 21843 | apache-2.0 | [timm](https://huggingface.co/timm/mixer_l16_224.goog_in21k) |
| `mixer_l16_224.goog_in21k_ft_in1k` | 224 | 72.084 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/mixer_l16_224.goog_in21k_ft_in1k) |

### `miil` tags

| Weight | Input size | Top-1 (%) | Eval size | Classes | License | Hub |
|---|---:|---:|---:|---:|---|---|
| `mixer_b16_224.miil_in21k` | 224 | - | - | 11221 | apache-2.0 | [timm](https://huggingface.co/timm/mixer_b16_224.miil_in21k) |
| `mixer_b16_224.miil_in21k_ft_in1k` | 224 | 82.314 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/mixer_b16_224.miil_in21k_ft_in1k) |

### `ra3` tags

| Weight | Input size | Top-1 (%) | Eval size | Classes | License | Hub |
|---|---:|---:|---:|---:|---|---|
| `gmixer_24_224.ra3_in1k` | 224 | 78.048 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/gmixer_24_224.ra3_in1k) |
| `gmlp_s16_224.ra3_in1k` | 224 | 79.652 | 224 | 1000 | apache-2.0 | [timm](https://huggingface.co/timm/gmlp_s16_224.ra3_in1k) |

## Citation

```bibtex
@article{tolstikhin2021,
  title={MLP-Mixer: An all-MLP Architecture for Vision},
  author={Tolstikhin, Ilya and Houlsby, Neil and Kolesnikov, Alexander and Beyer, Lucas and Zhai, Xiaohua and Unterthiner,
        Thomas and Yung, Jessica and Keysers, Daniel and Uszkoreit, Jakob and Lucic, Mario and Dosovitskiy, Alexey},
  journal={arXiv preprint arXiv:2105.01601},
  year={2021}
}

@misc{touvron2021resmlp,
      title={ResMLP: Feedforward networks for image classification with data-efficient training},
      author={Hugo Touvron and Piotr Bojanowski and Mathilde Caron and Matthieu Cord and Alaaeldin El-Nouby and
        Edouard Grave and Armand Joulin and Gabriel Synnaeve and Jakob Verbeek and Hervé Jégou},
      year={2021},
      eprint={2105.03404},
}

@misc{liu2021pay,
      title={Pay Attention to MLPs},
      author={Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le},
      year={2021},
      eprint={2105.08050},
}
```