# HParams

Over the years, many `timm` models have been trained with various hyper-parameters as the libraries and models evolved. I don't have a record of every instance, but have recorded instances of many that can serve as a very good starting point.

Start with the [config file gists](#config-file-gists) for recorded runs, or the
[selected recipes](#selected-recipes) below for configs you can copy into a YAML file.
The [training script examples](training_script#imagenet-training-recipes) show their command line equivalents.

## Tags

Most `timm` trained models have an identifier in their pretrained tag that relates them (roughly) to a family / version of hparams I've used over the years.

The recipe family is a hint, not a complete configuration. Match the full checkpoint name, training dataset, and
resolution when choosing a config. For example, `vit_medium_patch16_reg4_gap_256.sbb_in1k` was trained on ImageNet-1k,
while `.sbb_in12k_ft_in1k` identifies ImageNet-12k pretraining followed by ImageNet-1k fine-tuning.
Some tags encode a schedule and resolution instead of a family name: MobileNetV4's `e500_r256_in1k` belongs to the
`ra4` family even though `ra4` is absent from the tag.

| Tag(s) | Description | Optimizer | LR Schedule | Other Notes |
|--------|-------------|-----------|-------------|-------------|
| `a1h` | Based on [ResNet Strikes Back](https://arxiv.org/abs/2110.00476) `A1` recipe | LAMB | Cosine with warmup | Stronger dropout, stochastic depth, and RandAugment than paper `A1` recipe |
| `ah` | Based on [ResNet Strikes Back](https://arxiv.org/abs/2110.00476) `A1` recipe | LAMB | Cosine with warmup | No CutMix. Stronger dropout, stochastic depth, and RandAugment than paper `A1` recipe |
| `a1`, `a2`, `a3` | ResNet Strikes Back `A{1,2,3}` recipe | LAMB with BCE loss | Cosine with warmup | — |
| `b1`, `b2`, `b1k`, `b2k` | Based on [ResNet Strikes Back](https://arxiv.org/abs/2110.00476) `B` recipe (equivalent to `timm` `RA2` recipes) | RMSProp (TF 1.0 behaviour) | Step (exponential decay w/ staircase) with warmup | — |
| `c`, `c1`, `c2`, `c3` | Based on [ResNet Strikes Back](https://arxiv.org/abs/2110.00476) `C` recipes | SGD (Nesterov) with AGC | Cosine with warmup | — |
| `ch` | Based on [ResNet Strikes Back](https://arxiv.org/abs/2110.00476) `C` recipes | SGD (Nesterov) with AGC | Cosine with warmup | Stronger dropout, stochastic depth, and RandAugment than paper `C1`/`C2` recipes |
| `d`, `d1`, `d2` | Based on [ResNet Strikes Back](https://arxiv.org/abs/2110.00476) `D` recipe | AdamW with BCE loss | Cosine with warmup | — |
| `sw` | DeiT / Swin-style train/pretrain recipes, also used for ConvNeXt and hybrid models | AdamW, often with gradient clipping | Cosine with warmup | EMA varies by run |
| `sbb` | [Searching for Better ViT Baselines](https://huggingface.co/blog/rwightman/vit-sbb-imagenet-full) | AdamW or NAdamW | Cosine with warmup | Repeated augmentation; layer decay for fine-tuning |
| `ra`, `ra2`, `ra3`, `racm`, `raa` | RandAugment recipes. Inspired by EfficientNet RandAugment recipes. Covered by `B` recipe in [ResNet Strikes Back](https://arxiv.org/abs/2110.00476). | RMSProp (TF 1.0 behaviour), EMA | Step (exponential decay w/ staircase) with warmup | — |
| `ra4` | RandAugment v4. Inspired by MobileNetV4 hparams. | AdamW or NAdamW, EMA | Cosine with warmup | Includes long-schedule ResNet recipes; optimizer betas vary by model |
| `am` | AugMix recipe | SGD (Nesterov) with JSD loss | Cosine with warmup | — |
| `ram` | AugMix (with RandAugment) recipe | SGD (Nesterov) with JSD loss | Cosine with warmup | — |
| `bt` | Bag-of-Tricks recipe | SGD (Nesterov) | Cosine with warmup | — |

## Config File Gists

These collections contain the recorded YAML configs, including settings specific to the original machines and
training script version. Open a gist, choose a model's YAML file, and use **Raw** to save it locally.

| Gist collection | Recipe family | Contents and original setup |
|-----------------|---------------|-----------------------------|
| [MobileNetV4 and long-schedule ResNets](https://gist.github.com/rwightman/f6705cb65c03daeebca8aa129b1b94ad) | `ra4` | Conv and hybrid MobileNetV4 runs; GPU counts are in filenames. Also includes 3600-epoch ResNet configs. |
| [Searching for Better ViT Baselines](https://gist.github.com/rwightman/fb37c339efd2334177ff99a8083ebbc4) | `sbb` | ImageNet-1k training, ImageNet-12k pretraining, and fine-tuning; 2 or 4 GPUs as named in each file. |
| [ConvNeXt training and fine-tuning](https://gist.github.com/rwightman/ee0b02c1e99a0761264d1d1319e95e5b) | ConvNeXt recipes | Atto/Nano training on 4 GPUs; Tiny/Small ImageNet-12k to ImageNet-1k fine-tuning on 8 GPUs. |
| [ViT, ConvNeXt, CoAtNet, and MaxViT](https://gist.github.com/rwightman/943c0fe59293b44024bbd2d5d23e6303) | DeiT / Swin-style | Mixed training and fine-tuning configs; 4–8 GPU or TPU devices. Many use TFDS. |
| [ResNet-D and ECA-ResNet RA2](https://gist.github.com/rwightman/07839a82d0f50e42840168bc43df70b3) | `ra2` | ResNet-18D/34D/50D and ECA-ResNet-26D; 2 GPUs, explicit learning rates. |
| [RegNet and RegNetZ](https://gist.github.com/rwightman/37252f8d7d850a94e43f1fcb7b3b8322) | `ra3`, SGD + AGC | 8 TPU cores. Some files are templates with `model: xxx`; the README notes a recipe/checkpoint mismatch for RegNetY-064. |
| [NFNet with adaptive gradient clipping](https://gist.github.com/rwightman/e69d5f456047c16773a77182cea68c3c) | SGD + AGC | Historical ECA-NFNet-L1 config in `nfnet.yaml`; uses Apex `fusedsgd`. |
| [MLP models on TPU](https://gist.github.com/rwightman/d6c264a9001f9167e06c209f630b2cc6) | AdamW | Historical gMLP, gMixer, and ResMLP runs tied to the experimental `bits_and_tpu` branch. |
| [Tiny test models](https://gist.github.com/rwightman/9ba8efc39a546426e99055720d2f705f) | MobileNetV4-derived | Small test ViTs and ConvNeXts with 1600–1800 epoch schedules. |

### Adapting a recorded config

Pass a local file with `--config hparams.yaml`; command line arguments override its values. See
[config files](training_script#config-files) for details.

* Set `--data-dir` for your machine. For an ImageFolder dataset, also override a saved TFDS dataset with
  `--dataset imagefolder/`. Training expects class subfolders under `train/` and `validation/`
  (or choose the validation directory with `--val-split`).
* Check `model`, `pretrained`, `num_classes`, and any checkpoint paths. A fine-tuning config needs its original
  pretrained checkpoint; an architecture name alone may select different default weights today.
* Old files can contain removed options such as `apex_amp`, `native_amp`, `amp_impl`, or `force_cpu`.
  Current `train.py` warns about unknown keys. The inline configs below use current options.
  Old `fusedadamw` / `fusedsgd` optimizer names refer to Apex implementations; switching to `adamw` / `sgd`
  is an adaptation of that run.

For fixed-size training, effective global batch size is:

```text
batch_size × number of training processes × grad_accum_steps
```

`batch_size` is per process (normally one GPU). A nonzero `lr` is used directly and is **not** automatically
rescaled. With `lr` unset (`null` in YAML), `lr_base` is scaled relative to `lr_base_size`.
Set `lr_base_scale: sqrt` for the AdamW/NAdamW recipes below; linear scaling is commonly used for SGD/RMSProp.
Large batch-size changes still need tuning. Gradient accumulation preserves the effective batch size but does
not enlarge the batch seen by BatchNorm.

For cosine schedules, `warmup_prefix: true` adds warmup before the `epochs` schedule;
`cooldown_epochs` adds a final period at the minimum LR. Keep these settings when adapting an old run.

## Selected recipes

These examples retain the training settings from specific gist files and omit machine paths and unused options.
Model tags pin the preprocessing configuration; weights are loaded only when `pretrained: true`.
The reference accuracies below are evaluations of the published checkpoints from
[results-imagenet.csv](https://github.com/huggingface/pytorch-image-models/blob/main/results/results-imagenet.csv),
not measurements from rerunning these examples. Resolution and validation crop matter when comparing results.

| Published checkpoint | Training | Evaluation size / crop | ImageNet-1k top-1 |
|----------------------|----------|------------------------|------------------|
| [mobilenetv4_conv_medium.e500_r256_in1k](https://huggingface.co/timm/mobilenetv4_conv_medium.e500_r256_in1k) | ImageNet-1k, 256px | 256 / 0.95; 320 / 1.0 | 79.916%; 80.890% |
| [vit_medium_patch16_reg4_gap_256.sbb_in1k](https://huggingface.co/timm/vit_medium_patch16_reg4_gap_256.sbb_in1k) | ImageNet-1k, 256px | 256 / 0.95 | 83.466% |
| [convnext_tiny.in12k_ft_in1k](https://huggingface.co/timm/convnext_tiny.in12k_ft_in1k) | ImageNet-12k → ImageNet-1k, 224px | 224 / 0.95; 288 / 1.0 | 84.192%; 84.448% |

### MobileNetV4 Conv Medium

Source: [mnv4_cm_r256_e500_gpu2.yaml](https://gist.github.com/rwightman/f6705cb65c03daeebca8aa129b1b94ad#file-mnv4_cm_r256_e500_gpu2-yaml).
The original run used 2 GPUs × 512 images = 1024 global batch, with a fixed LR of 0.002.
The schedule is 20 warmup epochs followed by 500 cosine epochs (520 total). Save as `mobilenetv4.yaml`:

```yaml
model: mobilenetv4_conv_medium.e500_r256_in1k
pretrained: false
img_size: 256
crop_pct: 0.95
batch_size: 512
opt: adamw
lr: 0.002
weight_decay: 0.1
clip_grad: 5.0
sched: cosine
sched_on_updates: true
epochs: 500
warmup_epochs: 20
warmup_prefix: true
warmup_lr: 0.0
min_lr: 0.0
cooldown_epochs: 0
aa: rand-m8-inc1-mstd1.0
mixup: 0.2
mixup_prob: 0.8
smoothing: 0.1
reprob: 0.25
remode: pixel
drop: 0.2
drop_path: 0.1
model_ema: true
model_ema_decay: 0.9998
model_ema_warmup: true
dist_bn: reduce
amp: true
amp_dtype: bfloat16
torchcompile: inductor
```

This example uses bfloat16 AMP as in the source run; use `--amp-dtype float16` on GPUs without bfloat16 support.
`torchcompile` can be omitted when compilation is not wanted.

### ViT Medium with registers

Source: [train-vit_medium_reg4-4gpu-in1k.yaml](https://gist.github.com/rwightman/fb37c339efd2334177ff99a8083ebbc4#file-train-vit_medium_reg4-4gpu-in1k-yaml),
also available as the checkpoint's [train_hparams.yaml](https://huggingface.co/timm/vit_medium_patch16_reg4_gap_256.sbb_in1k/blob/main/train_hparams.yaml).
This SBB run used 4 GPUs × 256 images = 1024 global batch. Square-root scaling gives
`0.0008 × sqrt(1024 / 512) ≈ 0.001131`. Its 600 epochs include warmup, with 10 cooldown epochs added.
Save as `vit-medium.yaml`:

```yaml
model: vit_medium_patch16_reg4_gap_256.sbb_in1k
pretrained: false
model_kwargs:
  fix_init: true
head_init_scale: 0.0
img_size: 256
crop_pct: 0.95
batch_size: 256
opt: adamw
opt_eps: 1.0e-8
lr: null
lr_base: 0.0008
lr_base_size: 512
lr_base_scale: sqrt
weight_decay: 0.07
clip_grad: 5.0
sched: cosine
sched_on_updates: true
epochs: 600
warmup_epochs: 20
warmup_prefix: false
warmup_lr: 5.0e-7
min_lr: 5.0e-7
cooldown_epochs: 10
aa: rand-m7-inc1-mstd1.0-n3
aug_repeats: 3
color_jitter: 0.0
mixup: 0.8
cutmix: 1.0
smoothing: 0.1
reprob: 0.2
remode: pixel
drop_path: 0.2
model_ema: false
seed: 0
amp: true
amp_dtype: float16
torchcompile: inductor
```

The initialization settings (`fix_init` and a zero-initialized classifier) are part of this recipe.
This particular run used AdamW and no EMA; other SBB files use NAdamW or different pretraining settings.

### ConvNeXt Tiny fine-tuning

Source: [hparams_84_00-convnext_tiny.in12k-args.yaml](https://gist.github.com/rwightman/ee0b02c1e99a0761264d1d1319e95e5b#file-hparams_84_00-convnext_tiny-in12k-args-yaml).
Start from [convnext_tiny.in12k](https://huggingface.co/timm/convnext_tiny.in12k), replace its classifier with
1000 classes, and fine-tune on ImageNet-1k. The original run used 8 GPUs × 512 images = 4096 global batch and
a fixed LR of 0.0001. It has 75 epochs including warmup, plus 10 cooldown epochs.
The source evaluates with a 1.0 crop; the checkpoint's benchmark results above use the stated evaluation crops.
Save as `convnext-tiny-ft.yaml`:

```yaml
model: convnext_tiny.in12k
pretrained: true
num_classes: 1000
img_size: 224
crop_pct: 1.0
batch_size: 512
grad_checkpointing: true
opt: adamw
lr: 0.0001
weight_decay: 0.05
layer_decay: 0.75
clip_grad: 3.0
sched: cosine
sched_on_updates: false
epochs: 75
warmup_epochs: 10
warmup_prefix: false
warmup_lr: 1.0e-6
min_lr: 5.0e-7
cooldown_epochs: 10
lr_noise: [0.1, 1.0]
aa: rand-m8-inc1-mstd101
mixup: 0.8
cutmix: 1.0
smoothing: 0.1
reprob: 0.3
remode: pixel
drop_path: 0.1
model_ema: true
model_ema_decay: 0.9998
amp: true
amp_dtype: float16
```

For example, launch the saved MobileNetV4 config with:

```bash
torchrun --standalone --nproc_per_node=2 train.py \
  --config mobilenetv4.yaml --data-dir /data/imagenet --experiment mobilenetv4-cm
```

Use 4 processes for `vit-medium.yaml` or 8 for `convnext-tiny-ft.yaml` to match the original global batch sizes,
or adjust batch size, accumulation, and LR for your hardware.