HParams

Over the years, many timm models have been trained with various hyper-parameters as the libraries and models evolved. I don’t have a record of every instance, but have recorded instances of many that can serve as a very good starting point.

Start with the config file gists for recorded runs, or the selected recipes below for configs you can copy into a YAML file. The training script examples show their command line equivalents.

Tags

Most timm trained models have an identifier in their pretrained tag that relates them (roughly) to a family / version of hparams I’ve used over the years.

The recipe family is a hint, not a complete configuration. Match the full checkpoint name, training dataset, and resolution when choosing a config. For example, vit_medium_patch16_reg4_gap_256.sbb_in1k was trained on ImageNet-1k, while .sbb_in12k_ft_in1k identifies ImageNet-12k pretraining followed by ImageNet-1k fine-tuning. Some tags encode a schedule and resolution instead of a family name: MobileNetV4’s e500_r256_in1k belongs to the ra4 family even though ra4 is absent from the tag.

Config File Gists

These collections contain the recorded YAML configs, including settings specific to the original machines and training script version. Open a gist, choose a model’s YAML file, and use Raw to save it locally.

Adapting a recorded config

Pass a local file with --config hparams.yaml; command line arguments override its values. See config files for details.

• Set --data-dir for your machine. For an ImageFolder dataset, also override a saved TFDS dataset with --dataset imagefolder/. Training expects class subfolders under train/ and validation/ (or choose the validation directory with --val-split).
• Check model, pretrained, num_classes, and any checkpoint paths. A fine-tuning config needs its original pretrained checkpoint; an architecture name alone may select different default weights today.
• Old files can contain removed options such as apex_amp, native_amp, amp_impl, or force_cpu. Current train.py warns about unknown keys. The inline configs below use current options. Old fusedadamw / fusedsgd optimizer names refer to Apex implementations; switching to adamw / sgd is an adaptation of that run.

For fixed-size training, effective global batch size is:

batch_size × number of training processes × grad_accum_steps

batch_size is per process (normally one GPU). A nonzero lr is used directly and is not automatically rescaled. With lr unset (null in YAML), lr_base is scaled relative to lr_base_size. Set lr_base_scale: sqrt for the AdamW/NAdamW recipes below; linear scaling is commonly used for SGD/RMSProp. Large batch-size changes still need tuning. Gradient accumulation preserves the effective batch size but does not enlarge the batch seen by BatchNorm.

For cosine schedules, warmup_prefix: true adds warmup before the epochs schedule; cooldown_epochs adds a final period at the minimum LR. Keep these settings when adapting an old run.

Selected recipes

These examples retain the training settings from specific gist files and omit machine paths and unused options. Model tags pin the preprocessing configuration; weights are loaded only when pretrained: true. The reference accuracies below are evaluations of the published checkpoints from results-imagenet.csv, not measurements from rerunning these examples. Resolution and validation crop matter when comparing results.

MobileNetV4 Conv Medium

Source: mnv4_cm_r256_e500_gpu2.yaml. The original run used 2 GPUs × 512 images = 1024 global batch, with a fixed LR of 0.002. The schedule is 20 warmup epochs followed by 500 cosine epochs (520 total). Save as mobilenetv4.yaml:

model: mobilenetv4_conv_medium.e500_r256_in1k
pretrained: false
img_size: 256
crop_pct: 0.95
batch_size: 512
opt: adamw
lr: 0.002
weight_decay: 0.1
clip_grad: 5.0
sched: cosine
sched_on_updates: true
epochs: 500
warmup_epochs: 20
warmup_prefix: true
warmup_lr: 0.0
min_lr: 0.0
cooldown_epochs: 0
aa: rand-m8-inc1-mstd1.0
mixup: 0.2
mixup_prob: 0.8
smoothing: 0.1
reprob: 0.25
remode: pixel
drop: 0.2
drop_path: 0.1
model_ema: true
model_ema_decay: 0.9998
model_ema_warmup: true
dist_bn: reduce
amp: true
amp_dtype: bfloat16
torchcompile: inductor

This example uses bfloat16 AMP as in the source run; use --amp-dtype float16 on GPUs without bfloat16 support. torchcompile can be omitted when compilation is not wanted.

ViT Medium with registers

Source: train-vit_medium_reg4-4gpu-in1k.yaml, also available as the checkpoint’s train_hparams.yaml. This SBB run used 4 GPUs × 256 images = 1024 global batch. Square-root scaling gives 0.0008 × sqrt(1024 / 512) ≈ 0.001131. Its 600 epochs include warmup, with 10 cooldown epochs added. Save as vit-medium.yaml:

model: vit_medium_patch16_reg4_gap_256.sbb_in1k
pretrained: false
model_kwargs:
  fix_init: true
head_init_scale: 0.0
img_size: 256
crop_pct: 0.95
batch_size: 256
opt: adamw
opt_eps: 1.0e-8
lr: null
lr_base: 0.0008
lr_base_size: 512
lr_base_scale: sqrt
weight_decay: 0.07
clip_grad: 5.0
sched: cosine
sched_on_updates: true
epochs: 600
warmup_epochs: 20
warmup_prefix: false
warmup_lr: 5.0e-7
min_lr: 5.0e-7
cooldown_epochs: 10
aa: rand-m7-inc1-mstd1.0-n3
aug_repeats: 3
color_jitter: 0.0
mixup: 0.8
cutmix: 1.0
smoothing: 0.1
reprob: 0.2
remode: pixel
drop_path: 0.2
model_ema: false
seed: 0
amp: true
amp_dtype: float16
torchcompile: inductor

The initialization settings (fix_init and a zero-initialized classifier) are part of this recipe. This particular run used AdamW and no EMA; other SBB files use NAdamW or different pretraining settings.

ConvNeXt Tiny fine-tuning

Source: hparams_84_00-convnext_tiny.in12k-args.yaml. Start from convnext_tiny.in12k, replace its classifier with 1000 classes, and fine-tune on ImageNet-1k. The original run used 8 GPUs × 512 images = 4096 global batch and a fixed LR of 0.0001. It has 75 epochs including warmup, plus 10 cooldown epochs. The source evaluates with a 1.0 crop; the checkpoint’s benchmark results above use the stated evaluation crops. Save as convnext-tiny-ft.yaml:

model: convnext_tiny.in12k
pretrained: true
num_classes: 1000
img_size: 224
crop_pct: 1.0
batch_size: 512
grad_checkpointing: true
opt: adamw
lr: 0.0001
weight_decay: 0.05
layer_decay: 0.75
clip_grad: 3.0
sched: cosine
sched_on_updates: false
epochs: 75
warmup_epochs: 10
warmup_prefix: false
warmup_lr: 1.0e-6
min_lr: 5.0e-7
cooldown_epochs: 10
lr_noise: [0.1, 1.0]
aa: rand-m8-inc1-mstd101
mixup: 0.8
cutmix: 1.0
smoothing: 0.1
reprob: 0.3
remode: pixel
drop_path: 0.1
model_ema: true
model_ema_decay: 0.9998
amp: true
amp_dtype: float16

For example, launch the saved MobileNetV4 config with:

torchrun --standalone --nproc_per_node=2 train.py \
  --config mobilenetv4.yaml --data-dir /data/imagenet --experiment mobilenetv4-cm

Use 4 processes for vit-medium.yaml or 8 for convnext-tiny-ft.yaml to match the original global batch sizes, or adjust batch size, accumulation, and LR for your hardware.