timm documentation
HParams
You are viewing main version, which requires installation from source. If you'd like
regular pip install, checkout the latest stable version (v1.0.30).
HParams
Over the years, many timm models have been trained with various hyper-parameters as the libraries and models evolved. I don’t have a record of every instance, but have recorded instances of many that can serve as a very good starting point.
Tags
Most timm trained models have an identifier in their pretrained tag that relates them (roughly) to a family / version of hparams I’ve used over the years.
| Tag(s) | Description | Optimizer | LR Schedule | Other Notes |
|---|---|---|---|---|
a1h | Based on ResNet Strikes Back A1 recipe | LAMB | Cosine with warmup | Stronger dropout, stochastic depth, and RandAugment than paper A1 recipe |
ah | Based on ResNet Strikes Back A1 recipe | LAMB | Cosine with warmup | No CutMix. Stronger dropout, stochastic depth, and RandAugment than paper A1 recipe |
a1, a2, a3 | ResNet Strikes Back A{1,2,3} recipe | LAMB with BCE loss | Cosine with warmup | — |
b1, b2, b1k, b2k | Based on ResNet Strikes Back B recipe (equivalent to timm RA2 recipes) | RMSProp (TF 1.0 behaviour) | Step (exponential decay w/ staircase) with warmup | — |
c, c1, c2, c3 | Based on ResNet Strikes Back C recipes | SGD (Nesterov) with AGC | Cosine with warmup | — |
ch | Based on ResNet Strikes Back C recipes | SGD (Nesterov) with AGC | Cosine with warmup | Stronger dropout, stochastic depth, and RandAugment than paper C1/C2 recipes |
d, d1, d2 | Based on ResNet Strikes Back D recipe | AdamW with BCE loss | Cosine with warmup | — |
sw | Based on Swin Transformer train/pretrain recipe (basis of DeiT and ConvNeXt recipes) | AdamW with gradient clipping, EMA | Cosine with warmup | — |
ra, ra2, ra3, racm, raa | RandAugment recipes. Inspired by EfficientNet RandAugment recipes. Covered by B recipe in ResNet Strikes Back. | RMSProp (TF 1.0 behaviour), EMA | Step (exponential decay w/ staircase) with warmup | — |
ra4 | RandAugment v4. Inspired by MobileNetV4 hparams. | - | ||
am | AugMix recipe | SGD (Nesterov) with JSD loss | Cosine with warmup | — |
ram | AugMix (with RandAugment) recipe | SGD (Nesterov) with JSD loss | Cosine with warmup | — |
bt | Bag-of-Tricks recipe | SGD (Nesterov) | Cosine with warmup | — |
Config File Gists
I’ve collected several of the hparam families in a series of gists. These can be downloaded and used with the --config hparam.yaml argument with the timm train script. Some adjustment is always required for the LR vs effective global batch size.
| Tag | Key Model Architectures | Gist Link |
|---|---|---|
ra2 | ResNet, EfficientNet, RegNet, NFNet | Link |
ra3 | RegNet | Link |
ra4 | MobileNetV4 | Link |
sw | ViT, ConvNeXt, CoAtNet, MaxViT | Link |
sbb | ViT | Link |
| — | Tiny Test Models | Link |