coatnet / README.md
Flor's picture
card: update from fastforks 1cbf1671
eb4c1aa verified
|
Raw History Blame Contribute Delete
14.1 kB
metadata
library_name: fastforks
pipeline_tag: image-classification
license: apache-2.0
tags:
  - classification
  - image-classification
  - cuda
  - pytorch
  - fastforks
  - fastforks-card

CoAtNet (fastforks)

CoAtNet (Dai et al. 2021) in timm's rw configurations: MBConv stages followed by transformer stages with relative position bias.

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/pytorch-image-models at 0df212b369a5.

Author weight repos: https://huggingface.co/timm/coatnet_0_rw_224.sw_in1k, https://huggingface.co/timm/coatnet_1_rw_224.sw_in1k, https://huggingface.co/timm/coatnet_2_rw_224.sw_in12k_ft_in1k, https://huggingface.co/timm/coatnet_nano_rw_224.sw_in1k, https://huggingface.co/timm/coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k, https://huggingface.co/timm/coatnet_rmlp_1_rw_224.sw_in1k, https://huggingface.co/timm/coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k, https://huggingface.co/timm/coatnet_rmlp_2_rw_224.sw_in1k, https://huggingface.co/timm/coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k, https://huggingface.co/timm/coatnet_rmlp_nano_rw_224.sw_in1k.

Task: classification. Tier: experimental. Outputs: logits. Backends: portable.

Input: one RGB image. Output: ImageNet-1k class logits.

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
coatnet_0_rw_224.sw_in1k@1 coatnet_0_rw_224.sw_in1k.safetensors efe01c8c139a Apache-2.0 yes
coatnet_1_rw_224.sw_in1k@1 coatnet_1_rw_224.sw_in1k.safetensors ef617b9c8bfd Apache-2.0 yes
coatnet_2_rw_224.sw_in12k_ft_in1k@1 coatnet_2_rw_224.sw_in12k_ft_in1k.safetensors 0a7274e508fc Apache-2.0 yes
coatnet_nano_rw_224.sw_in1k@1 coatnet_nano_rw_224.sw_in1k.safetensors 555f9ea6286c Apache-2.0 yes
coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k@1 coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k.safetensors 12e9f7a97070 Apache-2.0 yes
coatnet_rmlp_1_rw_224.sw_in1k@1 coatnet_rmlp_1_rw_224.sw_in1k.safetensors 22cca9389ecf Apache-2.0 yes
coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k@1 coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k.safetensors 8558fd38c0c6 Apache-2.0 yes
coatnet_rmlp_2_rw_224.sw_in1k@1 coatnet_rmlp_2_rw_224.sw_in1k.safetensors 8509ce832ab6 Apache-2.0 yes
coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k@1 coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k.safetensors 91570c7af10f Apache-2.0 yes
coatnet_rmlp_nano_rw_224.sw_in1k@1 coatnet_rmlp_nano_rw_224.sw_in1k.safetensors 2cabe244f520 Apache-2.0 yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.

Training data: ImageNet-1k (ILSVRC 2012 classification).

Licence evidence: https://huggingface.co/timm/coatnet_nano_rw_224.sw_in1k/blob/89fb9ccadc876a973c6ff954b089eb6723174c7a/README.md

Input size

  • coatnet_0_rw_224.sw_in1k, coatnet_1_rw_224.sw_in1k, coatnet_2_rw_224.sw_in12k_ft_in1k, coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k, coatnet_rmlp_1_rw_224.sw_in1k, coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k, coatnet_rmlp_2_rw_224.sw_in1k: fixed 224x224: the input is resized (short side 235) and centre-cropped to it
  • coatnet_nano_rw_224.sw_in1k, coatnet_rmlp_nano_rw_224.sw_in1k: fixed 224x224: the input is resized (short side 248) and centre-cropped to it
  • coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k: fixed 384x384: the input is resized (short side 384) and centre-cropped to it Tested up to (portable, fp16, RTX 5060 Ti 16 GB): coatnet_0_rw_224.sw_in1k 224x224 (0.1 GiB peak allocated, 0.1 reserved); coatnet_1_rw_224.sw_in1k 224x224 (0.1 GiB peak allocated, 0.2 reserved); coatnet_2_rw_224.sw_in12k_ft_in1k 224x224 (0.2 GiB peak allocated, 0.2 reserved); coatnet_nano_rw_224.sw_in1k 224x224 (0.1 GiB peak allocated, 0.1 reserved); coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k 224x224 (0.1 GiB peak allocated, 0.2 reserved); coatnet_rmlp_1_rw_224.sw_in1k 224x224 (0.1 GiB peak allocated, 0.2 reserved); coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k 224x224 (0.2 GiB peak allocated, 0.2 reserved); coatnet_rmlp_2_rw_224.sw_in1k 224x224 (0.2 GiB peak allocated, 0.2 reserved); coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k 384x384 (0.3 GiB peak allocated, 0.3 reserved); coatnet_rmlp_nano_rw_224.sw_in1k 224x224 (0.1 GiB peak allocated, 0.1 reserved)

Speed and memory

Classification, synthetic

model input time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
coatnet_0_rw_224.sw_in1k 224x224 7.64 9.59 6.49 – – – 194 / 210 102 / 124 –
coatnet_1_rw_224.sw_in1k 224x224 12.3 15.7 10.7 – – – 277 / 294 132 / 158 –
coatnet_2_rw_224.sw_in12k_ft_in1k 224x224 14.6 15.7 11.9 – – – 466 / 470 207 / 240 –
coatnet_nano_rw_224.sw_in1k 224x224 7.88 10.6 7.23 – – – 124 / 146 74 / 86 –
coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k 224x224 13.9 18.3 12.3 – – – 278 / 296 132 / 158 –
coatnet_rmlp_1_rw_224.sw_in1k 224x224 14.0 18.4 12.4 – – – 278 / 296 131 / 156 –
coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k 224x224 15.3 18.6 12.8 – – – 466 / 470 207 / 240 –
coatnet_rmlp_2_rw_224.sw_in1k 224x224 15.3 18.6 15.1 – – – 466 / 472 207 / 240 –
coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k 384x384 27.9 17.9 13.9 – – – 516 / 528 296 / 330 –
coatnet_rmlp_nano_rw_224.sw_in1k 224x224 8.95 12.1 8.09 – – – 124 / 146 78 / 106 –

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
coatnet_0_rw_224.sw_in1k 0.38 1.37 – 0.31 0.38 –
coatnet_1_rw_224.sw_in1k 0.47 1.44 – 0.15 0.37 –
coatnet_2_rw_224.sw_in12k_ft_in1k 0.74 1.52 – 0.31 0.51 –
coatnet_nano_rw_224.sw_in1k 0.28 1.37 – 0.54 0.42 –
coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k 0.47 1.49 – 0.02 0.37 –
coatnet_rmlp_1_rw_224.sw_in1k 0.47 1.47 – 0.02 0.37 –
coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k 0.71 1.80 – 0.02 0.50 –
coatnet_rmlp_2_rw_224.sw_in1k 0.71 1.54 – 0.02 0.51 –
coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k 0.74 (384x384) 1.52 (384x384) – 0.05 (384x384) 0.43 (384x384) –
coatnet_rmlp_nano_rw_224.sw_in1k 0.26 1.36 – 0.19 0.39 –

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
coatnet_0_rw_224.sw_in1k 224x224 6.49 ms 3.59 ms 0.849 ms – 12.1 s 38.2 s 102 / 124 120 / 142 – 158 / 170
coatnet_1_rw_224.sw_in1k 224x224 10.7 ms 5.58 ms 1.38 ms – 16.2 s 42.0 s 132 / 158 149 / 176 – 231 / 250
coatnet_2_rw_224.sw_in12k_ft_in1k 224x224 11.9 ms 5.53 ms 1.73 ms – 16.2 s 36.0 s 207 / 240 211 / 240 – 374 / 416
coatnet_nano_rw_224.sw_in1k 224x224 7.23 ms 4.71 ms 0.709 ms – 13.9 s 31.8 s 74 / 86 96 / 126 – 102 / 124
coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k 224x224 12.3 ms 6.63 ms 1.55 ms – 17.9 s 42.6 s 132 / 158 162 / 176 – 257 / 276
coatnet_rmlp_1_rw_224.sw_in1k 224x224 12.4 ms 6.54 ms 1.47 ms – 18.6 s 43.8 s 131 / 156 155 / 176 – 243 / 262
coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k 224x224 12.8 ms 6.52 ms 1.87 ms – 18.4 s 41.0 s 207 / 240 238 / 260 – 402 / 444
coatnet_rmlp_2_rw_224.sw_in1k 224x224 15.1 ms 6.34 ms 1.92 ms – 18.4 s 45.9 s 207 / 240 238 / 260 – 404 / 446
coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k 384x384 13.9 ms 6.92 ms 3.69 ms – 18.6 s 42.3 s 296 / 330 326 / 370 – 464 / 500
coatnet_rmlp_nano_rw_224.sw_in1k 224x224 8.09 ms 9.89 ms 0.868 ms – 13.9 s 37.1 s 78 / 106 110 / 126 – 122 / 144

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

fastforks: the model's own resize / crop on the GPU; original: the library's pretrained loader and its PIL eval transform. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable note
coatnet_nano_rw_224.sw_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 82.58 / 95.82 82.58 / 95.80 82.56 / 95.78
coatnet_rmlp_nano_rw_224.sw_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 82.48 / 95.98 82.54 / 95.98 82.44 / 95.96
coatnet_0_rw_224.sw_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 83.34 / 96.30 83.32 / 96.30 83.38 / 96.30
coatnet_1_rw_224.sw_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 84.32 / 96.54 84.38 / 96.56 84.44 / 96.56
coatnet_rmlp_1_rw_224.sw_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 84.58 / 96.74 84.58 / 96.74 84.46 / 96.56
coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 85.34 / 97.02 85.36 / 97.02 85.30 / 97.04
coatnet_2_rw_224.sw_in12k_ft_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 87.04 / 98.16 87.04 / 98.14 87.18 / 98.16
coatnet_rmlp_2_rw_224.sw_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 85.12 / 96.86 85.08 / 96.86 85.16 / 96.80
coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 87.04 / 98.18 87.02 / 98.18 87.02 / 98.12
coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k ImageNet-1k val, 5000 random images, top-1 / top-5 % 87.72 / 98.64 87.72 / 98.68 87.84 / 98.54

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 7ba7307f, 2026-11-11.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 8.6e-06 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.0024 and p99 <= 0.011 (absolute error of the output logits).

Usage

from fastforks import classification

model = classification.load("coatnet_0_rw_224.sw_in1k")  # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg")
result.top(5)  # top-5 classes
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: coatnet_0_rw_224.sw_in1k, coatnet_1_rw_224.sw_in1k, coatnet_2_rw_224.sw_in12k_ft_in1k, coatnet_nano_rw_224.sw_in1k, coatnet_rmlp_1_rw2_224.sw_in12k_ft_in1k, coatnet_rmlp_1_rw_224.sw_in1k, coatnet_rmlp_2_rw_224.sw_in12k_ft_in1k, coatnet_rmlp_2_rw_224.sw_in1k, coatnet_rmlp_2_rw_384.sw_in12k_ft_in1k, coatnet_rmlp_nano_rw_224.sw_in1k.

Kernels backend

No kernels backend: not implemented: MBConv stages plus attention with a relative position bias: needs its own runner.