convnext-features / README.md
Flor's picture
card: update from fastforks 1cbf1671
9cc8589 verified
|
Raw History Blame Contribute Delete
8.83 kB
metadata
library_name: fastforks
pipeline_tag: image-feature-extraction
license: apache-2.0
tags:
  - features
  - backbone
  - feature-extraction
  - cuda
  - pytorch
  - fastforks
  - fastforks-card

ConvNeXt (fastforks)

ConvNeXt (Liu et al. 2022) as a feature backbone: Meta's ImageNet-22k checkpoints in timm's format (tiny, small, base, large). last_hidden_state = the stride-32 stage-4 map as tokens (no prefix token, un-normalised, as timm's forward_features / features_only and HF ConvNextModel); pooler_output = the head LayerNorm of their mean (timm's pre-logits embedding).

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/pytorch-image-models at 0df212b369a5.

Author weight repos: https://huggingface.co/timm/convnext_base.fb_in22k, https://huggingface.co/timm/convnext_large.fb_in22k, https://huggingface.co/timm/convnext_small.fb_in22k, https://huggingface.co/timm/convnext_tiny.fb_in22k.

Task: features. Tier: experimental. Outputs: last_hidden_state, pooler_output. Backends: portable, kernels.

Input: one RGB image. Output: dense patch features and a pooled image feature.

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
convnext_base.fb_in22k@1 convnext_base.fb_in22k.safetensors 0f09df05894c Apache-2.0 yes
convnext_large.fb_in22k@1 convnext_large.fb_in22k.safetensors 7e02ea57d602 Apache-2.0 yes
convnext_small.fb_in22k@1 convnext_small.fb_in22k.safetensors 73a192db2a11 Apache-2.0 yes
convnext_tiny.fb_in22k@1 convnext_tiny.fb_in22k.safetensors 2b40f8d53b56 Apache-2.0 yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.

Training data: ImageNet-21k / ImageNet-22k (full ImageNet, fall 2011 release).

Licence evidence: https://huggingface.co/timm/convnext_base.fb_in22k/blob/9afd4146adabf1b6e861d0f1ac2e7baa7c8dd1b7/README.md

Input size

any H x W >= 1, resized or padded inside the model Tested up to (portable, fp16, RTX 5060 Ti 16 GB): convnext_base.fb_in22k 3000x4000 (2.5 GiB peak allocated, 3.6 reserved); convnext_large.fb_in22k 3000x4000 (3.8 GiB peak allocated, 5.2 reserved); convnext_small.fb_in22k 3000x4000 (1.9 GiB peak allocated, 2.5 reserved); convnext_tiny.fb_in22k 3000x4000 (1.9 GiB peak allocated, 2.4 reserved)

Speed and memory

Features, synthetic, 224x224

model time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
convnext_base.fb_in22k 7.68 11.2 5.57 1.96 3.9x 2.8x 537 / 544 289 / 296 204 / 216
convnext_large.fb_in22k 10.5 11.5 5.53 3.23 3.3x 1.7x 1160 / 1180 600 / 612 413 / 434
convnext_small.fb_in22k 7.51 11.2 5.64 1.79 4.2x 3.2x 318 / 336 179 / 190 130 / 150
convnext_tiny.fb_in22k 3.89 5.91 3.18 1.09 3.6x 2.9x 194 / 214 117 / 130 88 / 110

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
convnext_base.fb_in22k 0.90 2.18 2.05 0.02 0.24 0.24
convnext_large.fb_in22k 2.17 2.71 2.78 0.02 0.24 0.23
convnext_small.fb_in22k 0.65 1.93 1.73 0.01 0.21 0.18
convnext_tiny.fb_in22k 0.40 1.66 1.57 0.28 0.23 0.22

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
convnext_base.fb_in22k 224x224 5.57 ms 3.21 ms 1.51 ms 1.96 ms 10.2 s 29.9 s 289 / 296 321 / 334 204 / 216 480 / 494
convnext_large.fb_in22k 224x224 5.53 ms 3.34 ms 2.50 ms 3.23 ms 9.87 s 46.7 s 600 / 612 639 / 650 413 / 434 983 / 1006
convnext_small.fb_in22k 224x224 5.64 ms 3.36 ms 1.16 ms 1.79 ms 9.51 s 35.1 s 179 / 190 210 / 230 130 / 150 277 / 292
convnext_tiny.fb_in22k 224x224 3.18 ms 2.08 ms 0.726 ms 1.09 ms 12.0 s 33.9 s 117 / 130 148 / 170 88 / 110 183 / 200

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

Two disjoint seeded subsets of the val folders; each image shorter side 256 (bicubic), centre crop 224, the CLS / pooled feature; cosine kNN, vote weighted by similarity. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable kernels note
convnext_tiny.fb_in22k ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 66.80 66.88 66.82 66.84
convnext_small.fb_in22k ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 71.68 71.68 71.72 71.72
convnext_base.fb_in22k ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 73.04 73.04 73.08 73.04
convnext_large.fb_in22k ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 75.26 75.22 75.28 75.24

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8b2f2df5, 2026-11-10.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 8.1e-05 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 1.6 % and p99 <= 9.4 % of the reference's median |value| (absolute error of the output last_hidden_state).

Limitations

  • Only the last stage's map is returned: the maps of stages 1-3 (strides 4, 8, 16; timm features_only=True) have other sizes than the token grid of this task's outputs.
  • kernels replays a CUDA graph per small input shape (up to 2^20 input pixels per call), captured on the shape's second call, at most 4 per model; model.release_graphs() frees them.

Usage

from fastforks import features

model = features.load("convnext_base.fb_in22k")  # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg", size=518)
result.last_hidden_state  # tokens [N, C], as Hugging Face's output
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: convnext_base.fb_in22k, convnext_large.fb_in22k, convnext_small.fb_in22k, convnext_tiny.fb_in22k.

Kernels backend

features.load("convnext_base.fb_in22k", mode="kernels") runs our CUDA kernels (fvm_conv, fvm_seg). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.