CLIP (fastforks)

CLIP image towers (Radford et al. 2021): OpenAI, OpenCLIP (LAION-2B, DataComp-1B) and Apple DFN weights in timm's format; cls + patch tokens at any size, pooler_output = the post-LayerNorm cls token before the projection (QuickGELU for the OpenAI and DFN weights).

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/pytorch-image-models at 0df212b369a5.

Author weight repos: https://huggingface.co/timm/vit_base_patch16_clip_224.laion2b, https://huggingface.co/timm/vit_base_patch16_clip_224.openai, https://huggingface.co/timm/vit_huge_patch14_clip_224.dfn5b, https://huggingface.co/timm/vit_huge_patch14_clip_224.laion2b, https://huggingface.co/timm/vit_huge_patch14_clip_378.dfn5b, https://huggingface.co/timm/vit_large_patch14_clip_224.datacompxl, https://huggingface.co/timm/vit_large_patch14_clip_224.dfn2b, https://huggingface.co/timm/vit_large_patch14_clip_224.laion2b, https://huggingface.co/timm/vit_large_patch14_clip_224.openai, https://huggingface.co/timm/vit_large_patch14_clip_336.openai.

Task: features. Tier: stable. Outputs: last_hidden_state, pooler_output, hidden_states. Backends: portable, kernels.

Input: one RGB image. Output: dense patch features and a pooled image feature.

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
clip-b16-laion2b@1 vit_base_patch16_clip_224.laion2b.safetensors 35fcf2008962 MIT yes
clip-b16-openai@1 vit_base_patch16_clip_224.openai.safetensors 4b8699299b1e MIT yes
clip-h14-378-dfn5b@1 vit_huge_patch14_clip_378.dfn5b.safetensors 81d5bf47b916 LicenseRef-Apple-AMLR no
clip-h14-dfn5b@1 vit_huge_patch14_clip_224.dfn5b.safetensors 7ebd4bca4bae LicenseRef-Apple-AMLR no
clip-h14-laion2b@1 vit_huge_patch14_clip_224.laion2b.safetensors a18dfb0dedf0 MIT yes
clip-l14-336-openai@1 vit_large_patch14_clip_336.openai.safetensors 18012c780964 MIT yes
clip-l14-datacompxl@1 vit_large_patch14_clip_224.datacompxl.safetensors b63e3d1bc90a MIT yes
clip-l14-dfn2b@1 vit_large_patch14_clip_224.dfn2b.safetensors 7a20e7340404 LicenseRef-Apple-AMLR no
clip-l14-laion2b@1 vit_large_patch14_clip_224.laion2b.safetensors e7204b9868d1 MIT yes
clip-l14-openai@1 vit_large_patch14_clip_224.openai.safetensors 9ce2e8a8ebff MIT yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.

Training data: not stated by the authors.

Licence evidence: https://huggingface.co/timm/vit_large_patch14_clip_336.openai/blob/81e38efc4637de5023b10e75a7f9bd1c6fa6b010/README.md

Input size

any H x W >= 1, resized or padded inside the model Tested up to (portable, fp16, RTX 5060 Ti 16 GB): clip-b16-laion2b 3000x4000 (1.7 GiB peak allocated, 1.9 reserved); clip-b16-openai 3000x4000 (1.9 GiB peak allocated, 2.2 reserved); clip-h14-378-dfn5b 3000x4000 (4.5 GiB peak allocated, 5.0 reserved); clip-h14-dfn5b 3000x4000 (4.5 GiB peak allocated, 5.0 reserved); clip-h14-laion2b 3000x4000 (3.9 GiB peak allocated, 4.4 reserved); clip-l14-336-openai 3000x4000 (3.4 GiB peak allocated, 3.6 reserved); clip-l14-datacompxl 3000x4000 (2.9 GiB peak allocated, 3.2 reserved); clip-l14-dfn2b 3000x4000 (3.4 GiB peak allocated, 3.7 reserved); clip-l14-laion2b 3000x4000 (2.9 GiB peak allocated, 3.2 reserved); clip-l14-openai 3000x4000 (3.4 GiB peak allocated, 3.7 reserved)

Speed and memory

Features, synthetic, 224x224

model time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
clip-b16-laion2b 6.22 5.29 3.09 1.36 3.9x 2.3x 544 / 580 202 / 226 203 / 230
clip-b16-openai 6.28 5.65 3.35 1.36 4.2x 2.5x 544 / 580 204 / 226 203 / 250
clip-h14-378-dfn5b 47.9 19.5 11.4 8.24 2.4x 1.4x 3722 / 4038 1254 / 1288 1251 / 1312
clip-h14-dfn5b 47.5 19.4 11.4 8.54 2.3x 1.3x 3720 / 4038 1252 / 1268 1250 / 1292
clip-h14-laion2b 47.2 19.3 11.2 8.27 2.3x 1.4x 3720 / 4032 1249 / 1268 1250 / 1272
clip-l14-336-openai 22.1 10.8 6.16 4.76 2.3x 1.3x 1779 / 1814 625 / 646 622 / 668
clip-l14-datacompxl 21.6 10.1 5.99 4.76 2.1x 1.3x 1779 / 1812 621 / 666 622 / 670
clip-l14-dfn2b 22.0 10.6 6.16 4.76 2.2x 1.3x 1779 / 1812 623 / 666 622 / 690
clip-l14-laion2b 21.6 10.1 5.98 4.76 2.1x 1.3x 1779 / 1812 621 / 666 622 / 670
clip-l14-openai 22.0 10.8 6.16 4.76 2.3x 1.3x 1779 / 1812 623 / 666 622 / 690

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
clip-b16-laion2b 13.69 – – 0.16 0.06 0.05
clip-b16-openai 16.66 – – 0.09 0.07 0.06
clip-h14-378-dfn5b 78.39 – – 0.18 0.08 0.06
clip-h14-dfn5b 79.37 – – 0.18 0.09 0.06
clip-h14-laion2b 78.69 – – 0.18 0.07 0.07
clip-l14-336-openai 30.98 – – 0.17 0.06 0.03
clip-l14-datacompxl 39.60 – – 0.17 0.04 0.03
clip-l14-dfn2b 39.79 – – 0.17 0.06 0.04
clip-l14-laion2b 39.40 – – 0.17 0.04 0.03
clip-l14-openai 41.72 – – 0.07 0.06 0.03

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
clip-b16-laion2b 224x224 3.09 ms 2.41 ms 1.12 ms 1.36 ms 6.49 s 24.3 s 202 / 226 237 / 266 203 / 230 394 / 422
clip-b16-openai 224x224 3.35 ms 2.39 ms 1.11 ms 1.36 ms 6.37 s 14.6 s 204 / 226 237 / 266 203 / 250 394 / 422
clip-h14-378-dfn5b 224x224 11.4 ms 11.8 ms 7.85 ms 8.24 ms 13.6 s 58.9 s 1254 / 1288 1290 / 1328 1251 / 1312 2468 / 2510
clip-h14-dfn5b 224x224 11.4 ms 11.5 ms 7.76 ms 8.54 ms 13.0 s 36.3 s 1252 / 1268 1289 / 1310 1250 / 1292 2465 / 2490
clip-h14-laion2b 224x224 11.2 ms 11.6 ms 7.84 ms 8.27 ms 14.3 s 40.6 s 1249 / 1268 1289 / 1310 1250 / 1272 2467 / 2492
clip-l14-336-openai 224x224 6.16 ms 6.36 ms 4.25 ms 4.76 ms 10.7 s 33.1 s 625 / 646 658 / 686 622 / 668 1235 / 1264
clip-l14-datacompxl 224x224 5.99 ms 6.24 ms 4.19 ms 4.76 ms 10.5 s 32.1 s 621 / 666 658 / 686 622 / 670 1232 / 1282
clip-l14-dfn2b 224x224 6.16 ms 6.16 ms 4.14 ms 4.76 ms 11.4 s 26.6 s 623 / 666 658 / 686 622 / 690 1230 / 1280
clip-l14-laion2b 224x224 5.98 ms 6.19 ms 4.19 ms 4.76 ms 10.4 s 32.0 s 621 / 666 658 / 686 622 / 670 1232 / 1282
clip-l14-openai 224x224 6.16 ms 6.17 ms 4.14 ms 4.76 ms 10.2 s 23.0 s 623 / 666 658 / 686 622 / 690 1230 / 1280

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

Two disjoint seeded subsets of the val folders; each image shorter side 256 (bicubic), centre crop 224, the CLS / pooled feature; cosine kNN, vote weighted by similarity. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable kernels note
clip-b16-openai ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 51.98 52.00 52.00 52.00
clip-l14-openai ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 58.70 58.64 58.56 58.58
clip-l14-336-openai ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 58.82 58.72 58.80 58.86
clip-b16-laion2b ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 54.14 54.24 54.22 54.16
clip-l14-laion2b ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 59.30 59.30 59.36 59.32
clip-h14-laion2b ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 62.56 62.46 62.46 62.56
clip-l14-datacompxl ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 62.50 62.44 62.44 62.38
clip-l14-dfn2b ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 66.86 66.74 66.76 66.78
clip-h14-dfn5b ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 68.72 68.70 68.80 68.86
clip-h14-378-dfn5b ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 64.84 65.10 65.10 65.00

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8bce3316 / b2ffd65d, 2026-11-11 to 2026-11-12.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 9.9 % and p99 <= 87 % of the reference's median |value| (absolute error of the output last_hidden_state).

Limitations

  • kernels replays a CUDA graph per small input shape without hidden states (up to about 1M activations: ViT-B ~460 px, ViT-L ~300 px), captured on the shape's second call, at most 4 per model; model.release_graphs() frees them.
  • Small kernels calls (graph-sized, with or without layers) run GEMMs of at most 600 tokens on cuBLAS: faster in a graph replay (ViT-B/16 224 px: 1.26 against 1.52 ms), about 10 % slower eagerly on ViT-T / B; a replay equals the eager call bit for bit.
  • fp32_accumulate = true puts every GEMM on cuBLAS with fp32 accumulation (the portable precision; slower on GeForce).
  • fp16 error against the fp32 original is higher than for the other ViTs on L/14 DataComp (mean 2.8 % portable, 4.8 % kernels): one residual channel sits near 120, where fp16 steps are 0.06, about 10 % of the median activation. No overflow; float32 portable gives 0.3 %.
  • DFN weights: the relative error (mean 1.3-3.9 % portable, 3.9-6.5 % kernels) is measured against a median |output| of only 0.015-0.021 (other weights: 0.5-0.6) with values up to 0.7; fp32_accumulate = true lowers kernels to about 1.5 %.
  • Kernels at 224 x 224 peak above portable (B/16 257 vs 204 MB, L/14 814 vs 623 MB; original 544 / 1779 MB): the QuickGELU fold keeps a folded copy of each fc2 weight next to the original parameter, and the second call's CUDA-graph capture adds ~30 MB.

Usage

from fastforks import features

model = features.load("clip-b16-laion2b")  # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg", size=518)
result.last_hidden_state  # tokens [N, C], as Hugging Face's output
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: clip-b16-laion2b, clip-b16-openai, clip-h14-378-dfn5b, clip-h14-dfn5b, clip-h14-laion2b, clip-l14-336-openai, clip-l14-datacompxl, clip-l14-dfn2b, clip-l14-laion2b, clip-l14-openai.

Kernels backend

features.load("clip-b16-laion2b", mode="kernels") runs our CUDA kernels (cudnn_hacc, fvm_conv, fvm_extra, fvm_mb, fvm_r3). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including fastforks/clip