Download README.md from fastforks/clip: direct link, hf CLI and curl.
- Browser
- Download file 14.9 kB
-
https://huggingface.co/fastforks/clip/resolve/main/README.md
- Command line
-
hf download hf://fastforks/clip/README.md
-
curl -L -o README.md https://huggingface.co/fastforks/clip/resolve/main/README.md
library_name: fastforks
pipeline_tag: image-feature-extraction
license: other
license_name: mixed-see-variant-table
license_link: >-
https://huggingface.co/timm/vit_base_patch16_clip_224.laion2b/blob/fbadfdd1c0e903c58855b70ae04740b22da9d305/README.md
tags:
- features
- backbone
- feature-extraction
- cuda
- pytorch
- fastforks
- fastforks-card
CLIP (fastforks)
CLIP image towers (Radford et al. 2021): OpenAI, OpenCLIP (LAION-2B, DataComp-1B) and Apple DFN weights in timm's format; cls + patch tokens at any size, pooler_output = the post-LayerNorm cls token before the projection (QuickGELU for the OpenAI and DFN weights).
This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/pytorch-image-models at 0df212b369a5.
Author weight repos: https://huggingface.co/timm/vit_base_patch16_clip_224.laion2b, https://huggingface.co/timm/vit_base_patch16_clip_224.openai, https://huggingface.co/timm/vit_huge_patch14_clip_224.dfn5b, https://huggingface.co/timm/vit_huge_patch14_clip_224.laion2b, https://huggingface.co/timm/vit_huge_patch14_clip_378.dfn5b, https://huggingface.co/timm/vit_large_patch14_clip_224.datacompxl, https://huggingface.co/timm/vit_large_patch14_clip_224.dfn2b, https://huggingface.co/timm/vit_large_patch14_clip_224.laion2b, https://huggingface.co/timm/vit_large_patch14_clip_224.openai, https://huggingface.co/timm/vit_large_patch14_clip_336.openai.
Task: features. Tier: stable. Outputs: last_hidden_state, pooler_output, hidden_states. Backends: portable, kernels.
Input: one RGB image. Output: dense patch features and a pooled image feature.
Variants
Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.
| variant | weights (author link) | SHA-256 | weights licence | commercial use |
|---|---|---|---|---|
clip-b16-laion2b@1 |
vit_base_patch16_clip_224.laion2b.safetensors | 35fcf2008962 |
MIT | yes |
clip-b16-openai@1 |
vit_base_patch16_clip_224.openai.safetensors | 4b8699299b1e |
MIT | yes |
clip-h14-378-dfn5b@1 |
vit_huge_patch14_clip_378.dfn5b.safetensors | 81d5bf47b916 |
LicenseRef-Apple-AMLR | no |
clip-h14-dfn5b@1 |
vit_huge_patch14_clip_224.dfn5b.safetensors | 7ebd4bca4bae |
LicenseRef-Apple-AMLR | no |
clip-h14-laion2b@1 |
vit_huge_patch14_clip_224.laion2b.safetensors | a18dfb0dedf0 |
MIT | yes |
clip-l14-336-openai@1 |
vit_large_patch14_clip_336.openai.safetensors | 18012c780964 |
MIT | yes |
clip-l14-datacompxl@1 |
vit_large_patch14_clip_224.datacompxl.safetensors | b63e3d1bc90a |
MIT | yes |
clip-l14-dfn2b@1 |
vit_large_patch14_clip_224.dfn2b.safetensors | 7a20e7340404 |
LicenseRef-Apple-AMLR | no |
clip-l14-laion2b@1 |
vit_large_patch14_clip_224.laion2b.safetensors | e7204b9868d1 |
MIT | yes |
clip-l14-openai@1 |
vit_large_patch14_clip_224.openai.safetensors | 9ce2e8a8ebff |
MIT | yes |
Licence
Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.
Training data: not stated by the authors.
Licence evidence: https://huggingface.co/timm/vit_large_patch14_clip_336.openai/blob/81e38efc4637de5023b10e75a7f9bd1c6fa6b010/README.md
Input size
any H x W >= 1, resized or padded inside the model
Tested up to (portable, fp16, RTX 5060 Ti 16 GB): clip-b16-laion2b 3000x4000 (1.7 GiB peak allocated, 1.9 reserved); clip-b16-openai 3000x4000 (1.9 GiB peak allocated, 2.2 reserved); clip-h14-378-dfn5b 3000x4000 (4.5 GiB peak allocated, 5.0 reserved); clip-h14-dfn5b 3000x4000 (4.5 GiB peak allocated, 5.0 reserved); clip-h14-laion2b 3000x4000 (3.9 GiB peak allocated, 4.4 reserved); clip-l14-336-openai 3000x4000 (3.4 GiB peak allocated, 3.6 reserved); clip-l14-datacompxl 3000x4000 (2.9 GiB peak allocated, 3.2 reserved); clip-l14-dfn2b 3000x4000 (3.4 GiB peak allocated, 3.7 reserved); clip-l14-laion2b 3000x4000 (2.9 GiB peak allocated, 3.2 reserved); clip-l14-openai 3000x4000 (3.4 GiB peak allocated, 3.7 reserved)
Speed and memory
Features, synthetic, 224x224
| model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
|---|---|---|---|---|---|---|---|---|---|
clip-b16-laion2b |
6.22 | 5.29 | 3.09 | 1.36 | 3.9x | 2.3x | 544 / 580 | 202 / 226 | 203 / 230 |
clip-b16-openai |
6.28 | 5.65 | 3.35 | 1.36 | 4.2x | 2.5x | 544 / 580 | 204 / 226 | 203 / 250 |
clip-h14-378-dfn5b |
47.9 | 19.5 | 11.4 | 8.24 | 2.4x | 1.4x | 3722 / 4038 | 1254 / 1288 | 1251 / 1312 |
clip-h14-dfn5b |
47.5 | 19.4 | 11.4 | 8.54 | 2.3x | 1.3x | 3720 / 4038 | 1252 / 1268 | 1250 / 1292 |
clip-h14-laion2b |
47.2 | 19.3 | 11.2 | 8.27 | 2.3x | 1.4x | 3720 / 4032 | 1249 / 1268 | 1250 / 1272 |
clip-l14-336-openai |
22.1 | 10.8 | 6.16 | 4.76 | 2.3x | 1.3x | 1779 / 1814 | 625 / 646 | 622 / 668 |
clip-l14-datacompxl |
21.6 | 10.1 | 5.99 | 4.76 | 2.1x | 1.3x | 1779 / 1812 | 621 / 666 | 622 / 670 |
clip-l14-dfn2b |
22.0 | 10.6 | 6.16 | 4.76 | 2.2x | 1.3x | 1779 / 1812 | 623 / 666 | 622 / 690 |
clip-l14-laion2b |
21.6 | 10.1 | 5.98 | 4.76 | 2.1x | 1.3x | 1779 / 1812 | 621 / 666 | 622 / 670 |
clip-l14-openai |
22.0 | 10.8 | 6.16 | 4.76 | 2.3x | 1.3x | 1779 / 1812 | 623 / 666 | 622 / 690 |
Startup and first call
| model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
|---|---|---|---|---|---|---|
clip-b16-laion2b |
13.69 | β | β | 0.16 | 0.06 | 0.05 |
clip-b16-openai |
16.66 | β | β | 0.09 | 0.07 | 0.06 |
clip-h14-378-dfn5b |
78.39 | β | β | 0.18 | 0.08 | 0.06 |
clip-h14-dfn5b |
79.37 | β | β | 0.18 | 0.09 | 0.06 |
clip-h14-laion2b |
78.69 | β | β | 0.18 | 0.07 | 0.07 |
clip-l14-336-openai |
30.98 | β | β | 0.17 | 0.06 | 0.03 |
clip-l14-datacompxl |
39.60 | β | β | 0.17 | 0.04 | 0.03 |
clip-l14-dfn2b |
39.79 | β | β | 0.17 | 0.06 | 0.04 |
clip-l14-laion2b |
39.40 | β | β | 0.17 | 0.04 | 0.03 |
clip-l14-openai |
41.72 | β | β | 0.07 | 0.06 | 0.03 |
torch.compile and TensorRT
| model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
|---|---|---|---|---|---|---|---|---|---|---|---|
clip-b16-laion2b |
224x224 | 3.09 ms | 2.41 ms | 1.12 ms | 1.36 ms | 6.49 s | 24.3 s | 202 / 226 | 237 / 266 | 203 / 230 | 394 / 422 |
clip-b16-openai |
224x224 | 3.35 ms | 2.39 ms | 1.11 ms | 1.36 ms | 6.37 s | 14.6 s | 204 / 226 | 237 / 266 | 203 / 250 | 394 / 422 |
clip-h14-378-dfn5b |
224x224 | 11.4 ms | 11.8 ms | 7.85 ms | 8.24 ms | 13.6 s | 58.9 s | 1254 / 1288 | 1290 / 1328 | 1251 / 1312 | 2468 / 2510 |
clip-h14-dfn5b |
224x224 | 11.4 ms | 11.5 ms | 7.76 ms | 8.54 ms | 13.0 s | 36.3 s | 1252 / 1268 | 1289 / 1310 | 1250 / 1292 | 2465 / 2490 |
clip-h14-laion2b |
224x224 | 11.2 ms | 11.6 ms | 7.84 ms | 8.27 ms | 14.3 s | 40.6 s | 1249 / 1268 | 1289 / 1310 | 1250 / 1272 | 2467 / 2492 |
clip-l14-336-openai |
224x224 | 6.16 ms | 6.36 ms | 4.25 ms | 4.76 ms | 10.7 s | 33.1 s | 625 / 646 | 658 / 686 | 622 / 668 | 1235 / 1264 |
clip-l14-datacompxl |
224x224 | 5.99 ms | 6.24 ms | 4.19 ms | 4.76 ms | 10.5 s | 32.1 s | 621 / 666 | 658 / 686 | 622 / 670 | 1232 / 1282 |
clip-l14-dfn2b |
224x224 | 6.16 ms | 6.16 ms | 4.14 ms | 4.76 ms | 11.4 s | 26.6 s | 623 / 666 | 658 / 686 | 622 / 690 | 1230 / 1280 |
clip-l14-laion2b |
224x224 | 5.98 ms | 6.19 ms | 4.19 ms | 4.76 ms | 10.4 s | 32.0 s | 621 / 666 | 658 / 686 | 622 / 670 | 1232 / 1282 |
clip-l14-openai |
224x224 | 6.16 ms | 6.17 ms | 4.14 ms | 4.76 ms | 10.2 s | 23.0 s | 623 / 666 | 658 / 686 | 622 / 690 | 1230 / 1280 |
Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
Accuracy
On public datasets
Two disjoint seeded subsets of the val folders; each image shorter side 256 (bicubic), centre crop 224, the CLS / pooled feature; cosine kNN, vote weighted by similarity. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | kernels | note |
|---|---|---|---|---|---|---|
clip-b16-openai |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 51.98 | 52.00 | 52.00 | 52.00 | |
clip-l14-openai |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 58.70 | 58.64 | 58.56 | 58.58 | |
clip-l14-336-openai |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 58.82 | 58.72 | 58.80 | 58.86 | |
clip-b16-laion2b |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 54.14 | 54.24 | 54.22 | 54.16 | |
clip-l14-laion2b |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 59.30 | 59.30 | 59.36 | 59.32 | |
clip-h14-laion2b |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 62.56 | 62.46 | 62.46 | 62.56 | |
clip-l14-datacompxl |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 62.50 | 62.44 | 62.44 | 62.38 | |
clip-l14-dfn2b |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 66.86 | 66.74 | 66.76 | 66.78 | |
clip-h14-dfn5b |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 68.72 | 68.70 | 68.80 | 68.86 | |
clip-h14-378-dfn5b |
ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 64.84 | 65.10 | 65.10 | 65.00 |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8bce3316 / b2ffd65d, 2026-11-11 to 2026-11-12.
Parity with the original code
Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 9.9 % and p99 <= 87 % of the reference's median |value| (absolute error of the output last_hidden_state).
Limitations
- kernels replays a CUDA graph per small input shape without hidden states (up to about 1M activations: ViT-B ~460 px, ViT-L ~300 px), captured on the shape's second call, at most 4 per model;
model.release_graphs()frees them. - Small kernels calls (graph-sized, with or without
layers) run GEMMs of at most 600 tokens on cuBLAS: faster in a graph replay (ViT-B/16 224 px: 1.26 against 1.52 ms), about 10 % slower eagerly on ViT-T / B; a replay equals the eager call bit for bit. fp32_accumulate = trueputs every GEMM on cuBLAS with fp32 accumulation (the portable precision; slower on GeForce).- fp16 error against the fp32 original is higher than for the other ViTs on L/14 DataComp (mean 2.8 % portable, 4.8 % kernels): one residual channel sits near 120, where fp16 steps are 0.06, about 10 % of the median activation. No overflow; float32 portable gives 0.3 %.
- DFN weights: the relative error (mean 1.3-3.9 % portable, 3.9-6.5 % kernels) is measured against a median |output| of only 0.015-0.021 (other weights: 0.5-0.6) with values up to 0.7;
fp32_accumulate = truelowers kernels to about 1.5 %. - Kernels at 224 x 224 peak above portable (B/16 257 vs 204 MB, L/14 814 vs 623 MB; original 544 / 1779 MB): the QuickGELU fold keeps a folded copy of each fc2 weight next to the original parameter, and the second call's CUDA-graph capture adds ~30 MB.
Usage
from fastforks import features
model = features.load("clip-b16-laion2b") # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg", size=518)
result.last_hidden_state # tokens [N, C], as Hugging Face's output
result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: clip-b16-laion2b, clip-b16-openai, clip-h14-378-dfn5b, clip-h14-dfn5b, clip-h14-laion2b, clip-l14-336-openai, clip-l14-datacompxl, clip-l14-dfn2b, clip-l14-laion2b, clip-l14-openai.
Kernels backend
features.load("clip-b16-laion2b", mode="kernels") runs our CUDA kernels (cudnn_hacc, fvm_conv, fvm_extra, fvm_mb, fvm_r3). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.