--- library_name: "fastforks" pipeline_tag: "image-feature-extraction" license: "other" license_name: "mixed-see-variant-table" license_link: "https://huggingface.co/timm/vit_base_patch16_clip_224.laion2b/blob/fbadfdd1c0e903c58855b70ae04740b22da9d305/README.md" tags: - "features" - "backbone" - "feature-extraction" - "cuda" - "pytorch" - "fastforks" - "fastforks-card" --- # CLIP (fastforks) CLIP image towers (Radford et al. 2021): OpenAI, OpenCLIP (LAION-2B, DataComp-1B) and Apple DFN weights in timm's format; cls + patch tokens at any size, pooler_output = the post-LayerNorm cls token before the projection (QuickGELU for the OpenAI and DFN weights). This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: . Kernels for the `kernels` backend: . Original code: at `0df212b369a5`. Author weight repos: , , , , , , , , , . Task: features. Tier: stable. Outputs: `last_hidden_state`, `pooler_output`, `hidden_states`. Backends: `portable`, `kernels`. Input: one RGB image. Output: dense patch features and a pooled image feature. ## Variants Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/features/models/clip/card.toml). | variant | weights (author link) | SHA-256 | weights licence | commercial use | |---|---|---|---|---| | `clip-b16-laion2b@1` | [vit_base_patch16_clip_224.laion2b.safetensors](https://huggingface.co/timm/vit_base_patch16_clip_224.laion2b/resolve/fbadfdd1c0e903c58855b70ae04740b22da9d305/model.safetensors) | `35fcf2008962` | MIT | yes | | `clip-b16-openai@1` | [vit_base_patch16_clip_224.openai.safetensors](https://huggingface.co/timm/vit_base_patch16_clip_224.openai/resolve/977e3dd0ec55ab8da155f2fbeb6b5f54948b6e3d/open_clip_model.safetensors) | `4b8699299b1e` | MIT | yes | | `clip-h14-378-dfn5b@1` | [vit_huge_patch14_clip_378.dfn5b.safetensors](https://huggingface.co/timm/vit_huge_patch14_clip_378.dfn5b/resolve/ae5cf5b82fdef1db1128b946ef3635ec99b5660a/model.safetensors) | `81d5bf47b916` | LicenseRef-Apple-AMLR | **no** | | `clip-h14-dfn5b@1` | [vit_huge_patch14_clip_224.dfn5b.safetensors](https://huggingface.co/timm/vit_huge_patch14_clip_224.dfn5b/resolve/f578dd8feffd415d1dd999618fe2fc55ed262d78/model.safetensors) | `7ebd4bca4bae` | LicenseRef-Apple-AMLR | **no** | | `clip-h14-laion2b@1` | [vit_huge_patch14_clip_224.laion2b.safetensors](https://huggingface.co/timm/vit_huge_patch14_clip_224.laion2b/resolve/b8441fa3f968a5e469c166176ee82f8ce8dbc4eb/model.safetensors) | `a18dfb0dedf0` | MIT | yes | | `clip-l14-336-openai@1` | [vit_large_patch14_clip_336.openai.safetensors](https://huggingface.co/timm/vit_large_patch14_clip_336.openai/resolve/81e38efc4637de5023b10e75a7f9bd1c6fa6b010/model.safetensors) | `18012c780964` | MIT | yes | | `clip-l14-datacompxl@1` | [vit_large_patch14_clip_224.datacompxl.safetensors](https://huggingface.co/timm/vit_large_patch14_clip_224.datacompxl/resolve/7fb391265b74ed5bdcc84c84b892595addb833be/model.safetensors) | `b63e3d1bc90a` | MIT | yes | | `clip-l14-dfn2b@1` | [vit_large_patch14_clip_224.dfn2b.safetensors](https://huggingface.co/timm/vit_large_patch14_clip_224.dfn2b/resolve/0052124c38aa20f89fb2b186b6c2599ffa91ca5b/model.safetensors) | `7a20e7340404` | LicenseRef-Apple-AMLR | **no** | | `clip-l14-laion2b@1` | [vit_large_patch14_clip_224.laion2b.safetensors](https://huggingface.co/timm/vit_large_patch14_clip_224.laion2b/resolve/ce143c10d3592fce8cfabbee2b990656c2aaead6/model.safetensors) | `e7204b9868d1` | MIT | yes | | `clip-l14-openai@1` | [vit_large_patch14_clip_224.openai.safetensors](https://huggingface.co/timm/vit_large_patch14_clip_224.openai/resolve/18d0535469bb561bf468d76c1d73aa35156c922b/open_clip_model.safetensors) | `9ce2e8a8ebff` | MIT | yes | ## Licence Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0. Training data: not stated by the authors. Licence evidence: ## Input size any H x W >= 1, resized or padded inside the model Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `clip-b16-laion2b` 3000x4000 (1.7 GiB peak allocated, 1.9 reserved); `clip-b16-openai` 3000x4000 (1.9 GiB peak allocated, 2.2 reserved); `clip-h14-378-dfn5b` 3000x4000 (4.5 GiB peak allocated, 5.0 reserved); `clip-h14-dfn5b` 3000x4000 (4.5 GiB peak allocated, 5.0 reserved); `clip-h14-laion2b` 3000x4000 (3.9 GiB peak allocated, 4.4 reserved); `clip-l14-336-openai` 3000x4000 (3.4 GiB peak allocated, 3.6 reserved); `clip-l14-datacompxl` 3000x4000 (2.9 GiB peak allocated, 3.2 reserved); `clip-l14-dfn2b` 3000x4000 (3.4 GiB peak allocated, 3.7 reserved); `clip-l14-laion2b` 3000x4000 (2.9 GiB peak allocated, 3.2 reserved); `clip-l14-openai` 3000x4000 (3.4 GiB peak allocated, 3.7 reserved) ## Speed and memory ### Features, synthetic, 224x224 | model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `clip-b16-laion2b` | 6.22 | 5.29 | 3.09 | 1.36 | 3.9x | 2.3x | 544 / 580 | 202 / 226 | 203 / 230 | | `clip-b16-openai` | 6.28 | 5.65 | 3.35 | 1.36 | 4.2x | 2.5x | 544 / 580 | 204 / 226 | 203 / 250 | | `clip-h14-378-dfn5b` | 47.9 | 19.5 | 11.4 | 8.24 | 2.4x | 1.4x | 3722 / 4038 | 1254 / 1288 | 1251 / 1312 | | `clip-h14-dfn5b` | 47.5 | 19.4 | 11.4 | 8.54 | 2.3x | 1.3x | 3720 / 4038 | 1252 / 1268 | 1250 / 1292 | | `clip-h14-laion2b` | 47.2 | 19.3 | 11.2 | 8.27 | 2.3x | 1.4x | 3720 / 4032 | 1249 / 1268 | 1250 / 1272 | | `clip-l14-336-openai` | 22.1 | 10.8 | 6.16 | 4.76 | 2.3x | 1.3x | 1779 / 1814 | 625 / 646 | 622 / 668 | | `clip-l14-datacompxl` | 21.6 | 10.1 | 5.99 | 4.76 | 2.1x | 1.3x | 1779 / 1812 | 621 / 666 | 622 / 670 | | `clip-l14-dfn2b` | 22.0 | 10.6 | 6.16 | 4.76 | 2.2x | 1.3x | 1779 / 1812 | 623 / 666 | 622 / 690 | | `clip-l14-laion2b` | 21.6 | 10.1 | 5.98 | 4.76 | 2.1x | 1.3x | 1779 / 1812 | 621 / 666 | 622 / 670 | | `clip-l14-openai` | 22.0 | 10.8 | 6.16 | 4.76 | 2.3x | 1.3x | 1779 / 1812 | 623 / 666 | 622 / 690 | ### Startup and first call | model | load (s): original | portable | kernels | first call (s): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | | `clip-b16-laion2b` | 13.69 | – | – | 0.16 | 0.06 | 0.05 | | `clip-b16-openai` | 16.66 | – | – | 0.09 | 0.07 | 0.06 | | `clip-h14-378-dfn5b` | 78.39 | – | – | 0.18 | 0.08 | 0.06 | | `clip-h14-dfn5b` | 79.37 | – | – | 0.18 | 0.09 | 0.06 | | `clip-h14-laion2b` | 78.69 | – | – | 0.18 | 0.07 | 0.07 | | `clip-l14-336-openai` | 30.98 | – | – | 0.17 | 0.06 | 0.03 | | `clip-l14-datacompxl` | 39.60 | – | – | 0.17 | 0.04 | 0.03 | | `clip-l14-dfn2b` | 39.79 | – | – | 0.17 | 0.06 | 0.04 | | `clip-l14-laion2b` | 39.40 | – | – | 0.17 | 0.04 | 0.03 | | `clip-l14-openai` | 41.72 | – | – | 0.07 | 0.06 | 0.03 | ### torch.compile and TensorRT | model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `clip-b16-laion2b` | 224x224 | 3.09 ms | 2.41 ms | 1.12 ms | 1.36 ms | 6.49 s | 24.3 s | 202 / 226 | 237 / 266 | 203 / 230 | 394 / 422 | | `clip-b16-openai` | 224x224 | 3.35 ms | 2.39 ms | 1.11 ms | 1.36 ms | 6.37 s | 14.6 s | 204 / 226 | 237 / 266 | 203 / 250 | 394 / 422 | | `clip-h14-378-dfn5b` | 224x224 | 11.4 ms | 11.8 ms | 7.85 ms | 8.24 ms | 13.6 s | 58.9 s | 1254 / 1288 | 1290 / 1328 | 1251 / 1312 | 2468 / 2510 | | `clip-h14-dfn5b` | 224x224 | 11.4 ms | 11.5 ms | 7.76 ms | 8.54 ms | 13.0 s | 36.3 s | 1252 / 1268 | 1289 / 1310 | 1250 / 1292 | 2465 / 2490 | | `clip-h14-laion2b` | 224x224 | 11.2 ms | 11.6 ms | 7.84 ms | 8.27 ms | 14.3 s | 40.6 s | 1249 / 1268 | 1289 / 1310 | 1250 / 1272 | 2467 / 2492 | | `clip-l14-336-openai` | 224x224 | 6.16 ms | 6.36 ms | 4.25 ms | 4.76 ms | 10.7 s | 33.1 s | 625 / 646 | 658 / 686 | 622 / 668 | 1235 / 1264 | | `clip-l14-datacompxl` | 224x224 | 5.99 ms | 6.24 ms | 4.19 ms | 4.76 ms | 10.5 s | 32.1 s | 621 / 666 | 658 / 686 | 622 / 670 | 1232 / 1282 | | `clip-l14-dfn2b` | 224x224 | 6.16 ms | 6.16 ms | 4.14 ms | 4.76 ms | 11.4 s | 26.6 s | 623 / 666 | 658 / 686 | 622 / 690 | 1230 / 1280 | | `clip-l14-laion2b` | 224x224 | 5.98 ms | 6.19 ms | 4.19 ms | 4.76 ms | 10.4 s | 32.0 s | 621 / 666 | 658 / 686 | 622 / 670 | 1232 / 1282 | | `clip-l14-openai` | 224x224 | 6.16 ms | 6.17 ms | 4.14 ms | 4.76 ms | 10.2 s | 23.0 s | 623 / 666 | 658 / 686 | 622 / 690 | 1230 / 1280 | Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB). Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's. ## Accuracy ### On public datasets Two disjoint seeded subsets of the val folders; each image shorter side 256 (bicubic), centre crop 224, the CLS / pooled feature; cosine kNN, vote weighted by similarity. Original = the original code named under Details, same checkpoint and protocol. | variant | dataset | original fp32 | original fp16 | portable | kernels | note | |---|---|---|---|---|---|---| | `clip-b16-openai` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 51.98 | 52.00 | 52.00 | 52.00 | | | `clip-l14-openai` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 58.70 | 58.64 | 58.56 | 58.58 | | | `clip-l14-336-openai` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 58.82 | 58.72 | 58.80 | 58.86 | | | `clip-b16-laion2b` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 54.14 | 54.24 | 54.22 | 54.16 | | | `clip-l14-laion2b` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 59.30 | 59.30 | 59.36 | 59.32 | | | `clip-h14-laion2b` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 62.56 | 62.46 | 62.46 | 62.56 | | | `clip-l14-datacompxl` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 62.50 | 62.44 | 62.44 | 62.38 | | | `clip-l14-dfn2b` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 66.86 | 66.74 | 66.76 | 66.78 | | | `clip-h14-dfn5b` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 68.72 | 68.70 | 68.80 | 68.86 | | | `clip-h14-378-dfn5b` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 64.84 | 65.10 | 65.10 | 65.00 | | Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8bce3316 / b2ffd65d, 2026-11-11 to 2026-11-12. ### Parity with the original code Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 9.9 % and p99 <= 87 % of the reference's median |value| (absolute error of the output `last_hidden_state`). ## Limitations - kernels replays a CUDA graph per small input shape without hidden states (up to about 1M activations: ViT-B ~460 px, ViT-L ~300 px), captured on the shape's second call, at most 4 per model; `model.release_graphs()` frees them. - Small kernels calls (graph-sized, with or without `layers`) run GEMMs of at most 600 tokens on cuBLAS: faster in a graph replay (ViT-B/16 224 px: 1.26 against 1.52 ms), about 10 % slower eagerly on ViT-T / B; a replay equals the eager call bit for bit. - `fp32_accumulate = true` puts every GEMM on cuBLAS with fp32 accumulation (the portable precision; slower on GeForce). - fp16 error against the fp32 original is higher than for the other ViTs on L/14 DataComp (mean 2.8 % portable, 4.8 % kernels): one residual channel sits near 120, where fp16 steps are 0.06, about 10 % of the median activation. No overflow; float32 portable gives 0.3 %. - DFN weights: the relative error (mean 1.3-3.9 % portable, 3.9-6.5 % kernels) is measured against a median |output| of only 0.015-0.021 (other weights: 0.5-0.6) with values up to 0.7; `fp32_accumulate = true` lowers kernels to about 1.5 %. - Kernels at 224 x 224 peak above portable (B/16 257 vs 204 MB, L/14 814 vs 623 MB; original 544 / 1779 MB): the QuickGELU fold keeps a folded copy of each fc2 weight next to the original parameter, and the second call's CUDA-graph capture adds ~30 MB. ## Usage ```python from fastforks import features model = features.load("clip-b16-laion2b") # mode="auto": the preferred backend from recorded measurements result = model("photo.jpg", size=518) result.last_hidden_state # tokens [N, C], as Hugging Face's output result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16') ``` Install with `pip install fastforks` (see ). Other variants: `clip-b16-laion2b`, `clip-b16-openai`, `clip-h14-378-dfn5b`, `clip-h14-dfn5b`, `clip-h14-laion2b`, `clip-l14-336-openai`, `clip-l14-datacompxl`, `clip-l14-dfn2b`, `clip-l14-laion2b`, `clip-l14-openai`. ## Kernels backend `features.load("clip-b16-laion2b", mode="kernels")` runs our CUDA kernels (`cudnn_hacc`, `fvm_conv`, `fvm_extra`, `fvm_mb`, `fvm_r3`). It needs a prebuilt kernel for your GPU architecture (downloaded from on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.