perception-encoder / README.md
Flor's picture
card: update from fastforks 1cbf1671
3e9833c verified
|
Raw History Blame Contribute Delete
14 kB
metadata
library_name: fastforks
pipeline_tag: image-feature-extraction
license: apache-2.0
tags:
  - features
  - backbone
  - feature-extraction
  - cuda
  - pytorch
  - fastforks
  - fastforks-card

Perception Encoder (fastforks)

Perception Encoder (Bolya et al. 2025, Meta) image towers with 2D RoPE: PE core (contrastive, attention-pool embedding) and PE spatial (dense features for detection / segmentation; no learned pool: pooler_output is the mean of the patch tokens, as timm); tokens at any size.

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/pytorch-image-models at 0df212b369a5.

Author weight repos: https://huggingface.co/timm/vit_pe_core_base_patch16_224.fb, https://huggingface.co/timm/vit_pe_core_gigantic_patch14_448.fb, https://huggingface.co/timm/vit_pe_core_large_patch14_336.fb, https://huggingface.co/timm/vit_pe_core_small_patch16_384.fb, https://huggingface.co/timm/vit_pe_core_tiny_patch16_384.fb, https://huggingface.co/timm/vit_pe_spatial_base_patch16_512.fb, https://huggingface.co/timm/vit_pe_spatial_gigantic_patch14_448.fb, https://huggingface.co/timm/vit_pe_spatial_large_patch14_448.fb, https://huggingface.co/timm/vit_pe_spatial_small_patch16_512.fb, https://huggingface.co/timm/vit_pe_spatial_tiny_patch16_512.fb.

Task: features. Tier: stable. Outputs: last_hidden_state, pooler_output, hidden_states. Backends: portable, kernels.

Input: one RGB image. Output: dense patch features and a pooled image feature.

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
pe-core-b16-224@1 vit_pe_core_base_patch16_224.fb.safetensors 802a8c86fed6 Apache-2.0 yes
pe-core-g14-448@1 vit_pe_core_gigantic_patch14_448.fb.safetensors a0a499d4a7c6 Apache-2.0 yes
pe-core-l14-336@1 vit_pe_core_large_patch14_336.fb.safetensors 35edcdb68eb4 Apache-2.0 yes
pe-core-s16-384@1 vit_pe_core_small_patch16_384.fb.safetensors b07ccd4489dc Apache-2.0 yes
pe-core-t16-384@1 vit_pe_core_tiny_patch16_384.fb.safetensors d97f3436c9bf Apache-2.0 yes
pe-spatial-b16-512@1 vit_pe_spatial_base_patch16_512.fb.safetensors aa56bd9f500f Apache-2.0 yes
pe-spatial-g14-448@1 vit_pe_spatial_gigantic_patch14_448.fb.safetensors db69c2e41d38 Apache-2.0 yes
pe-spatial-l14-448@1 vit_pe_spatial_large_patch14_448.fb.safetensors c21925194dee Apache-2.0 yes
pe-spatial-s16-512@1 vit_pe_spatial_small_patch16_512.fb.safetensors ed334e352f84 Apache-2.0 yes
pe-spatial-t16-512@1 vit_pe_spatial_tiny_patch16_512.fb.safetensors 7242055f36d8 Apache-2.0 yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.

Training data: PE core: billions of web image-text pairs curated with the MetaCLIP pipeline, then video-text fine-tuning; PE spatial: PE core-G distilled with SAM 2 mask-feature alignment (Perception Encoder paper); not released.

Licence evidence: https://huggingface.co/timm/vit_pe_core_small_patch16_384.fb/blob/255fa76e26fdf8daa47e0d607da46b7051879045/README.md

Input size

any H x W >= 1, resized or padded inside the model Tested up to (portable, fp16, RTX 5060 Ti 16 GB): pe-core-b16-224 3000x4000 (2.0 GiB peak allocated, 2.3 reserved); pe-core-g14-448 3000x4000 (7.7 GiB peak allocated, 10.2 reserved); pe-core-l14-336 3000x4000 (3.5 GiB peak allocated, 3.8 reserved); pe-core-s16-384 3000x4000 (1.2 GiB peak allocated, 1.3 reserved); pe-core-t16-384 3000x4000 (0.8 GiB peak allocated, 1.1 reserved); pe-spatial-b16-512 3000x4000 (2.0 GiB peak allocated, 2.3 reserved); pe-spatial-g14-448 1988x2880 (5.6 GiB peak allocated, 5.8 reserved); pe-spatial-l14-448 3000x4000 (3.4 GiB peak allocated, 3.7 reserved); pe-spatial-s16-512 3000x4000 (1.2 GiB peak allocated, 1.3 reserved); pe-spatial-t16-512 3000x4000 (0.8 GiB peak allocated, 1.0 reserved)

Speed and memory

Features, synthetic, 224x224

model time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
pe-core-b16-224 7.14 9.35 4.97 1.47 4.9x 3.4x 584 / 620 218 / 266 218 / 268
pe-core-g14-448 97.7 52.4 25.4 17.0 3.1x 1.5x 10848 / 11330 3642 / 3670 5180 / 5256
pe-core-l14-336 23.3 17.7 9.09 4.97 3.6x 1.8x 1848 / 1888 650 / 668 648 / 672
pe-core-s16-384 7.02 9.28 4.80 0.957 7.3x 5.0x 171 / 180 83 / 102 82 / 104
pe-core-t16-384 6.74 9.04 4.68 0.683 9.9x 6.9x 68 / 70 48 / 56 53 / 82
pe-spatial-b16-512 7.02 8.79 4.74 1.40 5.0x 3.4x 544 / 580 207 / 226 204 / 248
pe-spatial-g14-448 97.9 52.7 25.5 16.9 3.1x 1.5x 10688 / 11178 3584 / 3612 5123 / 5156
pe-spatial-l14-448 23.0 17.2 8.84 4.87 3.5x 1.8x 1780 / 1816 626 / 668 623 / 672
pe-spatial-s16-512 6.70 8.79 4.51 0.922 7.3x 4.9x 161 / 178 81 / 100 78 / 104
pe-spatial-t16-512 6.56 8.73 4.48 0.666 9.8x 6.7x 66 / 68 49 / 56 53 / 82

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
pe-core-b16-224 14.55 – – 0.19 0.29 0.07
pe-core-g14-448 232.98 – – 0.28 0.20 0.13
pe-core-l14-336 41.39 – – 0.20 0.11 0.05
pe-core-s16-384 5.70 – – 0.18 0.10 0.04
pe-core-t16-384 3.20 – – 0.19 0.12 0.07
pe-spatial-b16-512 13.65 – – 0.16 0.10 0.07
pe-spatial-g14-448 227.90 – – 0.36 0.22 0.32
pe-spatial-l14-448 44.48 – – 0.17 0.09 0.06
pe-spatial-s16-512 5.40 – – 0.16 0.08 0.06
pe-spatial-t16-512 3.09 – – 0.16 0.10 0.08

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
pe-core-b16-224 224x224 4.97 ms 2.50 ms 1.17 ms 1.47 ms 8.71 s 25.8 s 218 / 266 251 / 286 218 / 268 409 / 460
pe-core-g14-448 224x224 25.4 ms 24.5 ms 15.0 ms 17.0 ms 29.9 s 106 s 3642 / 3670 3667 / 3690 5180 / 5256 7189 / 7230
pe-core-l14-336 224x224 9.09 ms 6.53 ms 4.34 ms 4.97 ms 13.9 s 50.1 s 650 / 668 683 / 706 648 / 672 1260 / 1286
pe-core-s16-384 224x224 4.80 ms 2.39 ms 0.581 ms 0.957 ms 8.75 s 36.7 s 83 / 102 113 / 122 82 / 104 145 / 166
pe-core-t16-384 224x224 4.68 ms 2.29 ms 0.432 ms 0.683 ms 8.89 s 36.1 s 48 / 56 78 / 96 53 / 82 79 / 88
pe-spatial-b16-512 224x224 4.74 ms 2.67 ms 1.20 ms 1.40 ms 8.67 s 35.1 s 207 / 226 237 / 266 204 / 248 397 / 422
pe-spatial-g14-448 224x224 25.5 ms 24.3 ms 14.9 ms 16.9 ms 28.3 s 126 s 3584 / 3612 3608 / 3630 5123 / 5156 7136 / 7156
pe-spatial-l14-448 224x224 8.84 ms 6.59 ms 4.35 ms 4.87 ms 13.7 s 53.4 s 626 / 668 658 / 686 623 / 672 1235 / 1284
pe-spatial-s16-512 224x224 4.51 ms 2.67 ms 0.580 ms 0.922 ms 8.67 s 35.3 s 81 / 100 110 / 120 78 / 104 142 / 144
pe-spatial-t16-512 224x224 4.48 ms 2.12 ms 0.426 ms 0.666 ms 8.45 s 33.1 s 49 / 56 77 / 96 53 / 82 78 / 88

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

Two disjoint seeded subsets of the val folders; each image shorter side 256 (bicubic), centre crop 224, the CLS / pooled feature; cosine kNN, vote weighted by similarity. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable kernels note
pe-core-s16-384 ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 45.56 45.54 45.48 45.38
pe-core-b16-224 ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 58.54 58.56 58.54 58.50
pe-core-l14-336 ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 65.98 66.06 65.96 66.28
pe-spatial-t16-512 ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 5.52 5.54 5.50 5.46
pe-spatial-s16-512 ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 8.02 8.06 8.02 7.98
pe-spatial-b16-512 ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 13.58 13.64 13.56 13.58
pe-spatial-l14-448 ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 17.10 17.14 17.14 17.12
pe-spatial-g14-448 ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) 16.36 16.30 16.32 16.30

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8bce3316 / b2ffd65d / c16d5d41, 2026-11-11 to 2026-11-13.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 16 % and p99 <= 68 % of the reference's median |value| (absolute error of the output last_hidden_state).

Limitations

  • kernels replays a CUDA graph per small input shape without hidden states (up to about 1M activations: ViT-B ~460 px, ViT-L ~300 px), captured on the shape's second call, at most 4 per model; model.release_graphs() frees them.
  • Small kernels calls (graph-sized, with or without layers) run GEMMs of at most 600 tokens on cuBLAS: faster in a graph replay (ViT-B/16 224 px: 1.26 against 1.52 ms), about 10 % slower eagerly on ViT-T / B; a replay equals the eager call bit for bit.
  • fp32_accumulate = true puts every GEMM on cuBLAS with fp32 accumulation (the portable precision; slower on GeForce).

Usage

from fastforks import features

model = features.load("pe-core-b16-224")  # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg", size=518)
result.last_hidden_state  # tokens [N, C], as Hugging Face's output
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: pe-core-b16-224, pe-core-g14-448, pe-core-l14-336, pe-core-s16-384, pe-core-t16-384, pe-spatial-b16-512, pe-spatial-g14-448, pe-spatial-l14-448, pe-spatial-s16-512, pe-spatial-t16-512.

Kernels backend

features.load("pe-core-b16-224", mode="kernels") runs our CUDA kernels (cudnn_hacc, fvm_conv, fvm_extra, fvm_mb, fvm_r3). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.