File size: 13,955 Bytes
3e9833c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 | ---
library_name: "fastforks"
pipeline_tag: "image-feature-extraction"
license: "apache-2.0"
tags:
- "features"
- "backbone"
- "feature-extraction"
- "cuda"
- "pytorch"
- "fastforks"
- "fastforks-card"
---
# Perception Encoder (fastforks)
Perception Encoder (Bolya et al. 2025, Meta) image towers with 2D RoPE: PE core (contrastive, attention-pool embedding) and PE spatial (dense features for detection / segmentation; no learned pool: pooler_output is the mean of the patch tokens, as timm); tokens at any size.
This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/huggingface/pytorch-image-models> at `0df212b369a5`.
Author weight repos: <https://huggingface.co/timm/vit_pe_core_base_patch16_224.fb>, <https://huggingface.co/timm/vit_pe_core_gigantic_patch14_448.fb>, <https://huggingface.co/timm/vit_pe_core_large_patch14_336.fb>, <https://huggingface.co/timm/vit_pe_core_small_patch16_384.fb>, <https://huggingface.co/timm/vit_pe_core_tiny_patch16_384.fb>, <https://huggingface.co/timm/vit_pe_spatial_base_patch16_512.fb>, <https://huggingface.co/timm/vit_pe_spatial_gigantic_patch14_448.fb>, <https://huggingface.co/timm/vit_pe_spatial_large_patch14_448.fb>, <https://huggingface.co/timm/vit_pe_spatial_small_patch16_512.fb>, <https://huggingface.co/timm/vit_pe_spatial_tiny_patch16_512.fb>.
Task: features. Tier: stable. Outputs: `last_hidden_state`, `pooler_output`, `hidden_states`. Backends: `portable`, `kernels`.
Input: one RGB image. Output: dense patch features and a pooled image feature.
## Variants
Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/features/models/perception_encoder/card.toml).
| variant | weights (author link) | SHA-256 | weights licence | commercial use |
|---|---|---|---|---|
| `pe-core-b16-224@1` | [vit_pe_core_base_patch16_224.fb.safetensors](https://huggingface.co/timm/vit_pe_core_base_patch16_224.fb/resolve/0f9fb1eea04d135faa1f15af05734fb996fcaf94/model.safetensors) | `802a8c86fed6` | Apache-2.0 | yes |
| `pe-core-g14-448@1` | [vit_pe_core_gigantic_patch14_448.fb.safetensors](https://huggingface.co/timm/vit_pe_core_gigantic_patch14_448.fb/resolve/fb096c09d13bd2e3827aed4dbdffb8a6b8e0c10b/model.safetensors) | `a0a499d4a7c6` | Apache-2.0 | yes |
| `pe-core-l14-336@1` | [vit_pe_core_large_patch14_336.fb.safetensors](https://huggingface.co/timm/vit_pe_core_large_patch14_336.fb/resolve/e63206c8e3a0e9b699e40f31080eebd78fd2258e/model.safetensors) | `35edcdb68eb4` | Apache-2.0 | yes |
| `pe-core-s16-384@1` | [vit_pe_core_small_patch16_384.fb.safetensors](https://huggingface.co/timm/vit_pe_core_small_patch16_384.fb/resolve/255fa76e26fdf8daa47e0d607da46b7051879045/model.safetensors) | `b07ccd4489dc` | Apache-2.0 | yes |
| `pe-core-t16-384@1` | [vit_pe_core_tiny_patch16_384.fb.safetensors](https://huggingface.co/timm/vit_pe_core_tiny_patch16_384.fb/resolve/29e4ebb6cc0502b84ba432add45499f4ffa44238/model.safetensors) | `d97f3436c9bf` | Apache-2.0 | yes |
| `pe-spatial-b16-512@1` | [vit_pe_spatial_base_patch16_512.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_base_patch16_512.fb/resolve/53267ce15082ad3e246bc5b70adc6da5f9819a70/model.safetensors) | `aa56bd9f500f` | Apache-2.0 | yes |
| `pe-spatial-g14-448@1` | [vit_pe_spatial_gigantic_patch14_448.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_gigantic_patch14_448.fb/resolve/f2cea21ca9d5900061bee5e896031ab4e8352936/model.safetensors) | `db69c2e41d38` | Apache-2.0 | yes |
| `pe-spatial-l14-448@1` | [vit_pe_spatial_large_patch14_448.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_large_patch14_448.fb/resolve/ac6f424680a305470340dac75a0a9b4c47ee030f/model.safetensors) | `c21925194dee` | Apache-2.0 | yes |
| `pe-spatial-s16-512@1` | [vit_pe_spatial_small_patch16_512.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_small_patch16_512.fb/resolve/bde8c2a1ca579f414140672d311c3a4fcf1058a7/model.safetensors) | `ed334e352f84` | Apache-2.0 | yes |
| `pe-spatial-t16-512@1` | [vit_pe_spatial_tiny_patch16_512.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_tiny_patch16_512.fb/resolve/2531addff69b565668fc6f2065a1c222cf1d431b/model.safetensors) | `7242055f36d8` | Apache-2.0 | yes |
## Licence
Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.
Training data: PE core: billions of web image-text pairs curated with the MetaCLIP pipeline, then video-text fine-tuning; PE spatial: PE core-G distilled with SAM 2 mask-feature alignment (Perception Encoder paper); not released.
Licence evidence: <https://huggingface.co/timm/vit_pe_core_small_patch16_384.fb/blob/255fa76e26fdf8daa47e0d607da46b7051879045/README.md>
## Input size
any H x W >= 1, resized or padded inside the model
Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `pe-core-b16-224` 3000x4000 (2.0 GiB peak allocated, 2.3 reserved); `pe-core-g14-448` 3000x4000 (7.7 GiB peak allocated, 10.2 reserved); `pe-core-l14-336` 3000x4000 (3.5 GiB peak allocated, 3.8 reserved); `pe-core-s16-384` 3000x4000 (1.2 GiB peak allocated, 1.3 reserved); `pe-core-t16-384` 3000x4000 (0.8 GiB peak allocated, 1.1 reserved); `pe-spatial-b16-512` 3000x4000 (2.0 GiB peak allocated, 2.3 reserved); `pe-spatial-g14-448` 1988x2880 (5.6 GiB peak allocated, 5.8 reserved); `pe-spatial-l14-448` 3000x4000 (3.4 GiB peak allocated, 3.7 reserved); `pe-spatial-s16-512` 3000x4000 (1.2 GiB peak allocated, 1.3 reserved); `pe-spatial-t16-512` 3000x4000 (0.8 GiB peak allocated, 1.0 reserved)
## Speed and memory
### Features, synthetic, 224x224
| model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `pe-core-b16-224` | 7.14 | 9.35 | 4.97 | 1.47 | 4.9x | 3.4x | 584 / 620 | 218 / 266 | 218 / 268 |
| `pe-core-g14-448` | 97.7 | 52.4 | 25.4 | 17.0 | 3.1x | 1.5x | 10848 / 11330 | 3642 / 3670 | 5180 / 5256 |
| `pe-core-l14-336` | 23.3 | 17.7 | 9.09 | 4.97 | 3.6x | 1.8x | 1848 / 1888 | 650 / 668 | 648 / 672 |
| `pe-core-s16-384` | 7.02 | 9.28 | 4.80 | 0.957 | 7.3x | 5.0x | 171 / 180 | 83 / 102 | 82 / 104 |
| `pe-core-t16-384` | 6.74 | 9.04 | 4.68 | 0.683 | 9.9x | 6.9x | 68 / 70 | 48 / 56 | 53 / 82 |
| `pe-spatial-b16-512` | 7.02 | 8.79 | 4.74 | 1.40 | 5.0x | 3.4x | 544 / 580 | 207 / 226 | 204 / 248 |
| `pe-spatial-g14-448` | 97.9 | 52.7 | 25.5 | 16.9 | 3.1x | 1.5x | 10688 / 11178 | 3584 / 3612 | 5123 / 5156 |
| `pe-spatial-l14-448` | 23.0 | 17.2 | 8.84 | 4.87 | 3.5x | 1.8x | 1780 / 1816 | 626 / 668 | 623 / 672 |
| `pe-spatial-s16-512` | 6.70 | 8.79 | 4.51 | 0.922 | 7.3x | 4.9x | 161 / 178 | 81 / 100 | 78 / 104 |
| `pe-spatial-t16-512` | 6.56 | 8.73 | 4.48 | 0.666 | 9.8x | 6.7x | 66 / 68 | 49 / 56 | 53 / 82 |
### Startup and first call
| model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- |
| `pe-core-b16-224` | 14.55 | β | β | 0.19 | 0.29 | 0.07 |
| `pe-core-g14-448` | 232.98 | β | β | 0.28 | 0.20 | 0.13 |
| `pe-core-l14-336` | 41.39 | β | β | 0.20 | 0.11 | 0.05 |
| `pe-core-s16-384` | 5.70 | β | β | 0.18 | 0.10 | 0.04 |
| `pe-core-t16-384` | 3.20 | β | β | 0.19 | 0.12 | 0.07 |
| `pe-spatial-b16-512` | 13.65 | β | β | 0.16 | 0.10 | 0.07 |
| `pe-spatial-g14-448` | 227.90 | β | β | 0.36 | 0.22 | 0.32 |
| `pe-spatial-l14-448` | 44.48 | β | β | 0.17 | 0.09 | 0.06 |
| `pe-spatial-s16-512` | 5.40 | β | β | 0.16 | 0.08 | 0.06 |
| `pe-spatial-t16-512` | 3.09 | β | β | 0.16 | 0.10 | 0.08 |
### torch.compile and TensorRT
| model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `pe-core-b16-224` | 224x224 | 4.97 ms | 2.50 ms | 1.17 ms | 1.47 ms | 8.71 s | 25.8 s | 218 / 266 | 251 / 286 | 218 / 268 | 409 / 460 |
| `pe-core-g14-448` | 224x224 | 25.4 ms | 24.5 ms | 15.0 ms | 17.0 ms | 29.9 s | 106 s | 3642 / 3670 | 3667 / 3690 | 5180 / 5256 | 7189 / 7230 |
| `pe-core-l14-336` | 224x224 | 9.09 ms | 6.53 ms | 4.34 ms | 4.97 ms | 13.9 s | 50.1 s | 650 / 668 | 683 / 706 | 648 / 672 | 1260 / 1286 |
| `pe-core-s16-384` | 224x224 | 4.80 ms | 2.39 ms | 0.581 ms | 0.957 ms | 8.75 s | 36.7 s | 83 / 102 | 113 / 122 | 82 / 104 | 145 / 166 |
| `pe-core-t16-384` | 224x224 | 4.68 ms | 2.29 ms | 0.432 ms | 0.683 ms | 8.89 s | 36.1 s | 48 / 56 | 78 / 96 | 53 / 82 | 79 / 88 |
| `pe-spatial-b16-512` | 224x224 | 4.74 ms | 2.67 ms | 1.20 ms | 1.40 ms | 8.67 s | 35.1 s | 207 / 226 | 237 / 266 | 204 / 248 | 397 / 422 |
| `pe-spatial-g14-448` | 224x224 | 25.5 ms | 24.3 ms | 14.9 ms | 16.9 ms | 28.3 s | 126 s | 3584 / 3612 | 3608 / 3630 | 5123 / 5156 | 7136 / 7156 |
| `pe-spatial-l14-448` | 224x224 | 8.84 ms | 6.59 ms | 4.35 ms | 4.87 ms | 13.7 s | 53.4 s | 626 / 668 | 658 / 686 | 623 / 672 | 1235 / 1284 |
| `pe-spatial-s16-512` | 224x224 | 4.51 ms | 2.67 ms | 0.580 ms | 0.922 ms | 8.67 s | 35.3 s | 81 / 100 | 110 / 120 | 78 / 104 | 142 / 144 |
| `pe-spatial-t16-512` | 224x224 | 4.48 ms | 2.12 ms | 0.426 ms | 0.666 ms | 8.45 s | 33.1 s | 49 / 56 | 77 / 96 | 53 / 82 | 78 / 88 |
Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
## Accuracy
### On public datasets
Two disjoint seeded subsets of the val folders; each image shorter side 256 (bicubic), centre crop 224, the CLS / pooled feature; cosine kNN, vote weighted by similarity. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | kernels | note |
|---|---|---|---|---|---|---|
| `pe-core-s16-384` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 45.56 | 45.54 | 45.48 | 45.38 | |
| `pe-core-b16-224` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 58.54 | 58.56 | 58.54 | 58.50 | |
| `pe-core-l14-336` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 65.98 | 66.06 | 65.96 | 66.28 | |
| `pe-spatial-t16-512` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 5.52 | 5.54 | 5.50 | 5.46 | |
| `pe-spatial-s16-512` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 8.02 | 8.06 | 8.02 | 7.98 | |
| `pe-spatial-b16-512` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 13.58 | 13.64 | 13.56 | 13.58 | |
| `pe-spatial-l14-448` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 17.10 | 17.14 | 17.14 | 17.12 | |
| `pe-spatial-g14-448` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 16.36 | 16.30 | 16.32 | 16.30 | |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8bce3316 / b2ffd65d / c16d5d41, 2026-11-11 to 2026-11-13.
### Parity with the original code
Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 16 % and p99 <= 68 % of the reference's median |value| (absolute error of the output `last_hidden_state`).
## Limitations
- kernels replays a CUDA graph per small input shape without hidden states (up to about 1M activations: ViT-B ~460 px, ViT-L ~300 px), captured on the shape's second call, at most 4 per model; `model.release_graphs()` frees them.
- Small kernels calls (graph-sized, with or without `layers`) run GEMMs of at most 600 tokens on cuBLAS: faster in a graph replay (ViT-B/16 224 px: 1.26 against 1.52 ms), about 10 % slower eagerly on ViT-T / B; a replay equals the eager call bit for bit.
- `fp32_accumulate = true` puts every GEMM on cuBLAS with fp32 accumulation (the portable precision; slower on GeForce).
## Usage
```python
from fastforks import features
model = features.load("pe-core-b16-224") # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg", size=518)
result.last_hidden_state # tokens [N, C], as Hugging Face's output
result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
```
Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `pe-core-b16-224`, `pe-core-g14-448`, `pe-core-l14-336`, `pe-core-s16-384`, `pe-core-t16-384`, `pe-spatial-b16-512`, `pe-spatial-g14-448`, `pe-spatial-l14-448`, `pe-spatial-s16-512`, `pe-spatial-t16-512`.
## Kernels backend
`features.load("pe-core-b16-224", mode="kernels")` runs our CUDA kernels (`cudnn_hacc`, `fvm_conv`, `fvm_extra`, `fvm_mb`, `fvm_r3`). It needs a prebuilt kernel for your GPU architecture (downloaded from <https://huggingface.co/fastforks/kernels> on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.
<!-- staged from fastforks 1cbf1671 (2026-11-15) -->
|