Flor commited on
Commit
3e9833c
Β·
verified Β·
1 Parent(s): 62d81da

card: update from fastforks 1cbf1671

Browse files
Files changed (1) hide show
  1. README.md +153 -0
README.md ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: "fastforks"
3
+ pipeline_tag: "image-feature-extraction"
4
+ license: "apache-2.0"
5
+ tags:
6
+ - "features"
7
+ - "backbone"
8
+ - "feature-extraction"
9
+ - "cuda"
10
+ - "pytorch"
11
+ - "fastforks"
12
+ - "fastforks-card"
13
+ ---
14
+ # Perception Encoder (fastforks)
15
+
16
+ Perception Encoder (Bolya et al. 2025, Meta) image towers with 2D RoPE: PE core (contrastive, attention-pool embedding) and PE spatial (dense features for detection / segmentation; no learned pool: pooler_output is the mean of the patch tokens, as timm); tokens at any size.
17
+
18
+ This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/huggingface/pytorch-image-models> at `0df212b369a5`.
19
+
20
+ Author weight repos: <https://huggingface.co/timm/vit_pe_core_base_patch16_224.fb>, <https://huggingface.co/timm/vit_pe_core_gigantic_patch14_448.fb>, <https://huggingface.co/timm/vit_pe_core_large_patch14_336.fb>, <https://huggingface.co/timm/vit_pe_core_small_patch16_384.fb>, <https://huggingface.co/timm/vit_pe_core_tiny_patch16_384.fb>, <https://huggingface.co/timm/vit_pe_spatial_base_patch16_512.fb>, <https://huggingface.co/timm/vit_pe_spatial_gigantic_patch14_448.fb>, <https://huggingface.co/timm/vit_pe_spatial_large_patch14_448.fb>, <https://huggingface.co/timm/vit_pe_spatial_small_patch16_512.fb>, <https://huggingface.co/timm/vit_pe_spatial_tiny_patch16_512.fb>.
21
+
22
+ Task: features. Tier: stable. Outputs: `last_hidden_state`, `pooler_output`, `hidden_states`. Backends: `portable`, `kernels`.
23
+
24
+ Input: one RGB image. Output: dense patch features and a pooled image feature.
25
+
26
+ ## Variants
27
+
28
+ Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/features/models/perception_encoder/card.toml).
29
+
30
+ | variant | weights (author link) | SHA-256 | weights licence | commercial use |
31
+ |---|---|---|---|---|
32
+ | `pe-core-b16-224@1` | [vit_pe_core_base_patch16_224.fb.safetensors](https://huggingface.co/timm/vit_pe_core_base_patch16_224.fb/resolve/0f9fb1eea04d135faa1f15af05734fb996fcaf94/model.safetensors) | `802a8c86fed6` | Apache-2.0 | yes |
33
+ | `pe-core-g14-448@1` | [vit_pe_core_gigantic_patch14_448.fb.safetensors](https://huggingface.co/timm/vit_pe_core_gigantic_patch14_448.fb/resolve/fb096c09d13bd2e3827aed4dbdffb8a6b8e0c10b/model.safetensors) | `a0a499d4a7c6` | Apache-2.0 | yes |
34
+ | `pe-core-l14-336@1` | [vit_pe_core_large_patch14_336.fb.safetensors](https://huggingface.co/timm/vit_pe_core_large_patch14_336.fb/resolve/e63206c8e3a0e9b699e40f31080eebd78fd2258e/model.safetensors) | `35edcdb68eb4` | Apache-2.0 | yes |
35
+ | `pe-core-s16-384@1` | [vit_pe_core_small_patch16_384.fb.safetensors](https://huggingface.co/timm/vit_pe_core_small_patch16_384.fb/resolve/255fa76e26fdf8daa47e0d607da46b7051879045/model.safetensors) | `b07ccd4489dc` | Apache-2.0 | yes |
36
+ | `pe-core-t16-384@1` | [vit_pe_core_tiny_patch16_384.fb.safetensors](https://huggingface.co/timm/vit_pe_core_tiny_patch16_384.fb/resolve/29e4ebb6cc0502b84ba432add45499f4ffa44238/model.safetensors) | `d97f3436c9bf` | Apache-2.0 | yes |
37
+ | `pe-spatial-b16-512@1` | [vit_pe_spatial_base_patch16_512.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_base_patch16_512.fb/resolve/53267ce15082ad3e246bc5b70adc6da5f9819a70/model.safetensors) | `aa56bd9f500f` | Apache-2.0 | yes |
38
+ | `pe-spatial-g14-448@1` | [vit_pe_spatial_gigantic_patch14_448.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_gigantic_patch14_448.fb/resolve/f2cea21ca9d5900061bee5e896031ab4e8352936/model.safetensors) | `db69c2e41d38` | Apache-2.0 | yes |
39
+ | `pe-spatial-l14-448@1` | [vit_pe_spatial_large_patch14_448.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_large_patch14_448.fb/resolve/ac6f424680a305470340dac75a0a9b4c47ee030f/model.safetensors) | `c21925194dee` | Apache-2.0 | yes |
40
+ | `pe-spatial-s16-512@1` | [vit_pe_spatial_small_patch16_512.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_small_patch16_512.fb/resolve/bde8c2a1ca579f414140672d311c3a4fcf1058a7/model.safetensors) | `ed334e352f84` | Apache-2.0 | yes |
41
+ | `pe-spatial-t16-512@1` | [vit_pe_spatial_tiny_patch16_512.fb.safetensors](https://huggingface.co/timm/vit_pe_spatial_tiny_patch16_512.fb/resolve/2531addff69b565668fc6f2065a1c222cf1d431b/model.safetensors) | `7242055f36d8` | Apache-2.0 | yes |
42
+
43
+ ## Licence
44
+
45
+ Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.
46
+
47
+ Training data: PE core: billions of web image-text pairs curated with the MetaCLIP pipeline, then video-text fine-tuning; PE spatial: PE core-G distilled with SAM 2 mask-feature alignment (Perception Encoder paper); not released.
48
+
49
+ Licence evidence: <https://huggingface.co/timm/vit_pe_core_small_patch16_384.fb/blob/255fa76e26fdf8daa47e0d607da46b7051879045/README.md>
50
+
51
+ ## Input size
52
+
53
+ any H x W >= 1, resized or padded inside the model
54
+ Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `pe-core-b16-224` 3000x4000 (2.0 GiB peak allocated, 2.3 reserved); `pe-core-g14-448` 3000x4000 (7.7 GiB peak allocated, 10.2 reserved); `pe-core-l14-336` 3000x4000 (3.5 GiB peak allocated, 3.8 reserved); `pe-core-s16-384` 3000x4000 (1.2 GiB peak allocated, 1.3 reserved); `pe-core-t16-384` 3000x4000 (0.8 GiB peak allocated, 1.1 reserved); `pe-spatial-b16-512` 3000x4000 (2.0 GiB peak allocated, 2.3 reserved); `pe-spatial-g14-448` 1988x2880 (5.6 GiB peak allocated, 5.8 reserved); `pe-spatial-l14-448` 3000x4000 (3.4 GiB peak allocated, 3.7 reserved); `pe-spatial-s16-512` 3000x4000 (1.2 GiB peak allocated, 1.3 reserved); `pe-spatial-t16-512` 3000x4000 (0.8 GiB peak allocated, 1.0 reserved)
55
+
56
+ ## Speed and memory
57
+
58
+ ### Features, synthetic, 224x224
59
+
60
+ | model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
61
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
62
+ | `pe-core-b16-224` | 7.14 | 9.35 | 4.97 | 1.47 | 4.9x | 3.4x | 584 / 620 | 218 / 266 | 218 / 268 |
63
+ | `pe-core-g14-448` | 97.7 | 52.4 | 25.4 | 17.0 | 3.1x | 1.5x | 10848 / 11330 | 3642 / 3670 | 5180 / 5256 |
64
+ | `pe-core-l14-336` | 23.3 | 17.7 | 9.09 | 4.97 | 3.6x | 1.8x | 1848 / 1888 | 650 / 668 | 648 / 672 |
65
+ | `pe-core-s16-384` | 7.02 | 9.28 | 4.80 | 0.957 | 7.3x | 5.0x | 171 / 180 | 83 / 102 | 82 / 104 |
66
+ | `pe-core-t16-384` | 6.74 | 9.04 | 4.68 | 0.683 | 9.9x | 6.9x | 68 / 70 | 48 / 56 | 53 / 82 |
67
+ | `pe-spatial-b16-512` | 7.02 | 8.79 | 4.74 | 1.40 | 5.0x | 3.4x | 544 / 580 | 207 / 226 | 204 / 248 |
68
+ | `pe-spatial-g14-448` | 97.9 | 52.7 | 25.5 | 16.9 | 3.1x | 1.5x | 10688 / 11178 | 3584 / 3612 | 5123 / 5156 |
69
+ | `pe-spatial-l14-448` | 23.0 | 17.2 | 8.84 | 4.87 | 3.5x | 1.8x | 1780 / 1816 | 626 / 668 | 623 / 672 |
70
+ | `pe-spatial-s16-512` | 6.70 | 8.79 | 4.51 | 0.922 | 7.3x | 4.9x | 161 / 178 | 81 / 100 | 78 / 104 |
71
+ | `pe-spatial-t16-512` | 6.56 | 8.73 | 4.48 | 0.666 | 9.8x | 6.7x | 66 / 68 | 49 / 56 | 53 / 82 |
72
+
73
+ ### Startup and first call
74
+
75
+ | model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
76
+ | --- | --- | --- | --- | --- | --- | --- |
77
+ | `pe-core-b16-224` | 14.55 | – | – | 0.19 | 0.29 | 0.07 |
78
+ | `pe-core-g14-448` | 232.98 | – | – | 0.28 | 0.20 | 0.13 |
79
+ | `pe-core-l14-336` | 41.39 | – | – | 0.20 | 0.11 | 0.05 |
80
+ | `pe-core-s16-384` | 5.70 | – | – | 0.18 | 0.10 | 0.04 |
81
+ | `pe-core-t16-384` | 3.20 | – | – | 0.19 | 0.12 | 0.07 |
82
+ | `pe-spatial-b16-512` | 13.65 | – | – | 0.16 | 0.10 | 0.07 |
83
+ | `pe-spatial-g14-448` | 227.90 | – | – | 0.36 | 0.22 | 0.32 |
84
+ | `pe-spatial-l14-448` | 44.48 | – | – | 0.17 | 0.09 | 0.06 |
85
+ | `pe-spatial-s16-512` | 5.40 | – | – | 0.16 | 0.08 | 0.06 |
86
+ | `pe-spatial-t16-512` | 3.09 | – | – | 0.16 | 0.10 | 0.08 |
87
+
88
+ ### torch.compile and TensorRT
89
+
90
+ | model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
91
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
92
+ | `pe-core-b16-224` | 224x224 | 4.97 ms | 2.50 ms | 1.17 ms | 1.47 ms | 8.71 s | 25.8 s | 218 / 266 | 251 / 286 | 218 / 268 | 409 / 460 |
93
+ | `pe-core-g14-448` | 224x224 | 25.4 ms | 24.5 ms | 15.0 ms | 17.0 ms | 29.9 s | 106 s | 3642 / 3670 | 3667 / 3690 | 5180 / 5256 | 7189 / 7230 |
94
+ | `pe-core-l14-336` | 224x224 | 9.09 ms | 6.53 ms | 4.34 ms | 4.97 ms | 13.9 s | 50.1 s | 650 / 668 | 683 / 706 | 648 / 672 | 1260 / 1286 |
95
+ | `pe-core-s16-384` | 224x224 | 4.80 ms | 2.39 ms | 0.581 ms | 0.957 ms | 8.75 s | 36.7 s | 83 / 102 | 113 / 122 | 82 / 104 | 145 / 166 |
96
+ | `pe-core-t16-384` | 224x224 | 4.68 ms | 2.29 ms | 0.432 ms | 0.683 ms | 8.89 s | 36.1 s | 48 / 56 | 78 / 96 | 53 / 82 | 79 / 88 |
97
+ | `pe-spatial-b16-512` | 224x224 | 4.74 ms | 2.67 ms | 1.20 ms | 1.40 ms | 8.67 s | 35.1 s | 207 / 226 | 237 / 266 | 204 / 248 | 397 / 422 |
98
+ | `pe-spatial-g14-448` | 224x224 | 25.5 ms | 24.3 ms | 14.9 ms | 16.9 ms | 28.3 s | 126 s | 3584 / 3612 | 3608 / 3630 | 5123 / 5156 | 7136 / 7156 |
99
+ | `pe-spatial-l14-448` | 224x224 | 8.84 ms | 6.59 ms | 4.35 ms | 4.87 ms | 13.7 s | 53.4 s | 626 / 668 | 658 / 686 | 623 / 672 | 1235 / 1284 |
100
+ | `pe-spatial-s16-512` | 224x224 | 4.51 ms | 2.67 ms | 0.580 ms | 0.922 ms | 8.67 s | 35.3 s | 81 / 100 | 110 / 120 | 78 / 104 | 142 / 144 |
101
+ | `pe-spatial-t16-512` | 224x224 | 4.48 ms | 2.12 ms | 0.426 ms | 0.666 ms | 8.45 s | 33.1 s | 49 / 56 | 77 / 96 | 53 / 82 | 78 / 88 |
102
+
103
+ Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
104
+
105
+ Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
106
+
107
+ ## Accuracy
108
+
109
+ ### On public datasets
110
+
111
+ Two disjoint seeded subsets of the val folders; each image shorter side 256 (bicubic), centre crop 224, the CLS / pooled feature; cosine kNN, vote weighted by similarity. Original = the original code named under Details, same checkpoint and protocol.
112
+
113
+ | variant | dataset | original fp32 | original fp16 | portable | kernels | note |
114
+ |---|---|---|---|---|---|---|
115
+ | `pe-core-s16-384` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 45.56 | 45.54 | 45.48 | 45.38 | |
116
+ | `pe-core-b16-224` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 58.54 | 58.56 | 58.54 | 58.50 | |
117
+ | `pe-core-l14-336` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 65.98 | 66.06 | 65.96 | 66.28 | |
118
+ | `pe-spatial-t16-512` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 5.52 | 5.54 | 5.50 | 5.46 | |
119
+ | `pe-spatial-s16-512` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 8.02 | 8.06 | 8.02 | 7.98 | |
120
+ | `pe-spatial-b16-512` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 13.58 | 13.64 | 13.56 | 13.58 | |
121
+ | `pe-spatial-l14-448` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 17.10 | 17.14 | 17.14 | 17.12 | |
122
+ | `pe-spatial-g14-448` | ImageNet-1k val, kNN top-1 % (k = 20, 10000 labelled / 5000 test images) | 16.36 | 16.30 | 16.32 | 16.30 | |
123
+
124
+ Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 8bce3316 / b2ffd65d / c16d5d41, 2026-11-11 to 2026-11-13.
125
+
126
+ ### Parity with the original code
127
+
128
+ Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0001 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 16 % and p99 <= 68 % of the reference's median |value| (absolute error of the output `last_hidden_state`).
129
+
130
+ ## Limitations
131
+
132
+ - kernels replays a CUDA graph per small input shape without hidden states (up to about 1M activations: ViT-B ~460 px, ViT-L ~300 px), captured on the shape's second call, at most 4 per model; `model.release_graphs()` frees them.
133
+ - Small kernels calls (graph-sized, with or without `layers`) run GEMMs of at most 600 tokens on cuBLAS: faster in a graph replay (ViT-B/16 224 px: 1.26 against 1.52 ms), about 10 % slower eagerly on ViT-T / B; a replay equals the eager call bit for bit.
134
+ - `fp32_accumulate = true` puts every GEMM on cuBLAS with fp32 accumulation (the portable precision; slower on GeForce).
135
+
136
+ ## Usage
137
+
138
+ ```python
139
+ from fastforks import features
140
+
141
+ model = features.load("pe-core-b16-224") # mode="auto": the preferred backend from recorded measurements
142
+ result = model("photo.jpg", size=518)
143
+ result.last_hidden_state # tokens [N, C], as Hugging Face's output
144
+ result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
145
+ ```
146
+
147
+ Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `pe-core-b16-224`, `pe-core-g14-448`, `pe-core-l14-336`, `pe-core-s16-384`, `pe-core-t16-384`, `pe-spatial-b16-512`, `pe-spatial-g14-448`, `pe-spatial-l14-448`, `pe-spatial-s16-512`, `pe-spatial-t16-512`.
148
+
149
+ ## Kernels backend
150
+
151
+ `features.load("pe-core-b16-224", mode="kernels")` runs our CUDA kernels (`cudnn_hacc`, `fvm_conv`, `fvm_extra`, `fvm_mb`, `fvm_r3`). It needs a prebuilt kernel for your GPU architecture (downloaded from <https://huggingface.co/fastforks/kernels> on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.
152
+
153
+ <!-- staged from fastforks 1cbf1671 (2026-11-15) -->