Hugging Face's logo

ERISLab
/
TokenReductionPT

Image Classification
PyTorch
timm
fine-grained-image-recognition
token-reduction
vision-transformer
Model card Files
xet
 Community
xedwin23x commited on
Commit
c6e79ef
·
verified ·
1 Parent(s): aa21246

Add model card and manifest

Browse files
Files changed (2) hide show
  1. README.md +189 -0
  2. manifest.csv +0 -0
README.md ADDED
@@ -0,0 +1,189 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: image-classification
3
+ library_name: pytorch
4
+ tags:
5
+ - fine-grained-image-recognition
6
+ - image-classification
7
+ - token-reduction
8
+ - vision-transformer
9
+ - timm
10
+ ---
11
+
12
+ # TokenReductionPT: token reduction and cross-layer cache aggregation for fine-grained recognition
13
+
14
+ These are the checkpoints behind *Cross-Layer Cache Aggregation for Token Reduction in
15
+ Ultra-Fine-Grained Image Recognition* ([arXiv:2501.00243](https://arxiv.org/abs/2501.00243)),
16
+ ICASSP 2025, and behind its extension, in preparation. Vision transformers are fine-tuned with a
17
+ token-reduction method (EViT, TopK, DynamicViT, ATS, ToMe, PatchMerger, SiT, DPC-KNN and the
18
+ FGIR-aware MAWS, DMAWS, EDAR, NFEDAR and GLSF criteria), with and without cross-layer cache
19
+ aggregation (CLCA). CLCA adds a cross-layer aggregation head (CLA) and a
20
+ cross-layer cache (CLC) that keeps information from discarded tokens. Code:
21
+ [arkel23/TokenReductionPT](https://github.com/arkel23/TokenReductionPT).
22
+
23
+ 2656 checkpoints, one per configuration, each the last epoch of one training run. Each file
24
+ is a `torch.save` dict with `config` (the full training configuration), `model` (the state dict),
25
+ `accuracy` and `epoch`, with no optimizer state. File names are the runs' experiment-log names:
26
+ dataset, model (reduction method as prefix), keep rate, `cla` for the aggregation head, `clc_1`
27
+ for the cache, and the serial. Load them with [fgir-zoo](https://github.com/arkel23/fgir-zoo). The
28
+ [collection](https://huggingface.co/collections/ERISLab/clca-cross-layer-cache-aggregation-icassp-2025-6ab46269c88fe64198165089) groups this repo with the ICASSP
29
+ 2025 paper.
30
+
31
+ ## Layout
32
+
33
+ One folder per serial. `setting` is `bl` (no CLCA), `cla` (aggregation head only) or `clca` (head
34
+ and cache). "Extension" is the extension of the ICASSP 2025 paper, in preparation.
35
+
36
+ | Folder | What | Used in | Files | Mean accuracy |
37
+ |---|---|---|---|---|
38
+ | `serial_30` | CLCA main results: EViT and the plain ViT-B, nine pretrainings, bl/cla/clca, 448 px | ICASSP 2025, extension | 405 | 58.04 |
39
+ | `serial_31` | Plain ViT-B, two pretrainings, bl/cla, on CUB and Aircraft, 448 px | ICASSP 2025, extension | 8 | 89.74 |
40
+ | `serial_15` | 13 reduction methods x bl/clca, DeiT-T, 224 px | extension | 702 | 65.39 |
41
+ | `serial_16` | 13 reduction methods x bl/clca, DeiT-S, 224 px | extension | 702 | 71.68 |
42
+ | `serial_62` | 13 reduction methods x bl/clca, DeiT-T, 448 px | extension | 702 | 73.37 |
43
+ | `serial_408` | TopK, DeiT-T, CUB, 288 px, keep rates 0.25-1.0 | extension | 4 | 77.01 |
44
+ | `serial_20` | DMAWS with CLCA, six ViT-B pretrainings, cotton and soybean datasets, 448 px | extension | 93 | 75.47 |
45
+ | `serial_21` | ImageNet-1k: EViT and the five FGIR-aware criteria, DeiT-T/S/B, 224 px | extension | 40 | 74.25 |
46
+
47
+ `manifest.csv` lists every file with its dataset, model, method, backbone, keep rate, setting,
48
+ serial, seed, image size, class count, accuracy, the papers that use it, SHA-256 and size. The
49
+ 224 px ablations of the ICASSP 2025 paper have no released checkpoints.
50
+
51
+ ## Load a checkpoint and classify an image
52
+
53
+ ```python
54
+ import torch
55
+ from PIL import Image
56
+ from torchvision import transforms
57
+ from fgir_zoo import tokenred
58
+
59
+ model = tokenred.create_model('serial_62/cub_evit_deit_tiny_patch16_224.fb_in1k_0.7_cla_clc_1_62')
60
+ cfg = model.config
61
+ # evaluation at 448 px resizes to a square without cropping
62
+ tf = transforms.Compose([
63
+ transforms.Resize((cfg.input_size, cfg.input_size),
64
+ interpolation=transforms.InterpolationMode.BICUBIC),
65
+ transforms.ToTensor(),
66
+ transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
67
+ ])
68
+ # a CUB-200-2011 test image, class index 50 (051.Horned_Grebe)
69
+ x = tf(Image.open('Horned_Grebe_0050_34561.jpg').convert('RGB')).unsqueeze(0)
70
+ with torch.no_grad():
71
+ logits = model(x)
72
+ print(logits.argmax(-1).item(), logits.softmax(-1).max().item()) # 50 0.9315
73
+ ```
74
+
75
+ ## Accuracy of the released checkpoints
76
+
77
+ Top-1 accuracy (%) is the run's own test accuracy after the last epoch, as logged in training.
78
+ The papers average three seeds; where a configuration ran with several seeds, the released file
79
+ is the seed with the highest accuracy, so the papers' tables can differ slightly. Per-file
80
+ accuracy is in `manifest.csv`; 1 file has no recorded accuracy
81
+ (`serial_16/cub_edar_deit_small_patch16_224.fb_in1k_0.7_16`). The table below covers `serial_30`, the ICASSP
82
+ 2025 main results; method `none` is the plain ViT without token reduction.
83
+
84
+ | Method, keep rate, setting | Backbone | cotton | soyageing | soygene | soyglobal | soylocal |
85
+ |---|---|---|---|---|---|---|
86
+ | evit 0.1 bl | `deit3_base_patch16_224.fb_in1k` | 18.33 | 66.48 | 57.99 | 30.00 | 29.17 |
87
+ | evit 0.1 bl | `deit3_base_patch16_224.fb_in22k_ft_in1k` | 20.83 | 64.36 | 38.25 | 12.13 | 21.00 |
88
+ | evit 0.1 bl | `deit_base_patch16_224.fb_in1k` | 22.08 | 71.70 | 56.81 | 16.94 | 33.67 |
89
+ | evit 0.1 bl | `vit_base_patch16_224.dino` | 27.08 | 72.57 | 47.86 | 29.72 | 29.00 |
90
+ | evit 0.1 bl | `vit_base_patch16_224.in1k_mocov3` | 28.75 | 66.10 | 40.89 | 3.85 | 29.50 |
91
+ | evit 0.1 bl | `vit_base_patch16_224.mae` | 10.42 | 22.59 | 37.31 | 3.89 | 18.00 |
92
+ | evit 0.1 bl | `vit_base_patch16_224.orig_in21k` | 35.42 | 73.29 | 57.10 | 39.03 | 33.67 |
93
+ | evit 0.1 bl | `vit_base_patch16_224_miil.in21k` | 17.50 | 69.52 | 49.02 | 21.57 | 27.67 |
94
+ | evit 0.1 bl | `vit_base_patch16_clip_224.laion2b` | 16.67 | 58.95 | 22.42 | 3.90 | 26.50 |
95
+ | evit 0.1 cla | `deit3_base_patch16_224.fb_in1k` | 46.25 | 86.36 | 76.15 | 60.27 | 51.67 |
96
+ | evit 0.1 cla | `deit3_base_patch16_224.fb_in22k_ft_in1k` | 51.67 | 87.21 | 77.17 | 51.48 | 50.50 |
97
+ | evit 0.1 cla | `deit_base_patch16_224.fb_in1k` | 40.42 | 83.15 | 71.96 | 46.94 | 48.33 |
98
+ | evit 0.1 cla | `vit_base_patch16_224.dino` | 52.08 | 85.23 | 70.52 | 58.57 | 47.33 |
99
+ | evit 0.1 cla | `vit_base_patch16_224.in1k_mocov3` | 40.00 | 81.03 | 66.44 | 41.76 | 44.17 |
100
+ | evit 0.1 cla | `vit_base_patch16_224.mae` | 43.33 | 78.73 | 66.72 | 58.00 | 17.83 |
101
+ | evit 0.1 cla | `vit_base_patch16_224.orig_in21k` | 55.83 | 85.45 | 70.82 | 41.73 | 51.67 |
102
+ | evit 0.1 cla | `vit_base_patch16_224_miil.in21k` | 46.25 | 79.88 | 64.35 | 52.92 | 52.50 |
103
+ | evit 0.1 cla | `vit_base_patch16_clip_224.laion2b` | 55.00 | 87.68 | 72.70 | 65.62 | 59.67 |
104
+ | evit 0.1 clca | `deit3_base_patch16_224.fb_in1k` | 61.25 | 87.31 | 69.35 | 61.99 | 57.50 |
105
+ | evit 0.1 clca | `deit3_base_patch16_224.fb_in22k_ft_in1k` | 56.67 | 88.42 | 76.72 | 62.56 | 53.67 |
106
+ | evit 0.1 clca | `deit_base_patch16_224.fb_in1k` | 53.33 | 85.60 | 74.65 | 59.03 | 56.83 |
107
+ | evit 0.1 clca | `vit_base_patch16_224.dino` | 54.17 | 85.49 | 71.61 | 60.80 | 48.67 |
108
+ | evit 0.1 clca | `vit_base_patch16_224.in1k_mocov3` | 49.17 | 82.89 | 69.15 | 49.48 | 47.50 |
109
+ | evit 0.1 clca | `vit_base_patch16_224.mae` | 25.42 | 85.54 | 65.34 | 24.25 | 42.67 |
110
+ | evit 0.1 clca | `vit_base_patch16_224.orig_in21k` | 54.58 | 83.78 | 71.34 | 48.74 | 51.67 |
111
+ | evit 0.1 clca | `vit_base_patch16_224_miil.in21k` | 55.00 | 85.07 | 71.95 | 55.71 | 51.17 |
112
+ | evit 0.1 clca | `vit_base_patch16_clip_224.laion2b` | 54.58 | 89.35 | 76.97 | 58.10 | 62.17 |
113
+ | evit 0.7 bl | `deit3_base_patch16_224.fb_in1k` | 37.50 | 76.81 | 66.72 | 29.38 | 33.83 |
114
+ | evit 0.7 bl | `deit3_base_patch16_224.fb_in22k_ft_in1k` | 35.00 | 73.41 | 53.06 | 6.60 | 32.67 |
115
+ | evit 0.7 bl | `deit_base_patch16_224.fb_in1k` | 54.58 | 82.26 | 69.75 | 33.35 | 47.83 |
116
+ | evit 0.7 bl | `vit_base_patch16_224.dino` | 40.42 | 76.95 | 55.75 | 33.20 | 41.00 |
117
+ | evit 0.7 bl | `vit_base_patch16_224.in1k_mocov3` | 55.83 | 76.22 | 58.58 | 2.10 | 46.33 |
118
+ | evit 0.7 bl | `vit_base_patch16_224.mae` | 15.42 | 73.41 | 20.88 | 21.93 | 36.33 |
119
+ | evit 0.7 bl | `vit_base_patch16_224.orig_in21k` | 59.17 | 79.64 | 60.28 | 39.54 | 46.33 |
120
+ | evit 0.7 bl | `vit_base_patch16_224_miil.in21k` | 40.00 | 76.32 | 48.94 | 32.01 | 37.33 |
121
+ | evit 0.7 bl | `vit_base_patch16_clip_224.laion2b` | 7.50 | 51.37 | 15.25 | 8.89 | 9.67 |
122
+ | evit 0.7 cla | `deit3_base_patch16_224.fb_in1k` | 62.08 | 88.77 | 80.97 | 58.51 | 55.33 |
123
+ | evit 0.7 cla | `deit3_base_patch16_224.fb_in22k_ft_in1k` | 61.67 | 86.89 | 76.80 | 53.58 | 53.83 |
124
+ | evit 0.7 cla | `deit_base_patch16_224.fb_in1k` | 67.92 | 87.82 | 79.40 | 59.15 | 58.83 |
125
+ | evit 0.7 cla | `vit_base_patch16_224.dino` | 68.33 | 88.40 | 73.26 | 60.75 | 57.83 |
126
+ | evit 0.7 cla | `vit_base_patch16_224.in1k_mocov3` | 65.83 | 86.83 | 74.32 | 53.13 | 58.33 |
127
+ | evit 0.7 cla | `vit_base_patch16_224.mae` | 59.58 | 90.67 | 77.47 | 51.38 | 57.83 |
128
+ | evit 0.7 cla | `vit_base_patch16_224.orig_in21k` | 58.75 | 86.51 | 71.75 | 55.26 | 55.17 |
129
+ | evit 0.7 cla | `vit_base_patch16_224_miil.in21k` | 59.58 | 87.09 | 73.39 | 64.00 | 59.00 |
130
+ | evit 0.7 cla | `vit_base_patch16_clip_224.laion2b` | 60.83 | 89.96 | 40.57 | 57.24 | 61.00 |
131
+ | evit 0.7 clca | `deit3_base_patch16_224.fb_in1k` | 66.67 | 88.08 | 80.18 | 59.86 | 55.83 |
132
+ | evit 0.7 clca | `deit3_base_patch16_224.fb_in22k_ft_in1k` | 62.50 | 86.51 | 73.76 | 52.05 | 59.00 |
133
+ | evit 0.7 clca | `deit_base_patch16_224.fb_in1k` | 65.83 | 88.65 | 78.93 | 62.37 | 60.17 |
134
+ | evit 0.7 clca | `vit_base_patch16_224.dino` | 67.08 | 88.14 | 75.12 | 64.65 | 59.17 |
135
+ | evit 0.7 clca | `vit_base_patch16_224.in1k_mocov3` | 65.83 | 87.15 | 75.26 | 55.90 | 59.00 |
136
+ | evit 0.7 clca | `vit_base_patch16_224.mae` | 57.92 | 90.51 | 74.34 | 41.40 | 54.83 |
137
+ | evit 0.7 clca | `vit_base_patch16_224.orig_in21k` | 64.17 | 85.84 | 74.65 | 56.81 | 57.33 |
138
+ | evit 0.7 clca | `vit_base_patch16_224_miil.in21k` | 65.00 | 86.59 | 72.46 | 60.17 | 58.83 |
139
+ | evit 0.7 clca | `vit_base_patch16_clip_224.laion2b` | 68.33 | 91.27 | 79.48 | 62.64 | 62.83 |
140
+ | none bl | `deit3_base_patch16_224.fb_in1k` | 47.92 | 76.40 | 66.23 | 26.32 | 33.83 |
141
+ | none bl | `deit3_base_patch16_224.fb_in22k_ft_in1k` | 35.00 | 72.59 | 52.88 | 15.38 | 32.33 |
142
+ | none bl | `deit_base_patch16_224.fb_in1k` | 56.25 | 81.90 | 72.84 | 39.34 | 44.00 |
143
+ | none bl | `vit_base_patch16_224.dino` | 43.75 | 75.56 | 55.42 | 35.72 | 39.50 |
144
+ | none bl | `vit_base_patch16_224.in1k_mocov3` | 55.42 | 76.71 | 56.56 | 1.86 | 46.00 |
145
+ | none bl | `vit_base_patch16_224.mae` | 28.75 | 74.12 | 30.82 | 10.56 | 22.50 |
146
+ | none bl | `vit_base_patch16_224.orig_in21k` | 58.75 | 79.70 | 61.90 | 37.39 | 49.00 |
147
+ | none bl | `vit_base_patch16_224_miil.in21k` | 31.67 | 75.21 | 46.06 | 35.31 | 36.50 |
148
+ | none bl | `vit_base_patch16_clip_224.laion2b` | 7.08 | 40.95 | 12.69 | 9.61 | 14.17 |
149
+ | none cla | `deit3_base_patch16_224.fb_in1k` | 66.67 | 87.74 | 80.88 | 58.82 | 57.00 |
150
+ | none cla | `deit3_base_patch16_224.fb_in22k_ft_in1k` | 62.08 | 85.58 | 75.52 | 52.22 | 51.67 |
151
+ | none cla | `deit_base_patch16_224.fb_in1k` | 66.25 | 87.49 | 80.36 | 63.83 | 57.83 |
152
+ | none cla | `vit_base_patch16_224.dino` | 68.33 | 88.02 | 75.62 | 61.30 | 57.83 |
153
+ | none cla | `vit_base_patch16_224.in1k_mocov3` | 67.50 | 86.69 | 76.13 | 55.35 | 59.33 |
154
+ | none cla | `vit_base_patch16_224.mae` | 65.00 | 90.04 | 77.23 | 51.41 | 60.50 |
155
+ | none cla | `vit_base_patch16_224.orig_in21k` | 65.00 | 87.17 | 72.01 | 56.66 | 57.83 |
156
+ | none cla | `vit_base_patch16_224_miil.in21k` | 60.42 | 87.19 | 74.59 | 64.52 | 56.17 |
157
+ | none cla | `vit_base_patch16_clip_224.laion2b` | 53.33 | 90.93 | 76.59 | 55.21 | 59.33 |
158
+ | none clca | `deit3_base_patch16_224.fb_in1k` | 67.50 | 88.04 | 80.71 | 61.75 | 59.50 |
159
+ | none clca | `deit3_base_patch16_224.fb_in22k_ft_in1k` | 59.58 | 87.21 | 79.00 | 59.43 | 50.33 |
160
+ | none clca | `deit_base_patch16_224.fb_in1k` | 69.58 | 88.79 | 81.04 | 62.49 | 63.17 |
161
+ | none clca | `vit_base_patch16_224.dino` | 68.33 | 88.08 | 72.49 | 64.74 | 59.50 |
162
+ | none clca | `vit_base_patch16_224.in1k_mocov3` | 65.00 | 87.37 | 76.76 | 57.29 | 58.83 |
163
+ | none clca | `vit_base_patch16_224.mae` | 52.92 | 90.32 | 76.42 | 53.32 | 60.50 |
164
+ | none clca | `vit_base_patch16_224.orig_in21k` | 65.00 | 86.26 | 74.55 | 58.46 | 57.00 |
165
+ | none clca | `vit_base_patch16_224_miil.in21k` | 61.67 | 87.17 | 74.73 | 61.30 | 56.83 |
166
+ | none clca | `vit_base_patch16_clip_224.laion2b` | 65.83 | 90.71 | 77.13 | 62.95 | 63.50 |
167
+
168
+ ## Requirements
169
+
170
+ - `fgir-zoo` (`pip install git+https://github.com/arkel23/fgir-zoo.git`), which pins
171
+ `timm==0.9.12`
172
+ - `torch` (checked with 2.5.1)
173
+
174
+ ## Citation
175
+
176
+ ```bibtex
177
+ @inproceedings{rios_cross-layer_2025,
178
+ title = {Cross-Layer Cache Aggregation for Token Reduction in Ultra-Fine-Grained Image Recognition},
179
+ author = {Rios, Edwin Arkel and Yuanda, Jansen Christopher and Ghanz, Vincent Leon and
180
+ Yu, Cheng-Wei and Lai, Bo-Cheng and Hu, Min-Chun},
181
+ booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
182
+ year = {2025},
183
+ eprint = {2501.00243},
184
+ archivePrefix = {arXiv},
185
+ primaryClass = {cs.CV},
186
+ doi = {10.48550/arXiv.2501.00243},
187
+ url = {https://arxiv.org/abs/2501.00243}
188
+ }
189
+ ```
manifest.csv ADDED
The diff for this file is too large to render. See raw diff