README.md

metadata

pipeline_tag: image-classification
library_name: fgir-zoo
tags:
  - fine-grained
  - ultra-fine-grained
  - image-classification
  - vision-transformer
  - parameter-efficient
  - resnet
  - adapter
  - petl

FGIR-ViT

Vision-transformer checkpoints for ultra-fine-grained image recognition (UFGIR) — telling apart cultivars within a species (soybean, cotton and other leaf sets) where classes differ by tiny visual margins and samples per class are scarce. Almost every checkpoint is an ImageNet-21k ViT-B/16 with the backbone frozen; only a small set of adapter modules is trained, so a model is a 340 MB file of which ~2 M parameters were actually learned. The ila_saw_resnet_448 group applies SAW to a frozen timm ResNet-50 instead (96 MB files).

The repository gathers the checkpoints behind two lines of work from the ERIS Lab:

• ILA — the down-sampling inter-layer adapter, and SAW, a semantic-adaptation warmup added on top of it (the ila_* groups). ILA was introduced in the ECCV 2024 EFM workshop paper below.
• AAA — the attention-aggregation adapter, which replaces ILA's convolutional down-sampler with a bottleneck attention that fixes the adapter's misplaced attention (the aaa_* groups).

The full fine-tuning FGIR benchmark (FGIRFT) is now a separate repository, ERISLab/FGIRFT.

847 checkpoints, one seed per configuration, last epoch only. Everything is described in manifest.csv and loadable by name with the standalone fgir_zoo library (no training repo needed).

Part of the ERISLab FGIR-ViT collection.

Training recipe

Backbone vit_b16 (ImageNet-21k), frozen. The trainable part is the adapter family named in the filename (a plain PETL adapter, ILA's down-sampling adapter, or the AAA attention-aggregation adapter); the SAW runs add a supervised-contrastive warmup stage before the classifier is trained. Images are ImageNet-normalized and evaluated at 224, 448 or 672 px. Each embedded config (an argparse namespace saved in the checkpoint) records the exact settings.

What each folder holds

Files are grouped by experiment line and named <dataset>_<model>_<method>_<serial>[_<variant>].pth; the trailing number is the run's wandb serial, and the name ends in it because only the last epoch is kept (so "last" is implied). <variant> is a resolution tag (_is224/448/672) where one serial spans several resolutions.

Group	Line / paper	Backbone	What it is
ila_main_224 / ila_main_448	ILA/SAW	vit_b16	Main UFGIR comparison of ILA against PETL and FGIR baselines, at 224 / 448
ila_ablations	ILA/SAW	vit_b16	ILA design ablations
ila_saw, ila_saw_deit, ila_saw_no_ila	ILA/SAW	vit_b16 / pedeit	The SAW contrastive-warmup runs
ila_swin, ila_dinov2, ila_foundation	ILA/SAW	peswin / pevit	ILA on other backbones and frozen foundation baselines
ila_equal_compute	ILA/SAW	vit_b16	Equal-compute control
deit_grid_448	ILA/SAW	pedeit / pedeit3	DeiT and DeiT III at 448: frozen, adapter and ILA++ arms on the ten UFGIR sets
dinov2_adaptformer_224	ILA/SAW	pevit (DINOv2 ViT-B/14)	DINOv2 at 224: frozen head (3021), AdaptFormer (3002), AdaptFormer + ILA (3031)
saw_birds_448	ILA/SAW	vit_b16	NABirds adapter baseline at 448 (3112)
saw_resnet_warmup	ILA/SAW	peresnet50	The two ResNet-50 SAW warmups on the pooled leaves (291 adapter, 294 ILA)
ila_saw_resnet_448	ILA/SAW	peresnet50	SAW on a frozen ResNet-50 at 448: the SAW arms, their no-SAW and linear-probe baselines, and the two SAW warmups
aaa_main_224 / aaa_main_448 / aaa_main_672	AAA	vit_b16	The AAA finals (frozen / adapter / ILA++ / AAA arms), by resolution
aaa_ablations	AAA	vit_b16	AAA ablations on SoyGlobal and SoyLocal: insertion blocks (424/428/430, 224 px) and width (411/413/415, 448 px)
aaa_fungi	AAA (breadth)	vit_b16	FungiTastic breadth probe — kept for completeness, not a paper result
medical_idrid, medical_petl	AAA (medical)	vit_b16	Medical-imaging PETL and IDRiD resolution runs

The ILA/SAW paper evaluates the ila_main, ila_saw and ila_saw_resnet_448 serials; the AAA paper evaluates the aaa_main_* serials. manifest.csv maps every file to its dataset, method, serial, resolution, num_classes, top-1 accuracy and sha256.

SAW on ResNet-50 (ila_saw_resnet_448)

The CNN counterpart of the SAW ladder, for the SAW paper in IEEE Transactions on AgriFood Electronics. The backbone is timm's resnet50, frozen, at 448 px (peresnet50); the adapter is a 1x1 bottleneck ConvAdapter on every ResNet block's output, and the ILA variant adds the down-sampling adapter before the last stage. The ten datasets are cotton, soyageing, soyageingr1/r3/r4/r5/r6, soygene, soyglobal and soylocal.

Serial	Role	method in manifest.csv	Files
292	Method: adapter + SAW	cls_adapter_saw_fz	10
295	Method: adapter + ILA + SAW	ila_dso_cls_adapter_saw_fz	10
290	Baseline: adapter	cls_adapter_fz	10
293	Baseline: adapter + ILA	ila_dso_cls_adapter_fz	10
661	Baseline: FZ, frozen linear probe	fz	10
291	SAW stage-1 warmup for 292	cls_adapter_supcon_fz	1
294	SAW stage-1 warmup for 295	ila_dso_cls_adapter_supcon_fz	1

• 292 trains the adapter and classifier on each dataset, starting from the 291 warmup.
• 295 does the same with the ILA down-sampler, starting from the 294 warmup.
• 290 trains the same adapter from a fresh initialisation, without SAW.
• 293 trains adapter + ILA from a fresh initialisation, without SAW.
• 661 (FZ) keeps the whole backbone frozen and trains only a linear head. All ten datasets are released; wandb nycu_pcs/AAA holds all 30 runs.
• 291 and 294 are the supervised-contrastive warmups on the leaves set (100 epochs). Their head is a projection, so the forward returns a 2048-d embedding and the stored accuracy (0.03 and 0.00) is not a classification score.

The filenames of SAW and no-SAW runs differ only in the serial (_292 against _290), so the manifest's method marks SAW with _saw_, read from the run's warmup ckpt_path. Each file's accuracy is its own single seed (seed 3 for the fine-tunes, seed 2 for soygene 661, seed 1 for the warmups). The papers' three-seed means are in ILA_SAW_Analysis/results_all/acc/resnet_ladder.csv; over the ten datasets they average 56.05 (290), 64.64 (292), 54.30 (293) and 65.01 (295), and 30.73 for FZ.

A real forward on one SoyLocal test image with the 295 checkpoint:

import torch, torchvision.transforms as T
from PIL import Image
import fgir_zoo

model = fgir_zoo.load('fgirvit/ila_saw_resnet_448/soylocal_peresnet50_ila_dso_cls_adapter_fz_295')
tf = T.Compose([T.Resize(600), T.CenterCrop(448), T.ToTensor(),   # config.test_resize_size = 600
                T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
x = tf(Image.open('soylocal/soybean200_square/2_6_test.png').convert('RGB')).unsqueeze(0)
with torch.no_grad():
    probs = model(x).softmax(-1)        # -> (1, 200)
pred = probs.argmax(-1).item()
# pred = 1, confidence 0.905 — the true class is 1, so this sample is correct.

On the first 12 SoyLocal test images this checkpoint is right 3 times; its stored test accuracy on the full split is 49.8%.

Load a checkpoint

Standalone, no fgir_vit install — the model code is vendored inside fgir_zoo:

pip install git+https://github.com/arkel23/fgir-zoo.git

import fgir_zoo
model = fgir_zoo.load('fgirvit/aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403')
model.eval()
# fgir_zoo.list_models(family='fgirvit') lists every name; model.config holds the run's settings.

Or fetch the file directly and load it yourself:

import torch, torchvision.transforms as T
from PIL import Image
from huggingface_hub import hf_hub_download

path = hf_hub_download('ERISLab/FGIR-ViT',
                       'aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403.pth')
ckpt = torch.load(path, map_location='cpu', weights_only=False)   # dict: config, model, accuracy, epoch

A real forward on one SoyLocal test image, with the model loaded above (ImageNet-normalized, 448 px):

tf = T.Compose([T.Resize(600), T.CenterCrop(448), T.ToTensor(),   # config.test_resize_size = 600
                T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
x = tf(Image.open('soylocal/soybean200_square/1_2_test.png').convert('RGB')).unsqueeze(0)
logits = model(x)                       # -> (1, 200)
pred = logits.softmax(-1).argmax(-1).item()
# pred = 0, confidence 0.318 — the true class is 0, so this sample is correct.

On the first 12 SoyLocal test images this checkpoint is right 6 times, matching its 55.5% test accuracy on the full split.

Results (each file's own score, not the paper's aggregate)

SoyLocal (200 cultivars) top-1, the released single-seed checkpoints, by arm and resolution:

Arm	224	448	672
Frozen	28.0	45.3	27.5
Adapter	33.0	55.7	36.3
ILA++	54.7	55.2	54.0
AAA	45.7	55.5	56.7

These are individual checkpoints on one dataset and one seed; they are noisier than the papers' means over ten datasets and three seeds. The papers report the aggregate (ILA with SAW averages 73.5% at 448 across the ten UFGIR sets). Every per-file score is the accuracy column of manifest.csv.

Superseded and non-paper runs

Kept for completeness but not printed in any paper: aaa_fungi (on FungiTastic the plain adapter beats AAA, so it does not carry the method) and the medical breadth runs. Learning-rate searches, extra seeds, best-epoch copies and exploratory ablations were removed during consolidation and are not in this repository.

Requirements

• torch>=2.5, timm==0.9.12 (the checkpoints were trained on it and the vendored model code forks that version — newer timm changes the ViT internals), huggingface_hub, safetensors, einops, ml_collections.
• peresnet50 needs fgir_zoo with the FGIR-ViT model code at commit aabb221 or later.
• The 15 ila_saw SAW contrastive-head checkpoints carry an older projection-head layout; their manifest loads is False and fgir_zoo refuses them rather than loading loosely.

Citation

The published ILA method:

@inproceedings{rios2024ila,
  title     = {Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient
               Ultra-Fine-Grained Image Recognition},
  author    = {Rios, Edwin Arkel and Oyerinde, Femiloye and Hu, Min-Chun and Lai, Bo-Cheng},
  booktitle = {Efficient Deep Learning for Foundation Models (EFM) Workshop at the European
               Conference on Computer Vision (ECCV)},
  year      = {2024}
}

The SAW extension and the AAA (attention-aggregation adapter) paper are under submission; this card will be updated with their citations when they are available.