FGIR-ViT

Vision-transformer checkpoints for ultra-fine-grained image recognition (UFGIR) — telling apart cultivars within a species (soybean, cotton and other leaf sets) where classes differ by tiny visual margins and samples per class are scarce. Every checkpoint is an ImageNet-21k ViT-B/16 with the backbone frozen; only a small set of adapter modules is trained, so a model is a ~340 MB file of which ~2 M parameters were actually learned.

The repository gathers the checkpoints behind two lines of work from the ERIS Lab:

• ILA — the down-sampling inter-layer adapter, and SAW, a semantic-adaptation warmup added on top of it (the ila_* groups). ILA was introduced in the ECCV 2024 EFM workshop paper below.
• AAA — the attention-aggregation adapter, which replaces ILA's convolutional down-sampler with a bottleneck attention that fixes the adapter's misplaced attention (the aaa_* groups).

The full fine-tuning FGIR benchmark (FGIRFT) is now a separate repository, ERISLab/FGIRFT.

689 checkpoints, one seed per configuration, last epoch only. Everything is described in manifest.csv and loadable by name with the standalone fgir_zoo library (no training repo needed).

Part of the ERISLab FGIR-ViT collection.

Training recipe

Backbone vit_b16 (ImageNet-21k), frozen. The trainable part is the adapter family named in the filename (a plain PETL adapter, ILA's down-sampling adapter, or the AAA attention-aggregation adapter); the SAW runs add a supervised-contrastive warmup stage before the classifier is trained. Images are ImageNet-normalized and evaluated at 224, 448 or 672 px. Each embedded config (an argparse namespace saved in the checkpoint) records the exact settings.

What each folder holds

Files are grouped by experiment line and named <dataset>_<model>_<method>_<serial>[_<variant>].pth; the trailing number is the run's wandb serial, and the name ends in it because only the last epoch is kept (so "last" is implied). <variant> is a resolution tag (_is224/448/672) where one serial spans several resolutions.

Group	Line / paper	Backbone	What it is
ila_main_224 / ila_main_448	ILA/SAW	vit_b16	Main UFGIR comparison of ILA against PETL and FGIR baselines, at 224 / 448
ila_ablations	ILA/SAW	vit_b16	ILA design ablations
ila_saw, ila_saw_deit, ila_saw_no_ila	ILA/SAW	vit_b16 / pedeit	The SAW contrastive-warmup runs
ila_swin, ila_dinov2, ila_foundation	ILA/SAW	peswin / pevit	ILA on other backbones and frozen foundation baselines
ila_equal_compute	ILA/SAW	vit_b16	Equal-compute control
aaa_main_224 / aaa_main_448 / aaa_main_672	AAA	vit_b16	The AAA finals (frozen / adapter / ILA++ / AAA arms), by resolution
aaa_fungi	AAA (breadth)	vit_b16	FungiTastic breadth probe — kept for completeness, not a paper result
medical_idrid, medical_petl	AAA (medical)	vit_b16	Medical-imaging PETL and IDRiD resolution runs

The ILA/SAW paper evaluates the ila_main and ila_saw serials; the AAA paper evaluates the aaa_main_* serials. manifest.csv maps every file to its dataset, method, serial, resolution, num_classes, top-1 accuracy and sha256.

Load a checkpoint

Standalone, no fgir_vit install — the model code is vendored inside fgir_zoo:

pip install git+https://github.com/arkel23/fgir-zoo.git

import fgir_zoo
model = fgir_zoo.load('fgirvit/aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403')
model.eval()
# fgir_zoo.list_models(family='fgirvit') lists every name; model.config holds the run's settings.

Or fetch the file directly and load it yourself:

import torch, torchvision.transforms as T
from PIL import Image
from huggingface_hub import hf_hub_download

path = hf_hub_download('ERISLab/FGIR-ViT',
                       'aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403.pth')
ckpt = torch.load(path, map_location='cpu', weights_only=False)   # dict: config, model, accuracy, epoch

A real forward on one SoyLocal test image, with the model loaded above (ImageNet-normalized, 448 px):

tf = T.Compose([T.Resize(600), T.CenterCrop(448), T.ToTensor(),   # config.test_resize_size = 600
                T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
x = tf(Image.open('soylocal/soybean200_square/1_2_test.png').convert('RGB')).unsqueeze(0)
logits = model(x)                       # -> (1, 200)
pred = logits.softmax(-1).argmax(-1).item()
# pred = 0, confidence 0.318 — the true class is 0, so this sample is correct.

On the first 12 SoyLocal test images this checkpoint is right 6 times, matching its 55.5% test accuracy on the full split.

Results (each file's own score, not the paper's aggregate)

SoyLocal (200 cultivars) top-1, the released single-seed checkpoints, by arm and resolution:

Arm	224	448	672
Frozen	28.0	45.3	27.5
Adapter	33.0	55.7	36.3
ILA++	54.7	55.2	54.0
AAA	45.7	55.5	56.7

These are individual checkpoints on one dataset and one seed; they are noisier than the papers' means over ten datasets and three seeds. The papers report the aggregate (ILA with SAW averages 73.5% at 448 across the ten UFGIR sets). Every per-file score is the accuracy column of manifest.csv.

Superseded and non-paper runs

Kept for completeness but not printed in any paper: aaa_fungi (on FungiTastic the plain adapter beats AAA, so it does not carry the method) and the medical breadth runs. Learning-rate searches, extra seeds, best-epoch copies and exploratory ablations were removed during consolidation and are not in this repository.

Requirements

• torch>=2.5, timm==0.9.12 (the checkpoints were trained on it and the vendored model code forks that version — newer timm changes the ViT internals), huggingface_hub, safetensors, einops, ml_collections.
• The 15 ila_saw SAW contrastive-head checkpoints carry an older projection-head layout; their manifest loads is False and fgir_zoo refuses them rather than loading loosely.

Citation

The published ILA method:

@inproceedings{rios2024ila,
  title     = {Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient
               Ultra-Fine-Grained Image Recognition},
  author    = {Rios, Edwin Arkel and Oyerinde, Femiloye and Hu, Min-Chun and Lai, Bo-Cheng},
  booktitle = {Efficient Deep Learning for Foundation Models (EFM) Workshop at the European
               Conference on Computer Vision (ECCV)},
  year      = {2024}
}

The SAW extension and the AAA (attention-aggregation adapter) paper are under submission; this card will be updated with their citations when they are available.