README.md

---
pipeline_tag: image-classification
library_name: fgir-zoo
tags:
- fine-grained
- ultra-fine-grained
- image-classification
- vision-transformer
- parameter-efficient
- resnet
- adapter
- petl
---

# FGIR-ViT

Vision-transformer checkpoints for **ultra-fine-grained image recognition (UFGIR)** — telling apart
cultivars *within* a species (soybean, cotton and other leaf sets) where classes differ by tiny
visual margins and samples per class are scarce. Almost every checkpoint is an ImageNet-21k
**ViT-B/16 with the backbone frozen**; only a small set of adapter modules is trained, so a model is a
~340 MB file of which ~2 M parameters were actually learned. The `ila_saw_resnet_448` group applies
SAW to a frozen timm ResNet-50 instead (~96 MB files).

The repository gathers the checkpoints behind two lines of work from the ERIS Lab:

- **ILA** — the down-sampling inter-layer adapter, and **SAW**, a semantic-adaptation warmup added on
  top of it (the `ila_*` groups). ILA was introduced in the ECCV 2024 EFM workshop paper below.
- **AAA** — the attention-aggregation adapter, which replaces ILA's convolutional down-sampler with a
  bottleneck attention that fixes the adapter's misplaced attention (the `aaa_*` groups).

The full fine-tuning FGIR benchmark (FGIRFT) is now a separate repository,
[ERISLab/FGIRFT](https://huggingface.co/ERISLab/FGIRFT).

847 checkpoints, one seed per configuration, last epoch only. Everything is described in
`manifest.csv` and loadable by name with the standalone `fgir_zoo` library (no training repo needed).

Part of the [ERISLab FGIR-ViT collection](https://huggingface.co/collections/ERISLab/fgir-vit-ila-saw-and-aaa-6ab2dc3a730388cfe7a5f7b4).

## Training recipe

Backbone `vit_b16` (ImageNet-21k), frozen. The trainable part is the adapter family named in the
filename (a plain PETL adapter, ILA's down-sampling adapter, or the AAA attention-aggregation
adapter); the SAW runs add a supervised-contrastive warmup stage before the classifier is trained.
Images are ImageNet-normalized and evaluated at 224, 448 or 672 px. Each embedded `config` (an
argparse namespace saved in the checkpoint) records the exact settings.

## What each folder holds

Files are grouped by experiment line and named `<dataset>_<model>_<method>_<serial>[_<variant>].pth`;
the trailing number is the run's wandb serial, and the name ends in it because only the last epoch is
kept (so "last" is implied). `<variant>` is a resolution tag (`_is224/448/672`) where one serial
spans several resolutions.

| Group | Line / paper | Backbone | What it is |
|---|---|---|---|
| `ila_main_224` / `ila_main_448` | ILA/SAW | vit_b16 | Main UFGIR comparison of ILA against PETL and FGIR baselines, at 224 / 448 |
| `ila_ablations` | ILA/SAW | vit_b16 | ILA design ablations |
| `ila_saw`, `ila_saw_deit`, `ila_saw_no_ila` | ILA/SAW | vit_b16 / pedeit | The SAW contrastive-warmup runs |
| `ila_swin`, `ila_dinov2`, `ila_foundation` | ILA/SAW | peswin / pevit | ILA on other backbones and frozen foundation baselines |
| `ila_equal_compute` | ILA/SAW | vit_b16 | Equal-compute control |
| `deit_grid_448` | ILA/SAW | pedeit / pedeit3 | DeiT and DeiT III at 448: frozen, adapter and ILA++ arms on the ten UFGIR sets |
| `dinov2_adaptformer_224` | ILA/SAW | pevit (DINOv2 ViT-B/14) | DINOv2 at 224: frozen head (3021), AdaptFormer (3002), AdaptFormer + ILA (3031) |
| `saw_birds_448` | ILA/SAW | vit_b16 | NABirds adapter baseline at 448 (3112) |
| `saw_resnet_warmup` | ILA/SAW | peresnet50 | The two ResNet-50 SAW warmups on the pooled leaves (291 adapter, 294 ILA) |
| `ila_saw_resnet_448` | ILA/SAW | peresnet50 | SAW on a frozen ResNet-50 at 448: the SAW arms, their no-SAW and linear-probe baselines, and the two SAW warmups |
| `aaa_main_224` / `aaa_main_448` / `aaa_main_672` | AAA | vit_b16 | The AAA finals (frozen / adapter / ILA++ / AAA arms), by resolution |
| `aaa_ablations` | AAA | vit_b16 | AAA ablations on SoyGlobal and SoyLocal: insertion blocks (424/428/430, 224 px) and width (411/413/415, 448 px) |
| `aaa_fungi` | AAA (breadth) | vit_b16 | FungiTastic breadth probe — kept for completeness, not a paper result |
| `medical_idrid`, `medical_petl` | AAA (medical) | vit_b16 | Medical-imaging PETL and IDRiD resolution runs |

The ILA/SAW paper evaluates the `ila_main`, `ila_saw` and `ila_saw_resnet_448` serials; the AAA paper evaluates the
`aaa_main_*` serials. `manifest.csv` maps every file to its dataset, method, serial, resolution,
`num_classes`, top-1 accuracy and sha256.

## SAW on ResNet-50 (`ila_saw_resnet_448`)

The CNN counterpart of the SAW ladder, for the SAW paper in IEEE Transactions on AgriFood
Electronics. The backbone is timm's `resnet50`, frozen, at 448 px (`peresnet50`); the adapter is a
1x1 bottleneck `ConvAdapter` on every ResNet block's output, and the ILA variant adds the
down-sampling adapter before the last stage. The ten datasets are cotton, soyageing,
soyageingr1/r3/r4/r5/r6, soygene, soyglobal and soylocal.

| Serial | Role | `method` in `manifest.csv` | Files |
|---|---|---|---|
| 292 | Method: adapter + SAW | `cls_adapter_saw_fz` | 10 |
| 295 | Method: adapter + ILA + SAW | `ila_dso_cls_adapter_saw_fz` | 10 |
| 290 | Baseline: adapter | `cls_adapter_fz` | 10 |
| 293 | Baseline: adapter + ILA | `ila_dso_cls_adapter_fz` | 10 |
| 661 | Baseline: FZ, frozen linear probe | `fz` | 10 |
| 291 | SAW stage-1 warmup for 292 | `cls_adapter_supcon_fz` | 1 |
| 294 | SAW stage-1 warmup for 295 | `ila_dso_cls_adapter_supcon_fz` | 1 |

- **292** trains the adapter and classifier on each dataset, starting from the 291 warmup.
- **295** does the same with the ILA down-sampler, starting from the 294 warmup.
- **290** trains the same adapter from a fresh initialisation, without SAW.
- **293** trains adapter + ILA from a fresh initialisation, without SAW.
- **661** (FZ) keeps the whole backbone frozen and trains only a linear head. All ten datasets
  are released; wandb `nycu_pcs/AAA` holds all 30 runs.
- **291** and **294** are the supervised-contrastive warmups on the leaves set (100 epochs). Their
  head is a projection, so the forward returns a 2048-d embedding and the stored `accuracy`
  (0.03 and 0.00) is not a classification score.

The filenames of SAW and no-SAW runs differ only in the serial (`_292` against `_290`), so the
manifest's `method` marks SAW with `_saw_`, read from the run's warmup `ckpt_path`. Each file's
`accuracy` is its own single seed (seed 3 for the fine-tunes, seed 2 for soygene 661, seed 1 for
the warmups). The papers' three-seed means
are in `ILA_SAW_Analysis/results_all/acc/resnet_ladder.csv`; over the ten datasets they average
56.05 (290), 64.64 (292), 54.30 (293) and 65.01 (295), and 30.73 for FZ.

A real forward on one SoyLocal test image with the 295 checkpoint:

```python
import torch, torchvision.transforms as T
from PIL import Image
import fgir_zoo

model = fgir_zoo.load('fgirvit/ila_saw_resnet_448/soylocal_peresnet50_ila_dso_cls_adapter_fz_295')
tf = T.Compose([T.Resize(600), T.CenterCrop(448), T.ToTensor(),   # config.test_resize_size = 600
                T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
x = tf(Image.open('soylocal/soybean200_square/2_6_test.png').convert('RGB')).unsqueeze(0)
with torch.no_grad():
    probs = model(x).softmax(-1)        # -> (1, 200)
pred = probs.argmax(-1).item()
# pred = 1, confidence 0.905 — the true class is 1, so this sample is correct.
```

On the first 12 SoyLocal test images this checkpoint is right 3 times; its stored test accuracy on
the full split is 49.8%.

## Load a checkpoint

Standalone, no `fgir_vit` install — the model code is vendored inside `fgir_zoo`:

```bash
pip install git+https://github.com/arkel23/fgir-zoo.git
```

```python
import fgir_zoo
model = fgir_zoo.load('fgirvit/aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403')
model.eval()
# fgir_zoo.list_models(family='fgirvit') lists every name; model.config holds the run's settings.
```

Or fetch the file directly and load it yourself:

```python
import torch, torchvision.transforms as T
from PIL import Image
from huggingface_hub import hf_hub_download

path = hf_hub_download('ERISLab/FGIR-ViT',
                       'aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403.pth')
ckpt = torch.load(path, map_location='cpu', weights_only=False)   # dict: config, model, accuracy, epoch
```

A real forward on one SoyLocal test image, with the model loaded above (ImageNet-normalized,
448 px):

```python
tf = T.Compose([T.Resize(600), T.CenterCrop(448), T.ToTensor(),   # config.test_resize_size = 600
                T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
x = tf(Image.open('soylocal/soybean200_square/1_2_test.png').convert('RGB')).unsqueeze(0)
logits = model(x)                       # -> (1, 200)
pred = logits.softmax(-1).argmax(-1).item()
# pred = 0, confidence 0.318 — the true class is 0, so this sample is correct.
```

On the first 12 SoyLocal test images this checkpoint is right 6 times, matching its 55.5% test
accuracy on the full split.

## Results (each file's own score, not the paper's aggregate)

SoyLocal (200 cultivars) top-1, the released single-seed checkpoints, by arm and resolution:

| Arm | 224 | 448 | 672 |
|---|---|---|---|
| Frozen | 28.0 | 45.3 | 27.5 |
| Adapter | 33.0 | 55.7 | 36.3 |
| ILA++ | 54.7 | 55.2 | 54.0 |
| AAA | 45.7 | 55.5 | 56.7 |

These are individual checkpoints on one dataset and one seed; they are noisier than the papers'
means over ten datasets and three seeds. The papers report the aggregate (ILA with SAW averages
73.5% at 448 across the ten UFGIR sets). Every per-file score is the `accuracy` column of
`manifest.csv`.

## Superseded and non-paper runs

Kept for completeness but not printed in any paper: `aaa_fungi` (on FungiTastic the plain adapter
beats AAA, so it does not carry the method) and the medical breadth runs. Learning-rate searches,
extra seeds, best-epoch copies and exploratory ablations were removed during consolidation and are
not in this repository.

## Requirements

- `torch>=2.5`, `timm==0.9.12` (the checkpoints were trained on it and the vendored model code forks
  that version — newer timm changes the ViT internals), `huggingface_hub`, `safetensors`,
  `einops`, `ml_collections`.
- `peresnet50` needs `fgir_zoo` with the FGIR-ViT model code at commit `aabb221` or later.
- The 15 `ila_saw` SAW contrastive-head checkpoints carry an older projection-head layout; their
  manifest `loads` is `False` and `fgir_zoo` refuses them rather than loading loosely.

## Citation

The published ILA method:

```bibtex
@inproceedings{rios2024ila,
  title     = {Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient
               Ultra-Fine-Grained Image Recognition},
  author    = {Rios, Edwin Arkel and Oyerinde, Femiloye and Hu, Min-Chun and Lai, Bo-Cheng},
  booktitle = {Efficient Deep Learning for Foundation Models (EFM) Workshop at the European
               Conference on Computer Vision (ECCV)},
  year      = {2024}
}
```

The SAW extension and the AAA (attention-aggregation adapter) paper are under submission; this card
will be updated with their citations when they are available.