Home model card (LisTAya style): overview, loading, results, citation

README.md

@@ -1,70 +1,153 @@
-# FGIR-ViT checkpoint zoo
-
-One deduplicated folder holding every kept checkpoint from the FGIR-ViT family of experiments:
-the ILA inter-layer adapter and SAW contrastive-warmup paper (`ila_*`), the attention-aggregation
-adapter (AAA) paper (`aaa_*`), the medical PETL and IDRiD studies (`medical_*`), and the
-full-fine-tune fine-grained benchmark (`ft_*`). It backs the `fgirvit` family of the `fgir_zoo`
-library and the HuggingFace mirror `ERISLab/FGIR-ViT`.
-
-931 checkpoints, one seed per configuration, last epoch only.
-
-## Layout
-
-Files are grouped by experiment line, flat inside each group, named
-`<dataset>_<model>_<method>_<serial>.pth`. The serial is the wandb serial, and the name ends in it
-because only the last epoch is kept, so "last" is implied and never written into the name.
-
-| Group | Origin | wandb project | Serials | Backbone | Files |
-|---|---|---|---|---|---|
-| `ila_main_224` | ILA/SAW paper | ILA | 1 | vit_b16 | 206 |
-| `ila_main_448` | ILA/SAW paper | ILA | 3 | vit_b16 | 185 |
-| `ila_ablations` | ILA/SAW paper | ILA | 4–18 | vit_b16 | 33 |
-| `ila_saw` | ILA/SAW paper | ILA | 90–93, 210–260 | vit_b16 | 36 |
-| `ila_saw_deit` | ILA/SAW paper | ILA | 71, 81 | pedeit_base_patch16_224.fb_in1k | 20 |
-| `ila_saw_no_ila` | ILA/SAW paper | ILA | 270–272 | vit_b16 | 21 |
-| `ila_swin` | ILA/SAW paper | ILA | 80, 81, 161 | peswin_base_patch4_window7_224.ms_in22k | 31 |
-| `ila_dinov2` | ILA/SAW paper | ILA | 27, 70, 71 | pevit_base_patch14_dinov2.lvd142m | 9 |
-| `ila_foundation` | ILA/SAW paper | ila-backbones-baselines | 100/200/300 | MAE, DINOv2, CLIP pevit | 6 |
-| `ila_equal_compute` | ILA/SAW paper | ILA | 150 | vit_b16 | 10 |
-| `aaa_main_224` | AAA paper | ILA | 401 | vit_b16 | 13 |
-| `aaa_main_448` | AAA paper | ILA | 403 | vit_b16 | 15 |
-| `aaa_main_672` | AAA paper | AAA | 501 | vit_b16 | 40 |
-| `aaa_fungi` | AAA paper (breadth) | AAA | 561 | vit_b16 | 3 |
-| `medical_idrid` | AAA medical | AAA | 541 | vit_b16 | 6 |
-| `medical_petl` | AAA medical | AAA | 1 | vit_b16 | 55 |
-| `ft_224` | FGIR benchmark | FGIRFT | 1 | vit_b16 | 111 |
-| `ft_448` | FGIR benchmark | FGIRFT | 3 | vit_b16 | 131 |
-
-The wandb project shown is the current one. Some older configs carry a legacy `project_name` in
-their embedded config (`MLA` for the ILA main tables, `PEFGIR` for part of the 224 benchmark);
-grouping here follows the experiment line, not that field.
-
-`aaa_main_672` holds the four adapter arms (attention-aggregation, ILA++ conv, plain adapter,
-frozen) across ten datasets. `ila_foundation` holds the frozen MAE/DINOv2/CLIP baselines
-(launched as serials 100/200/300, renumbered to 29). The AAA finals (`aaa_main_224`/`448`) carry
-`save_images`, so attention visualisations are produced from the method itself. `aaa_fungi` is the
-FungiTastic-FewShot breadth probe (the AAA arm only, one seed, all three resolutions under serial
-561, the resolution kept in the `_is224/448/672` suffix); it is not printed in the paper — on fungi
-the plain adapter beats AAA.
-
-## Contents of a checkpoint
-
-Each file is a dict with `config` (the run's full argument namespace), `model` (the state dict),
-`accuracy` (top-1 on the test set) and `epoch`. Load it with `torch.load(path,
-weights_only=False)`; the embedded `config` drives model construction.
-
-## Loading by name
-
-Install the standalone library and load any checkpoint by its `<group>/<stem>` name — no
-`fgir_vit` install needed:
+---
+pipeline_tag: image-classification
+library_name: fgir-zoo
+tags:
+- fine-grained
+- ultra-fine-grained
+- image-classification
+- vision-transformer
+- parameter-efficient
+- adapter
+- petl
+---
 
-```
+# FGIR-ViT
+
+Vision-transformer checkpoints for **ultra-fine-grained image recognition (UFGIR)** — telling apart
+cultivars *within* a species (soybean, cotton and other leaf sets) where classes differ by tiny
+visual margins and samples per class are scarce. Every checkpoint is an ImageNet-21k **ViT-B/16 with
+the backbone frozen**; only a small set of adapter modules is trained, so a model is a ~340 MB file
+of which ~2 M parameters were actually learned.
+
+The repository gathers the checkpoints behind three lines of work from the ERIS Lab:
+
+- **ILA** — the down-sampling inter-layer adapter, and **SAW**, a semantic-adaptation warmup added on
+  top of it (the `ila_*` groups). ILA was introduced in the ECCV 2024 EFM workshop paper below.
+- **AAA** — the attention-aggregation adapter, which replaces ILA's convolutional down-sampler with a
+  bottleneck attention that fixes the adapter's misplaced attention (the `aaa_*` groups).
+- **FGIRFT** — a full fine-tuning FGIR benchmark on 21 datasets, for reference (the `ft_*` groups).
+
+931 checkpoints, one seed per configuration, last epoch only. Everything is described in
+`manifest.csv` and loadable by name with the standalone `fgir_zoo` library (no training repo needed).
+
+## Training recipe
+
+Backbone `vit_b16` (ImageNet-21k), frozen. The trainable part is the adapter family named in the
+filename (a plain PETL adapter, ILA's down-sampling adapter, or the AAA attention-aggregation
+adapter); the SAW runs add a supervised-contrastive warmup stage before the classifier is trained.
+Images are ImageNet-normalized and evaluated at 224, 448 or 672 px. Each embedded `config` (an
+argparse namespace saved in the checkpoint) records the exact settings.
+
+## What each folder holds
+
+Files are grouped by experiment line and named `<dataset>_<model>_<method>_<serial>[_<variant>].pth`;
+the trailing number is the run's wandb serial, and the name ends in it because only the last epoch is
+kept (so "last" is implied). `<variant>` is a resolution tag (`_is224/448/672`) where one serial
+spans several resolutions.
+
+| Group | Line / paper | Backbone | What it is |
+|---|---|---|---|
+| `ila_main_224` / `ila_main_448` | ILA/SAW | vit_b16 | Main UFGIR comparison of ILA against PETL and FGIR baselines, at 224 / 448 |
+| `ila_ablations` | ILA/SAW | vit_b16 | ILA design ablations |
+| `ila_saw`, `ila_saw_deit`, `ila_saw_no_ila` | ILA/SAW | vit_b16 / pedeit | The SAW contrastive-warmup runs |
+| `ila_swin`, `ila_dinov2`, `ila_foundation` | ILA/SAW | peswin / pevit | ILA on other backbones and frozen foundation baselines |
+| `ila_equal_compute` | ILA/SAW | vit_b16 | Equal-compute control |
+| `aaa_main_224` / `aaa_main_448` / `aaa_main_672` | AAA | vit_b16 | The AAA finals (frozen / adapter / ILA++ / AAA arms), by resolution |
+| `aaa_fungi` | AAA (breadth) | vit_b16 | FungiTastic breadth probe — kept for completeness, not a paper result |
+| `medical_idrid`, `medical_petl` | AAA (medical) | vit_b16 | Medical-imaging PETL and IDRiD resolution runs |
+| `ft_224` / `ft_448` | FGIRFT | vit_b16 | Full fine-tunes with FGIR heads (GLSim, MAWS, PSM, CAL, MPNCOV, …) on 21 datasets |
+
+The ILA/SAW paper evaluates the `ila_main` and `ila_saw` serials; the AAA paper evaluates the
+`aaa_main_*` serials. `manifest.csv` maps every file to its dataset, method, serial, resolution,
+`num_classes`, top-1 accuracy and sha256.
+
+## Load a checkpoint
+
+Standalone, no `fgir_vit` install — the model code is vendored inside `fgir_zoo`:
+
+```bash
 pip install git+https://github.com/arkel23/fgir-zoo.git
-python -c "import fgir_zoo; m = fgir_zoo.load('fgirvit/aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403')"
 ```
 
-`fgir_zoo.list_models(family='fgirvit')` lists the names; the row set is `manifest.csv`.
+```python
+import fgir_zoo
+model = fgir_zoo.load('fgirvit/aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403')
+model.eval()
+# fgir_zoo.list_models(family='fgirvit') lists every name; model.config holds the run's settings.
+```
+
+Or fetch the file directly and load it yourself:
+
+```python
+import torch, torchvision.transforms as T
+from PIL import Image
+from huggingface_hub import hf_hub_download
+
+path = hf_hub_download('ERISLab/FGIR-ViT',
+                       'aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403.pth')
+ckpt = torch.load(path, map_location='cpu', weights_only=False)   # dict: config, model, accuracy, epoch
+```
+
+A real forward on one SoyLocal test image, with the model loaded above (ImageNet-normalized,
+448 px):
+
+```python
+tf = T.Compose([T.Resize(600), T.CenterCrop(448), T.ToTensor(),   # config.test_resize_size = 600
+                T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
+x = tf(Image.open('soylocal/soybean200_square/1_2_test.png').convert('RGB')).unsqueeze(0)
+logits = model(x)                       # -> (1, 200)
+pred = logits.softmax(-1).argmax(-1).item()
+# pred = 0, confidence 0.318 — the true class is 0, so this sample is correct.
+```
+
+On the first 12 SoyLocal test images this checkpoint is right 6 times, matching its 55.5% test
+accuracy on the full split.
+
+## Results (each file's own score, not the paper's aggregate)
+
+SoyLocal (200 cultivars) top-1, the released single-seed checkpoints, by arm and resolution:
+
+| Arm | 224 | 448 | 672 |
+|---|---|---|---|
+| Frozen | 28.0 | 45.3 | 27.5 |
+| Adapter | 33.0 | 55.7 | 36.3 |
+| ILA++ | 54.7 | 55.2 | 54.0 |
+| AAA | 45.7 | 55.5 | 56.7 |
+
+These are individual checkpoints on one dataset and one seed; they are noisier than the papers'
+means over ten datasets and three seeds. The papers report the aggregate (ILA with SAW averages
+73.5% at 448 across the ten UFGIR sets). Every per-file score is the `accuracy` column of
+`manifest.csv`.
+
+## Superseded and non-paper runs
+
+Kept for completeness but not printed in any paper: `aaa_fungi` (on FungiTastic the plain adapter
+beats AAA, so it does not carry the method) and the medical breadth runs. Learning-rate searches,
+extra seeds, best-epoch copies and exploratory ablations were removed during consolidation and are
+not in this repository.
+
+## Requirements
+
+- `torch>=2.5`, `timm==0.9.12` (the checkpoints were trained on it and the vendored model code forks
+  that version — newer timm changes the ViT internals), `huggingface_hub`, `safetensors`,
+  `einops`, `ml_collections`.
+- The 15 `ila_saw` SAW contrastive-head checkpoints carry an older projection-head layout; their
+  manifest `loads` is `False` and `fgir_zoo` refuses them rather than loading loosely.
+
+## Citation
+
+The published ILA method:
+
+```bibtex
+@inproceedings{rios2024ila,
+  title     = {Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient
+               Ultra-Fine-Grained Image Recognition},
+  author    = {Rios, Edwin Arkel and Oyerinde, Femiloye and Hu, Min-Chun and Lai, Bo-Cheng},
+  booktitle = {Efficient Deep Learning for Foundation Models (EFM) Workshop at the European
+               Conference on Computer Vision (ECCV)},
+  year      = {2024}
+}
+```
 
-The 15 SAW contrastive-head checkpoints in `ila_saw` (serial 93 and the warmup probes) carry an
-older `ifc` layout the current code cannot build; their manifest rows are marked `loads=False` and
-`create_model` refuses them rather than loading loosely.
+The SAW extension and the AAA (attention-aggregation adapter) paper are under submission; this card
+will be updated with their citations when they are available.