--- library_name: transformers pipeline_tag: image-classification tags: - plankton - ecology - marine-science - long-tail - vision-transformer - environmental-monitoring --- # ViT-LASNet: plankton image classifiers These are the trained weights for **ViT-LASNet**, from *Improving in situ real-time classification of long-tail marine plankton images for ecosystem studies*. The models classify images from the Plankton Imager (Pi-10). Plankton datasets are long-tailed: a few classes, such as detritus and copepods, dominate, and many classes are rare. ViT-LASNet fine-tunes pretrained Vision Transformers with label-aware smoothing to handle this. - **Paper:** [Springer](https://link.springer.com/chapter/10.1007/978-3-031-92387-6_8) - **Code:** [github.com/alan-turing-institute/ViT-LASNet](https://github.com/alan-turing-institute/ViT-LASNet) - **Project page:** [alan-turing-institute.github.io/ViT-LASNet](https://alan-turing-institute.github.io/ViT-LASNet/) ## Models Each model is in its own sub-folder. | Sub-folder | Architecture | Classes | Training loss | | --- | --- | --- | --- | | `vit-mislas` | ViT-Base/16 (`google/vit-base-patch16-224`) | 18 | Dynamic label-aware smoothing (`MiSLAS_vit`), lr 5e-5, 30 epochs | | `beit-balce` | BEiT-Base/16 (`microsoft/beit-base-patch16-224-pt22k-ft22k`) | 18 | Logit-adjusted cross-entropy (`Bal_CE`), lr 5e-5, 20 epochs | | `vit-balce-3class` | ViT-Base/16 | 3 | Logit-adjusted cross-entropy (`Bal_CE`), lr 5e-5, 10 epochs | | `resnet18-18class-rgb` | ResNet18 (torchvision) | 18 | Baseline | | `resnet18-3class-rgb` | ResNet18 (torchvision) | 3 | Baseline | **18 classes:** Detritus, Phyto_diatom, Phyto_diatom_chaetocerotanae_Chaetoceros, Phyto_diatom_rhisoleniales_Guinardia flaccida, Phyto_diatom_rhisoleniales_Rhizosolenia, Phyto_dinoflagellate_gonyaulacales_Tripos, Phyto_dinoflagellate_gonyaulacales_Tripos macroceros, Phyto_dinoflagellate_gonyaulacales_Tripos muelleri, Zoo_cnidaria, Zoo_crustacea_copepod, Zoo_crustacea_copepod_calanoida, Zoo_crustacea_copepod_calanoida_Acartia, Zoo_crustacea_copepod_calanoida_Centropages, Zoo_crustacea_copepod_cyclopoida, Zoo_crustacea_copepod_cyclopoida_Oithona, Zoo_crustacea_copepod_nauplii, Zoo_other, Zoo_tintinnidae. **3 classes:** copepod, detritus, noncopepod. ## Usage ### ViT / BEiT (transformers) ```python from PIL import Image from transformers import AutoImageProcessor, ViTForImageClassification repo = "Noushineft/ViT-LASNet" processor = AutoImageProcessor.from_pretrained(repo, subfolder="vit-mislas") model = ViTForImageClassification.from_pretrained(repo, subfolder="vit-mislas").eval() image = Image.open("plankton.tif").convert("RGB") logits = model(**processor(images=image, return_tensors="pt")).logits print(model.config.id2label[logits.argmax(-1).item()]) ``` For BEiT, use `BeitForImageClassification` with `subfolder="beit-balce"`. ### ResNet18 (torchvision + safetensors) ```python import torch, torchvision from huggingface_hub import hf_hub_download from safetensors.torch import load_file path = hf_hub_download("Noushineft/ViT-LASNet", "resnet18-18class-rgb/model.safetensors") model = torchvision.models.resnet18() model.fc = torch.nn.Linear(model.fc.in_features, 18) model.load_state_dict(load_file(path)) model.eval() ``` The ResNet models take RGB images resized to 256×256 and scaled to [0, 1], without ImageNet normalisation. Each ResNet folder's `config.json` lists the class order. ### Command line The `test/test.py` script in the code repository can load these models directly: ```bash python test/test.py -m 2 -n 18 -w Noushineft/ViT-LASNet/vit-mislas -f path/to/images -o predictions.csv ``` ## Training data The models were trained on Pi-10 plankton images collected in situ, labelled into the classes above. The dataset is not distributed with these weights. ## Limitations - The models were trained on images from one instrument (Pi-10) and its imaging conditions. Expect lower accuracy on other imagers or regions without fine-tuning. - Rare classes have few training examples, so their predictions are less reliable than those for common classes. - These are research models and have not been validated for operational monitoring. ## Citation If you use these models, please cite the paper linked above.