ViT-LASNet: plankton image classifiers

These are the trained weights for ViT-LASNet, from Improving in situ real-time classification of long-tail marine plankton images for ecosystem studies. The models classify images from the Plankton Imager (Pi-10). Plankton datasets are long-tailed: a few classes, such as detritus and copepods, dominate, and many classes are rare. ViT-LASNet fine-tunes pretrained Vision Transformers with label-aware smoothing to handle this.

Models

Each model is in its own sub-folder.

Sub-folder Architecture Classes Training loss
vit-mislas ViT-Base/16 (google/vit-base-patch16-224) 18 Dynamic label-aware smoothing (MiSLAS_vit), lr 5e-5, 30 epochs
beit-balce BEiT-Base/16 (microsoft/beit-base-patch16-224-pt22k-ft22k) 18 Logit-adjusted cross-entropy (Bal_CE), lr 5e-5, 20 epochs
vit-balce-3class ViT-Base/16 3 Logit-adjusted cross-entropy (Bal_CE), lr 5e-5, 10 epochs
resnet18-18class-rgb ResNet18 (torchvision) 18 Baseline
resnet18-3class-rgb ResNet18 (torchvision) 3 Baseline

18 classes: Detritus, Phyto_diatom, Phyto_diatom_chaetocerotanae_Chaetoceros, Phyto_diatom_rhisoleniales_Guinardia flaccida, Phyto_diatom_rhisoleniales_Rhizosolenia, Phyto_dinoflagellate_gonyaulacales_Tripos, Phyto_dinoflagellate_gonyaulacales_Tripos macroceros, Phyto_dinoflagellate_gonyaulacales_Tripos muelleri, Zoo_cnidaria, Zoo_crustacea_copepod, Zoo_crustacea_copepod_calanoida, Zoo_crustacea_copepod_calanoida_Acartia, Zoo_crustacea_copepod_calanoida_Centropages, Zoo_crustacea_copepod_cyclopoida, Zoo_crustacea_copepod_cyclopoida_Oithona, Zoo_crustacea_copepod_nauplii, Zoo_other, Zoo_tintinnidae.

3 classes: copepod, detritus, noncopepod.

Usage

ViT / BEiT (transformers)

from PIL import Image
from transformers import AutoImageProcessor, ViTForImageClassification

repo = "Noushineft/ViT-LASNet"
processor = AutoImageProcessor.from_pretrained(repo, subfolder="vit-mislas")
model = ViTForImageClassification.from_pretrained(repo, subfolder="vit-mislas").eval()

image = Image.open("plankton.tif").convert("RGB")
logits = model(**processor(images=image, return_tensors="pt")).logits
print(model.config.id2label[logits.argmax(-1).item()])

For BEiT, use BeitForImageClassification with subfolder="beit-balce".

ResNet18 (torchvision + safetensors)

import torch, torchvision
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

path = hf_hub_download("Noushineft/ViT-LASNet", "resnet18-18class-rgb/model.safetensors")
model = torchvision.models.resnet18()
model.fc = torch.nn.Linear(model.fc.in_features, 18)
model.load_state_dict(load_file(path))
model.eval()

The ResNet models take RGB images resized to 256ร—256 and scaled to [0, 1], without ImageNet normalisation. Each ResNet folder's config.json lists the class order.

Command line

The test/test.py script in the code repository can load these models directly:

python test/test.py -m 2 -n 18 -w Noushineft/ViT-LASNet/vit-mislas -f path/to/images -o predictions.csv

Training data

The models were trained on Pi-10 plankton images collected in situ, labelled into the classes above. The dataset is not distributed with these weights.

Limitations

  • The models were trained on images from one instrument (Pi-10) and its imaging conditions. Expect lower accuracy on other imagers or regions without fine-tuning.
  • Rare classes have few training examples, so their predictions are less reliable than those for common classes.
  • These are research models and have not been validated for operational monitoring.

Citation

If you use these models, please cite the paper linked above.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support