Instructions to use Noushineft/ViT-LASNet with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Noushineft/ViT-LASNet with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-classification", model="Noushineft/ViT-LASNet") pipe("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/parrots.png")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Noushineft/ViT-LASNet", device_map="auto") - Notebooks
- Google Colab
- Kaggle
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Noushineft/ViT-LASNet", device_map="auto")ViT-LASNet: plankton image classifiers
These are the trained weights for ViT-LASNet, from Improving in situ real-time classification of long-tail marine plankton images for ecosystem studies. The models classify images from the Plankton Imager (Pi-10). Plankton datasets are long-tailed: a few classes, such as detritus and copepods, dominate, and many classes are rare. ViT-LASNet fine-tunes pretrained Vision Transformers with label-aware smoothing to handle this.
- Paper: Springer
- Code: github.com/alan-turing-institute/ViT-LASNet
- Project page: alan-turing-institute.github.io/ViT-LASNet
Models
Each model is in its own sub-folder.
| Sub-folder | Architecture | Classes | Training loss |
|---|---|---|---|
vit-mislas |
ViT-Base/16 (google/vit-base-patch16-224) |
18 | Dynamic label-aware smoothing (MiSLAS_vit), lr 5e-5, 30 epochs |
beit-balce |
BEiT-Base/16 (microsoft/beit-base-patch16-224-pt22k-ft22k) |
18 | Logit-adjusted cross-entropy (Bal_CE), lr 5e-5, 20 epochs |
vit-balce-3class |
ViT-Base/16 | 3 | Logit-adjusted cross-entropy (Bal_CE), lr 5e-5, 10 epochs |
resnet18-18class-rgb |
ResNet18 (torchvision) | 18 | Baseline |
resnet18-3class-rgb |
ResNet18 (torchvision) | 3 | Baseline |
18 classes: Detritus, Phyto_diatom, Phyto_diatom_chaetocerotanae_Chaetoceros, Phyto_diatom_rhisoleniales_Guinardia flaccida, Phyto_diatom_rhisoleniales_Rhizosolenia, Phyto_dinoflagellate_gonyaulacales_Tripos, Phyto_dinoflagellate_gonyaulacales_Tripos macroceros, Phyto_dinoflagellate_gonyaulacales_Tripos muelleri, Zoo_cnidaria, Zoo_crustacea_copepod, Zoo_crustacea_copepod_calanoida, Zoo_crustacea_copepod_calanoida_Acartia, Zoo_crustacea_copepod_calanoida_Centropages, Zoo_crustacea_copepod_cyclopoida, Zoo_crustacea_copepod_cyclopoida_Oithona, Zoo_crustacea_copepod_nauplii, Zoo_other, Zoo_tintinnidae.
3 classes: copepod, detritus, noncopepod.
Usage
ViT / BEiT (transformers)
from PIL import Image
from transformers import AutoImageProcessor, ViTForImageClassification
repo = "Noushineft/ViT-LASNet"
processor = AutoImageProcessor.from_pretrained(repo, subfolder="vit-mislas")
model = ViTForImageClassification.from_pretrained(repo, subfolder="vit-mislas").eval()
image = Image.open("plankton.tif").convert("RGB")
logits = model(**processor(images=image, return_tensors="pt")).logits
print(model.config.id2label[logits.argmax(-1).item()])
For BEiT, use BeitForImageClassification with subfolder="beit-balce".
ResNet18 (torchvision + safetensors)
import torch, torchvision
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
path = hf_hub_download("Noushineft/ViT-LASNet", "resnet18-18class-rgb/model.safetensors")
model = torchvision.models.resnet18()
model.fc = torch.nn.Linear(model.fc.in_features, 18)
model.load_state_dict(load_file(path))
model.eval()
The ResNet models take RGB images resized to 256ร256 and scaled to [0, 1], without ImageNet normalisation. Each ResNet folder's config.json lists the class order.
Command line
The test/test.py script in the code repository can load these models directly:
python test/test.py -m 2 -n 18 -w Noushineft/ViT-LASNet/vit-mislas -f path/to/images -o predictions.csv
Training data
The models were trained on Pi-10 plankton images collected in situ, labelled into the classes above. The dataset is not distributed with these weights.
Limitations
- The models were trained on images from one instrument (Pi-10) and its imaging conditions. Expect lower accuracy on other imagers or regions without fine-tuning.
- Rare classes have few training examples, so their predictions are less reliable than those for common classes.
- These are research models and have not been validated for operational monitoring.
Citation
If you use these models, please cite the paper linked above.
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-classification", model="Noushineft/ViT-LASNet") pipe("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/parrots.png")