File size: 4,293 Bytes
7a189b2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
---
library_name: transformers
pipeline_tag: image-classification
tags:
  - plankton
  - ecology
  - marine-science
  - long-tail
  - vision-transformer
  - environmental-monitoring
---

# ViT-LASNet: plankton image classifiers

These are the trained weights for **ViT-LASNet**, from *Improving in situ real-time classification of long-tail marine plankton images for ecosystem studies*. The models classify images from the Plankton Imager (Pi-10). Plankton datasets are long-tailed: a few classes, such as detritus and copepods, dominate, and many classes are rare. ViT-LASNet fine-tunes pretrained Vision Transformers with label-aware smoothing to handle this.

- **Paper:** [Springer](https://link.springer.com/chapter/10.1007/978-3-031-92387-6_8)
- **Code:** [github.com/alan-turing-institute/ViT-LASNet](https://github.com/alan-turing-institute/ViT-LASNet)
- **Project page:** [alan-turing-institute.github.io/ViT-LASNet](https://alan-turing-institute.github.io/ViT-LASNet/)

## Models

Each model is in its own sub-folder.

| Sub-folder | Architecture | Classes | Training loss |
| --- | --- | --- | --- |
| `vit-mislas` | ViT-Base/16 (`google/vit-base-patch16-224`) | 18 | Dynamic label-aware smoothing (`MiSLAS_vit`), lr 5e-5, 30 epochs |
| `beit-balce` | BEiT-Base/16 (`microsoft/beit-base-patch16-224-pt22k-ft22k`) | 18 | Logit-adjusted cross-entropy (`Bal_CE`), lr 5e-5, 20 epochs |
| `vit-balce-3class` | ViT-Base/16 | 3 | Logit-adjusted cross-entropy (`Bal_CE`), lr 5e-5, 10 epochs |
| `resnet18-18class-rgb` | ResNet18 (torchvision) | 18 | Baseline |
| `resnet18-3class-rgb` | ResNet18 (torchvision) | 3 | Baseline |

**18 classes:** Detritus, Phyto_diatom, Phyto_diatom_chaetocerotanae_Chaetoceros, Phyto_diatom_rhisoleniales_Guinardia flaccida, Phyto_diatom_rhisoleniales_Rhizosolenia, Phyto_dinoflagellate_gonyaulacales_Tripos, Phyto_dinoflagellate_gonyaulacales_Tripos macroceros, Phyto_dinoflagellate_gonyaulacales_Tripos muelleri, Zoo_cnidaria, Zoo_crustacea_copepod, Zoo_crustacea_copepod_calanoida, Zoo_crustacea_copepod_calanoida_Acartia, Zoo_crustacea_copepod_calanoida_Centropages, Zoo_crustacea_copepod_cyclopoida, Zoo_crustacea_copepod_cyclopoida_Oithona, Zoo_crustacea_copepod_nauplii, Zoo_other, Zoo_tintinnidae.

**3 classes:** copepod, detritus, noncopepod.

## Usage

### ViT / BEiT (transformers)

```python
from PIL import Image
from transformers import AutoImageProcessor, ViTForImageClassification

repo = "Noushineft/ViT-LASNet"
processor = AutoImageProcessor.from_pretrained(repo, subfolder="vit-mislas")
model = ViTForImageClassification.from_pretrained(repo, subfolder="vit-mislas").eval()

image = Image.open("plankton.tif").convert("RGB")
logits = model(**processor(images=image, return_tensors="pt")).logits
print(model.config.id2label[logits.argmax(-1).item()])
```

For BEiT, use `BeitForImageClassification` with `subfolder="beit-balce"`.

### ResNet18 (torchvision + safetensors)

```python
import torch, torchvision
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

path = hf_hub_download("Noushineft/ViT-LASNet", "resnet18-18class-rgb/model.safetensors")
model = torchvision.models.resnet18()
model.fc = torch.nn.Linear(model.fc.in_features, 18)
model.load_state_dict(load_file(path))
model.eval()
```

The ResNet models take RGB images resized to 256×256 and scaled to [0, 1], without ImageNet normalisation. Each ResNet folder's `config.json` lists the class order.

### Command line

The `test/test.py` script in the code repository can load these models directly:

```bash
python test/test.py -m 2 -n 18 -w Noushineft/ViT-LASNet/vit-mislas -f path/to/images -o predictions.csv
```

## Training data

The models were trained on Pi-10 plankton images collected in situ, labelled into the classes above. The dataset is not distributed with these weights.

## Limitations

- The models were trained on images from one instrument (Pi-10) and its imaging conditions. Expect lower accuracy on other imagers or regions without fine-tuning.
- Rare classes have few training examples, so their predictions are less reliable than those for common classes.
- These are research models and have not been validated for operational monitoring.

## Citation

If you use these models, please cite the paper linked above.