PlantNet/vit_base_patch14_reg4_dinov2_lvd142m.pc24_all

PlantCLEF 2024 - ViT-Base DINOv2 (All)

Model description

This model was shared for the PlantCLEF 2024 evaluation campaign. It is designed for plant species identification of the southwestern Europe flora (7,806 vascular plant species). The architecture is based on a ViT-Base (patch 14) initially pre-trained using the DINOv2 self-supervised learning (SSL) method.

Training process

The training was performed using the timm library (version 0.9.16) under torch (version 2.2.1+cu121), utilizing the Exponential Moving Average (EMA) option for improved performance. The procedure was carried out in two distinct steps:

• Step 1 (only classifier): The DINOv2 backbone was frozen, and only the classification head was fine-tuned on the data.
• Step 2 (all): The model provided in this repository corresponds to the second step. Training was continued on the entire model, simultaneously fine-tuning both the backbone and the classification head.

Dataset

The model was pre-trained on a subset of the Pl@ntNet collaborative platform data.

• Volume: Approximately 1.4 million images.
• Taxonomy: 7,806 vascular plant species.
• Structure: The images were split into three subsets: a training set, a validation set (to control overfitting and select the best model during training), and a test set.
• Data Access: The data can be retrieved via a tar file or retrieved from the metadata file (CSV format, in the image_backup_url column).

Usage

The weights can be loaded using the timm library:

import timm

# Load the model with weights trained on the entire architecture
model = timm.create_model("hf_hub:PlantNet/vit_base_patch14_reg4_dinov2_lvd142m.pc24_all", pretrained=True)

To run a full inference extracting the top-5 species predictions, you will need the class and species mapping files provided in this repository.

import pandas as pd
from urllib.request import urlopen
from PIL import Image
import timm
import torch

# Load mappings
with open("class_mapping.txt") as f:
    cid_to_spid = {i: line.strip() for i, line in enumerate(f)}

df = pd.read_csv("species_id_to_name.txt", sep=';', quoting=1, dtype={'species_id': str})
spid_to_sp = df.set_index('species_id')['species'].to_dict()

# Load model directly from Hugging Face Hub
model = timm.create_model("hf_hub:PlantNet/vit_base_patch14_reg4_dinov2_lvd142m.pc24_all", pretrained=True)
model = model.eval()

# Get model-specific transforms
data_config = timm.data.resolve_model_data_config(model)
transforms = timm.data.create_transform(**data_config, is_training=False)

# Load and process image
img_url = "[https://lab.plantnet.org/LifeCLEF/PlantCLEF2024/single_plant_training_data/PlantCLEF2024singleplanttrainingdata/test/1361687/bd2d3830ac3270218ba82fd24e2290becd01317c.jpg](https://lab.plantnet.org/LifeCLEF/PlantCLEF2024/single_plant_training_data/PlantCLEF2024singleplanttrainingdata/test/1361687/bd2d3830ac3270218ba82fd24e2290becd01317c.jpg)"
img = Image.open(urlopen(img_url))
img_tensor = transforms(img).unsqueeze(0)

# Inference
with torch.no_grad():
    output = model(img_tensor)

# Get top-5 predictions
top5_probabilities, top5_class_indices = torch.topk(output.softmax(dim=1) * 100, k=5)
top5_probabilities = top5_probabilities.numpy()[0]
top5_class_indices = top5_class_indices.numpy()[0]

print("Top 5 predictions:")
for proba, cid in zip(top5_probabilities, top5_class_indices):
    species_id = cid_to_spid[cid]
    species = spid_to_sp.get(species_id, "Unknown")
    print(f"{species_id} - {species} : {proba:.2f}%")

Citation

@misc{goeau_2024_10848263,
   author = {Goëau, Hervé and Lombardo, Jean-Chirstophe and Affouard, Antoine and Espitalier, Vincent and Bonnet, Pierre and Joly, Alexis},
   title = {{PlantCLEF 2024 pretrained models on the flora of the south western Europe based on a subset of Pl@ntNet collaborative images and a ViT base patch 14 dinoV2}},
   month = mar,
   year = 2024,
   publisher = {Zenodo},
   doi = {10.5281/zenodo.10848263}, url = {https://doi.org/10.5281/zenodo.10848263}
}
@article{plantclef2024overview,
  title={Overview of {PlantCLEF} 2024: multi-species plant identification in vegetation plot images},
  author={Goëau, Hervé and Espitalier, Vincent and Bonnet, Pierre and Joly, Alexis},
  journal={Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum},
  year={2024}
}