Marconcini's picture
Model card: demo link and use in the cited assistant
203f616 verified
|
Raw History Blame Contribute Delete
5.56 kB
---
license: cc-by-sa-4.0
library_name: pytorch
pipeline_tag: image-classification
tags:
- plant-disease
- agriculture
- convnext
- timm
datasets:
- mohanty/PlantVillage
metrics:
- accuracy
---
# LeafScan — ConvNeXt-T for plant disease classification (field-calibrated)
Classifies a leaf photo into **38 crop / disease classes** (14 crops) and says **"Not sure"**
when its confidence is too low. Research / portfolio project:
[github.com/marconcini/leafscan](https://github.com/marconcini/leafscan) — every number below
comes from experiments documented there, with hypotheses written before results.
- **Architecture:** ConvNeXt-T (`timm`: `convnext_tiny.fb_in22k_ft_in1k`), 28M parameters, 224×224 input.
- **Training:** ImageNet weights → PlantVillage (lab photos, 39k images) → 85% of PlantDoc train
(1,975 web/field photos). The other 15% of PlantDoc train (348 photos) is used only to fit a
temperature (T = 0.83) and an abstention threshold (0.75 on calibrated confidence).
## Results
| | PlantDoc test (field, 228 photos) | PlantVillage test (lab, 10,709 photos) |
|---|---:|---:|
| Accuracy | **72.4%** (95% CI 66.2–78.1) | 96.6% |
| Right crop | 86.4% | – |
| Healthy vs diseased | 94.7% | – |
| Answers above the threshold | 65% of photos, **87%** of them right | 94% of photos, 98.7% right |
For context: the same architecture trained on PlantVillage only reaches 99.7% on lab photos but
**31%** on field photos. Lab accuracy says little about real-world use.
## How to use
```bash
pip install "leafscan @ git+https://github.com/marconcini/leafscan.git"
```
```python
from PIL import Image
from leafscan.predictor import Predictor
predictor = Predictor("Marconcini/leafscan-convnext-tiny") # downloads config.json + model.safetensors
pred = predictor(Image.open("leaf.jpg"))
print(pred.verdict) # diagnosis, or "Not sure" below the field-calibrated threshold
print(pred.top) # top-3 (class, calibrated probability)
```
Files:
- `model.safetensors` — PyTorch weights; `config.json` — class names, input size, temperature,
threshold (and those of the int8 model under `onnx_int8`).
- `model.onnx` — fp32 ONNX export, identical top-1 to PyTorch on PlantDoc test.
- `model_int8.onnx` — 32 MB, linear layers quantised to int8 (convolutions kept fp32, as ONNX
Runtime Web has no int8 convolution). Used by the in-browser demo. Same predictions as fp32
on 96.5% of PlantDoc test photos; 71.9% accuracy. Because quantisation changes some
predictions, it has **its own temperature and threshold**, refitted on the same field
calibration set: on PlantDoc test it answers 66% of photos, 84.7% of them right.
In the browser (ONNX Runtime Web, image resizing done by the canvas instead of PIL), the demo
agrees with the Python int8 predictions on 96.1% of PlantDoc test photos.
## In the LeafScan app: diagnosis, then a cited assistant
👉 Try the model in your browser: [Marconcini/leafscan](https://huggingface.co/spaces/Marconcini/leafscan)
(int8 ONNX, nothing uploaded).
The project's local app goes one step further: after the diagnosis, a retrieval-augmented
assistant answers questions from 28 Wikipedia articles with a local LLM, citing its sources
(`LEAFSCAN_ASSISTANT=1 python -m leafscan.app`). How this model's output is used there was
tested on 500 (question, field photo) pairs with its real predictions on PlantDoc test:
- Restricting the search to the predicted disease **costs 16 points of recall@5**: a right
diagnosis adds 8 points, a wrong one (28% of field photos) removes the answer.
- Restricting to the top-2 when the model is unsure recovers less than half of the loss.
So the diagnosis does **not** filter the search: it suggests questions that name the disease
(both candidates when the model says "Not sure"), which the user can see and correct.
Details: [research log](https://github.com/marconcini/leafscan/blob/main/docs/research-log.md).
## Intended use and limitations
- **Intended use:** demonstration and research on robustness of plant-disease classifiers.
**Not** a diagnostic tool and not a substitute for an agronomist.
- Knows only the 38 PlantVillage classes; any other plant or disease is forced into one of them
(or answered "Not sure").
- Test set is small (228 photos): differences under ~6 points are within noise.
- Calibration and threshold were fitted on PlantDoc photos; they may not hold for photos from
other sources (the project shows lab-fitted thresholds failing on field photos).
- Rejection of non-leaf images is weak next to real field photos (AUROC 0.6–0.8 in the
project's tests): a photo of something else may still receive a diagnosis.
- PlantDoc is web-scraped and contains label noise (duplicates filed under different classes
were removed).
## Training data and licenses
- **PlantVillage** (Hughes & Salathé, 2015), CC BY-SA 3.0 — via `mohanty/PlantVillage`.
- **PlantDoc** (Singh et al., 2020), CC BY 4.0 — [github.com/pratikkayal/PlantDoc-Dataset](https://github.com/pratikkayal/PlantDoc-Dataset).
- Weights released under **CC BY-SA 4.0** (share-alike, following PlantVillage's license).
Code: MIT.
## References
- Hughes & Salathé (2015). *An open access repository of images on plant health to enable the development of mobile disease diagnostics.* arXiv:1511.08060
- Singh et al. (2020). *PlantDoc: A Dataset for Visual Plant Disease Detection.* CoDS-COMAD
- Liu et al. (2022). *A ConvNet for the 2020s.* CVPR
- Guo et al. (2017). *On Calibration of Modern Neural Networks.* ICML