agri-vision-detector / docs /MODEL_CARD_DETAILED.md
Solshine's picture
Self-contained release: phone_site with runtime, RUN.md, d1 GGUFs, scripts, weights, licences by upstream terms
f79b7a1 verified
|
Raw History Blame Contribute Delete
18.1 kB
# Detailed measurements (sections 1-8)
Moved from the first card draft. Model licensing is in the repo README and LICENSE files. Numbering kept so older references still work.
## 1. Quick start (detector, no deep-learning framework needed)
```bash
pip install onnxruntime pillow numpy
python infer_onnx.py best.onnx field_photo.jpg --conf 0.3 --iou 0.5 --save out.jpg
```
`infer_onnx.py` is in this repo: letterbox to 640x640, run, decode `(1, 7, 8400)` (cx, cy, w, h in letterboxed pixels + 3 class scores), per-class NMS.
Tested against Ultralytics' own predictor on 12 held-out test images: top-detection confidences matched (e.g. 0.92 vs 0.92) and detection counts at conf 0.3
were within +/-1 on 10 of 12 images (the rest differed by up to 5 because of PIL-vs-OpenCV resizing near the threshold). With Ultralytics installed:
`YOLO("best.onnx").predict("field_photo.jpg", imgsz=640, conf=0.3)`.
## 2. Detector results (held-out)
| model / split | images | mAP50 | mAP50-95 | precision | recall |
|---|---|---|---|---|---|
| PyTorch, val | 1,227 | 0.581 | 0.294 | 0.648 | 0.558 |
| **PyTorch, test** | 1,246 | **0.644** | **0.349** | 0.681 | 0.580 |
| ONNX, test | 1,246 | 0.641 | 0.346 | 0.675 | 0.575 |
Per-class test AP50-95: common_bean 0.328, cowpea 0.366, sorghum 0.352. Evaluated at conf 0.001 / IoU 0.6. Val was used for early stopping; **test is the unbiased number.**
Splits are by capture date (or source batch when no date is recoverable), per crop, so near-duplicate adjacent frames do not straddle train and test.
All three crops appear in every split. The 987 climbing-bean images have no recoverable date, so they are train-only; bean val/test are bush beans.
What the numbers hide (from looking at predictions overlaid on test images):
- **Tiny / distant plants are missed.** One drip-row photo with 22 labelled ~15 px plants got 1 detection at 640 px input.
- **Labels are subjective and incomplete.** Box boundaries around overlapping plants vary, and some visible plants/weeds are unlabelled, so mAP50-95 understates quality on medium-sized plants and precision is understated by missing labels.
## 3. What d1 can and cannot do zero-shot (honest baseline)
`d1-omni-600M` answers typed multiple-choice / yes-no questions about an image in one forward pass (no text generation). On 204 field photos (12 crops x 17, drawn from Image Safari;
I checked each image on contact sheets and flagged 13 as ambiguous):
| setting | crop top-1 (excl. 13 ambiguous) | Wilson 95% |
|---|---|---|
| d1 zero-shot, F16, 896 px | 40/191 = 20.9% | [15.8, 27.3] |
| d1 zero-shot, Q8_0, 896 px | 39/191 = 20.4% | [15.3, 26.7] |
| d1 zero-shot, F16, 448 px (the feature-extraction setting) | 40/191 = 20.9% | |
Only banana (17/17) and maize (15/16) are recognised. Most other crops are called "maize" (96 of 164 wrong answers had confidence > 0.9). This is not option-order bias:
reversing the option order and adding descriptions gave 8/60 with 41/60 still "maize". Q8_0 and F16 agree on 199/204 crop answers, so quantising is not the problem.
The zero-shot disease yes/no score was not usable (no ground truth exists in this dataset; of the 12 images it rated most diseased I judged 2 plausible, 5 ambiguous, 5 healthy-looking).
**This is why the heads in section 4 exist.**
## 4. d1 heads (Tier A: frozen trunk + small trained head)
Method: a local patch to llama.cpp's `llama-server` (env `D1_EXPORT_TRUNK=1`) appends d1's 1,024-d trunk state to each option's output, so we cache one
3,072-d vector per image (mean trunk state for each of three fixed questions: crop, condition concept, plant part) plus the options' 42 raw logits, and train a
logistic-regression/MLP head in plain PyTorch on CPU. No fine-tuning of d1's weights; no transformers library involved.
Every number below is reported next to (a) d1 zero-shot on the same images and (b) a **style-only control** (colour grid, white-background fraction, edge density, aspect
ratio; no semantics), because healthy images in PlantDoc are mostly white-background stock photos and would otherwise inflate healthy-vs-diseased accuracy.
All numbers: accuracy with Wilson 95% CI, **after removing evaluation images that have a near-duplicate (cosine >= 0.95 on d1 trunk features) in the training pool**.
Heads use the 3,072-d trunk features ("feat"); the "logit" and "feat+logit" variants are in `head_all_dedup.log` and were not better. Hyper-parameters were chosen on validation splits only.
| task (train -> test) | n | **d1 trunk head** | d1 zero-shot | style-only control | majority class |
|---|---|---|---|---|---|
| **Crop ID, 17 crops**, Safari train -> held-out Safari probe (same campaign) | 187 | **91.4%** [86.6, 94.7] | 18.7% [13.8, 24.9] | 27.8% [21.9, 34.6] | 0% |
| Crop ID, Safari train -> **Artemis detection images** (different campaign/camera; bean, cowpea, sorghum) | 299 | **31.1%** [26.1, 36.6] | 0.3% | 0% | 0% |
| **Healthy vs diseased**, PlantWild train -> PlantWild test | 644 | **87.4%** [84.6, 89.8] | 80.3% [77.0, 83.2] | 70.7% [67.0, 74.0] | 59.0% |
| Healthy vs diseased, PlantWild train -> **PlantDoc** (cross-source) | 2,523 | **90.8%** [89.6, 91.9] | 84.7% [83.3, 86.1] | 72.1% [70.4, 73.9] | 66.7% |
| **Condition (9 coarse classes)**, PlantWild+PlantSeg train -> PlantWild test | 639 | **68.1%** [64.4, 71.6] | 43.5% | 30.8% | 41.3% |
| Condition, -> PlantSeg test | 478 | **60.5%** [56.0, 64.7] | 11.5% | 21.5% | 0% |
| Condition, -> **PlantDoc** (cross-source) | 2,521 | **63.4%** [61.5, 65.2] | 36.6% [34.8, 38.5] | 26.7% | 33.3% |
| **Pest species (IP102, 102 classes)**, official split sample | 534 | **40.1%** [36.0, 44.3] (chance ~1%) | n/a (no species options) | 10.9% | 0.6% |
Honest reading:
- **The heads help a lot over d1 zero-shot**, and the style-only control (colours, white background, edges) is far below them, so they are not just learning photo style.
For healthy-vs-diseased the gain over zero-shot is modest (+6 points on PlantDoc) because d1's "healthy green leaves" concept already works reasonably zero-shot.
- **Crop ID does not generalise across campaigns.** The 91% is in-distribution. On the Artemis images the head is right for beans (85 of 100) but 1 of 100 for cowpea and 8 of 100 for sorghum (checked on the pre-dedup test set): it calls
almost everything "common bean" (an always-bean classifier would score ~33%, the head scores 31%). Treat crop ID as unsolved outside the Safari collection.
- **Duplicates were real.** The same photos appear in PlantDoc, PlantWild and PlantSeg (65 of 2,578 PlantDoc images matched at cosine >= 0.98, several verified by eye as identical). Removing near-duplicates
changed accuracies by up to ~4 points (largest: PlantSeg test 64.6% -> 60.5%), and without that check the cross-source numbers would have been optimistic. The 0.95 threshold only catches near-identical images; re-cropped copies can slip through.
- **Style confound partly present.** For the feat+logit variant of the healthy-vs-diseased head, PlantDoc accuracy is 96.1% on white-background images (n=362) vs 90.3% on the rest (n=2,027, balanced accuracy 0.84), so stock-photo style does help, but the head is clearly above the style-only control (72%) on the non-white subset too.
- **Label quality caps these numbers.** PlantWild is web-scraped and noisy; the coarse condition classes are my keyword mapping of 74 raw disease names (e.g. "leaf mold" -> mildew class; "leaf curl" dropped as ambiguous);
cassava is evaluated separately below.
- Small test sets (e.g. ~5 images per pest species); intervals above are wide where n is small.
**Cassava (Uganda field photos; the closest proxy we have for the African target)** — features extracted, evaluated separately (`eval_cassava.log`, `eval_multisource.log`):
| test | n | result | comparison |
|---|---|---|---|
| 5-class cassava head (bacterial blight / brown streak / green mottle / mosaic / healthy), trained on cassava | 300 | **60.3%** [54.7, 65.7] | style-only 39.0%, majority 20% |
| PlantWild-trained healthy-vs-diseased head applied cross-domain to all cassava (400 healthy / 1,600 diseased) | 2,000 | balanced accuracy **0.605** (healthy recall 71%, diseased recall 50%) |
| saved `condition9` head, grouped healthy / blight / virus | 2,000 | 46.4% (balanced 0.553) | zero-shot 24.9% (balanced 0.414); majority 60% |
| healthy-vs-diseased head trained on PlantWild **+ cassava train** | 300 | 83.3% (balanced **0.771**) | PlantWild-only head on the same test: 63.0% (balanced 0.625); PlantDoc 89.7% vs 90.8%, PlantWild test unchanged |
Reading: **heads trained on web/lab-style leaf photos do not transfer to field canopy photos** (cross-domain cassava is no better than d1 zero-shot); adding in-domain data fixes most of it without hurting the other test sets.
The cassava test is a random split of one Kaggle collection (no capture dates), so in-domain numbers are optimistic, and the 300-image test is 80% diseased, so use balanced accuracy.
A single end-to-end example (`diagnose.py`) on a cassava mosaic image: the condition head said mosaic/yellowing virus (74.6%, correct), but the detector called the leaf "sorghum" (0.68) and the crop head said yam/common bean: both are out-of-distribution there.
**Saved heads** (`artifacts/heads_v1/*.npz`, plain NumPy: standardise with `mu`/`sd`, then `x @ W.T + b`; 50 KB - 1.2 MB each, classes in the `.json`): `crop17`, `healthy_vs_diseased`, `condition9`, `pest102`.
Re-scored from the saved files on the same deduplicated test sets: crop17 probe 92.5% / Artemis 30.8%; healthy_vs_diseased PlantDoc 90.4%; condition9 PlantDoc 63.4%; pest102 42.7%. They were refit on train+val; test sets were never used for fitting.
## 4b. Tiny CNN diagnoser (end-to-end fine-tuned; added 2026-10-10)
A second, much smaller classifier that needs **no d1 and no llama.cpp**: MobileNetV3-Large (ImageNet-pretrained, fine-tuned end to end in plain PyTorch on the same disease sources) with two heads,
**healthy vs diseased** and a **12-way coarse condition** (same 12 concept labels as the d1 heads). 224x224 whole-image input. ONNX: **11.9 MB fp32, 3.4 MB INT8**. On a busy 2014-era Xeon core it ran at ~21 ms/image (fp32, 1 thread, ONNX Runtime); never run on a phone.
Release candidate: `artifacts/cnn_tiny_v1/` (trained on PlantWild, PlantSeg, PlantDoc, Cassava, KaraAgro, Makerere, Pretoria and CCMT train splits; `SHA256SUMS` included). `infer_tiny.py` runs detector -> per-plant crop -> classifier with an "unsure" band, ONNX Runtime only.
**How to read these numbers.** In-source test splits of KaraAgro, Makerere and CCMT come from random/hash splits over consecutive shots of the same fields and read ~0.97-0.99 (Pretoria's in-source split is unusable: 0.49-0.56, below its majority baseline); the same CCMT images score **0.84-0.87** accuracy (balanced 0.83-0.88) when CCMT was never trained on. Use the held-out-source rows below, not the in-source ones.
| question | test (never trained on) | n | CNN | comparison |
|---|---|---|---|---|
| web -> web: healthy vs diseased, trained on PlantWild+PlantSeg | PlantDoc (deduped) | 2,523 | **0.943** (3 seeds: 0.939, 0.947, 0.942) | d1 trunk head, same rows: 0.915 [0.904, 0.925] |
| 12-way condition, same training | PlantDoc | 2,523 | 0.780 (1 seed) | d1 head: 0.640 |
| web only -> tropical field: healthy vs diseased | KaraAgro (Ghana maize) | 768 | 0.535-0.555 (3 seeds) **below the 0.694 majority baseline** | style-only 0.483 |
| + other field sources in training, KaraAgro held out | KaraAgro | 768 | **0.801** [0.771, 0.828], balanced 0.805 | d1 head 0.784, balanced 0.805; style-only 0.483 |
| + other field sources, Makerere held out | Makerere (Uganda maize) | 2,113 | **0.969** [0.961, 0.976] (1 seed; web-only 0.805-0.841) | d1 head on 861 rows: 0.905; style-only 0.634 |
| external, never trained on | CCMT raw (Ghana; cashew/cassava/maize/tomato; 87% diseased) | 25,122 | balanced accuracy **0.83-0.88** (accuracy 0.84-0.87, equal to majority) | maize is weakest (0.63-0.76) |
| pest species, 97 IP102 classes (19k images, official split, pHash-deduped test) | IP102 test | 2,969 | top-1 **0.726** [0.710, 0.742], top-5 0.914, balanced top-1 0.537 | small-sample comparison on the d1 test rows: CNN 0.530 vs d1 head 0.401 |
| crop ID, 18 crops, site-disjoint folds | Safari sites | 2,824 | 0.523 [0.504, 0.541] | d1 head 0.518; style-only 0.127 |
| Amazonian/tropical useful-plant species ID, 11 taxa from GBIF photos, observer-disjoint folds | GBIF | 2,238 | 0.818 [0.801, 0.833] | majority 0.134 (internal experiment) |
What the CNN work showed (all logged in `LAB_NOTEBOOK.md` sections 26-46):
- **Adding field data from other farms/countries is what moves field numbers**; photometric augmentation, test-time augmentation, a 3-seed ensemble and EfficientNet-B0 did not (all within seed noise or worse).
- **Abstention helps, calibration does not transfer.** Taking the 80% most confident KaraAgro predictions raises balanced accuracy 0.805 -> 0.867; temperature scaling fitted on source validation data made held-out calibration worse (ECE 0.095 -> 0.117).
- **INT8** (ORT static QDQ) keeps accuracy (KaraAgro-held-out model 0.814 vs 0.801 fp32) only with percentile calibration; MinMax/Entropy calibration collapsed to chance. No speed claim.
- Error analysis, KaraAgro held out: maize streak virus 0.90 correct, healthy 0.82, fall-armyworm damage 0.65.
- Seed-to-seed spread is 1 point on web sets but 2-4 points on Makerere/KaraAgro and up to 14 on the small Pretoria split, so single-seed field differences below that are not evidence.
- A d1 head and the CNN tie on KaraAgro and on site-grouped crop ID; the CNN is better on web cross-source and on Makerere. Neither is deployment-grade for farmer-facing diagnosis.
## 5. Data and provenance
| source | use | size used |
|---|---|---|
| Artemis / Image Safari (CIAT/Alliance Bioversity, AWS Open Data `alliance-artemis-imagesafari`) | detector training/eval; crop-ID head; d1 probe | 8,160 detection images; 1,653 crop-ID images; 204 probe images |
| PlantDoc | disease head train/eval (cross-source test) | 2,578 |
| PlantWild (original release) | disease head | 3,560 sampled of 18,542 |
| PlantSeg | disease head | 2,792 sampled of 11,458 |
| IP102 (Zenodo 15546419 COCO re-upload by a third party, not the original authors) | pest-species head (d1: 2,308 sampled) and CNN (all 18,969 images, 97 classes present) | 2,308 / 18,969 |
| Cassava Leaf Disease (HF mirror `Engineer101/...`) | disease head / CNN | 2,000 (400/class) |
| KaraAgro AI Maize (Ghana; Harvard Dataverse 10.7910/DVN/CXUMDS v4) | CNN and d1 head (healthy / fall armyworm / maize streak virus), held-out-source tests | 5,172 unique photos (one per base image, label derived from the VOC boxes, not folder names) |
| Maize photos, Makerere AI Lab (Uganda; Harvard Dataverse 10.7910/DVN/LPGHKK) | CNN and d1 head (healthy / leaf blight / maize streak virus) | 7,421 photos, 9 capture days (class is tied to collection day: confounded) |
| Pretoria "Diseases of maize in the field" (Figshare 20237613) | CNN and d1 head | 2,355 photos; split unreliable (five camera-image groups hold most images) |
| CCMT, Mendeley Data bwh3zbpkpv (Ghana; cashew, cassava, maize, tomato) | CNN external test and training (raw .jpg only; the 105k augmented copies were not used) | 25,215 raw images (93 unreadable) |
| GBIF occurrence photos (iNaturalist-contributed) for 11 Amazonian/tropical useful-plant taxa | internal species-ID experiment only | 2,238 photos |
Not obtained: DLCPD-25 (Baidu only), AgriPest (never released), INSECT204. Hemp: no labelled public set was obtainable (notebook section 43).
Images are not redistributed; manifests and scripts are in the repo, so the data can be re-fetched. (An early version of this card said CCMT was blocked; it is now fetched, see notebook sections 31-32.)
## 7. Limitations (summary)
3 of 18 crops covered by the detector; tiny-plant recall is poor (tiled inference trades precision for recall; no setting beat full-image F1); disease labels are mostly web-scraped or lab-style and noisy (e.g. a cannabis leaf appeared under "maple, healthy" in PlantWild);
PlantDoc healthy/diseased differ in photographic style; cassava disease classes are visually subtle; no on-device latency measured (the patched llama.cpp server cross-compiles for Android arm64
with NDK r27c, but has never been run on a phone); TFLite export was attempted four times and failed (onnx2tf errors), so ONNX is the only mobile format.
## 8. Files
`best.onnx`, `best.pt`, `results.csv`, `args.yaml`, `detector_eval_yolo11n_v1.json`, `SHA256SUMS` (copies verified byte-identical to the training outputs), `infer_onnx.py`, dataset builders and manifests,
`train_detector.py`, `eval_detector.py`, `train_cnn.py`, `export_cnn_onnx.py`, `quantize_cnn_onnx.py`, `infer_tiny.py`, `artifacts/cnn_tiny_v1/`, `extract_features.py`, `train_head.py`, `export_heads.py`, `eval_cassava.py`, `eval_multisource.py`, `diagnose.py` (end-to-end demo), and the llama.cpp patch (see HOWTO.md).
## Citation / credits
Artemis & Image Safari: Mutuvi S. et al., agriRxiv 2026.00491 (CIAT/Alliance of Bioversity International). d1-omni: Liquid AI. YOLO11: Ultralytics. PlantDoc, PlantWild, PlantSeg, IP102, Cassava Leaf Disease: cite per their papers.
Trained on a single Tesla M40 (sm_52) with torch 1.13.1. [ORG / authors / contact: fill in]
## 4c. 18-crop identifier with web photos (added 2026-10-25)
`artifacts/crop18_web_v1/` (MobileNetV3-L, 224 px, fp32 ONNX 11.9 MB, 18 Safari crops). Trained on Safari field photos plus 2,235 GBIF/iNaturalist photos (16 of 18 crops; the images themselves are not redistributed). Held-out estimate = site-disjoint 5-fold CV of the same recipe, **0.595-0.605 pooled over 3 seeds (n=2,824) vs 0.503-0.533 without web photos**; the shipped model is fit on all rows so has no test set of its own. Expect about 0.60 or lower at a new site; crops without web photos (african yam, lentil) and finger millet (38 photos) are weaker by construction. Not INT8-quantised, not run on a phone, labels not audited by an expert. Details: LAB_NOTEBOOK sections 49-50.