agri-vision-detector / docs /MODEL_CARD_DETAILED.md
Solshine's picture
Self-contained release: phone_site with runtime, RUN.md, d1 GGUFs, scripts, weights, licences by upstream terms
f79b7a1 verified
|
Raw History Blame Contribute Delete
18.1 kB

Detailed measurements (sections 1-8)

Moved from the first card draft. Model licensing is in the repo README and LICENSE files. Numbering kept so older references still work.

1. Quick start (detector, no deep-learning framework needed)

pip install onnxruntime pillow numpy
python infer_onnx.py best.onnx field_photo.jpg --conf 0.3 --iou 0.5 --save out.jpg

infer_onnx.py is in this repo: letterbox to 640x640, run, decode (1, 7, 8400) (cx, cy, w, h in letterboxed pixels + 3 class scores), per-class NMS. Tested against Ultralytics' own predictor on 12 held-out test images: top-detection confidences matched (e.g. 0.92 vs 0.92) and detection counts at conf 0.3 were within +/-1 on 10 of 12 images (the rest differed by up to 5 because of PIL-vs-OpenCV resizing near the threshold). With Ultralytics installed: YOLO("best.onnx").predict("field_photo.jpg", imgsz=640, conf=0.3).

2. Detector results (held-out)

model / split images mAP50 mAP50-95 precision recall
PyTorch, val 1,227 0.581 0.294 0.648 0.558
PyTorch, test 1,246 0.644 0.349 0.681 0.580
ONNX, test 1,246 0.641 0.346 0.675 0.575

Per-class test AP50-95: common_bean 0.328, cowpea 0.366, sorghum 0.352. Evaluated at conf 0.001 / IoU 0.6. Val was used for early stopping; test is the unbiased number.

Splits are by capture date (or source batch when no date is recoverable), per crop, so near-duplicate adjacent frames do not straddle train and test. All three crops appear in every split. The 987 climbing-bean images have no recoverable date, so they are train-only; bean val/test are bush beans.

What the numbers hide (from looking at predictions overlaid on test images):

  • Tiny / distant plants are missed. One drip-row photo with 22 labelled ~15 px plants got 1 detection at 640 px input.
  • Labels are subjective and incomplete. Box boundaries around overlapping plants vary, and some visible plants/weeds are unlabelled, so mAP50-95 understates quality on medium-sized plants and precision is understated by missing labels.

3. What d1 can and cannot do zero-shot (honest baseline)

d1-omni-600M answers typed multiple-choice / yes-no questions about an image in one forward pass (no text generation). On 204 field photos (12 crops x 17, drawn from Image Safari; I checked each image on contact sheets and flagged 13 as ambiguous):

setting crop top-1 (excl. 13 ambiguous) Wilson 95%
d1 zero-shot, F16, 896 px 40/191 = 20.9% [15.8, 27.3]
d1 zero-shot, Q8_0, 896 px 39/191 = 20.4% [15.3, 26.7]
d1 zero-shot, F16, 448 px (the feature-extraction setting) 40/191 = 20.9%

Only banana (17/17) and maize (15/16) are recognised. Most other crops are called "maize" (96 of 164 wrong answers had confidence > 0.9). This is not option-order bias: reversing the option order and adding descriptions gave 8/60 with 41/60 still "maize". Q8_0 and F16 agree on 199/204 crop answers, so quantising is not the problem. The zero-shot disease yes/no score was not usable (no ground truth exists in this dataset; of the 12 images it rated most diseased I judged 2 plausible, 5 ambiguous, 5 healthy-looking). This is why the heads in section 4 exist.

4. d1 heads (Tier A: frozen trunk + small trained head)

Method: a local patch to llama.cpp's llama-server (env D1_EXPORT_TRUNK=1) appends d1's 1,024-d trunk state to each option's output, so we cache one 3,072-d vector per image (mean trunk state for each of three fixed questions: crop, condition concept, plant part) plus the options' 42 raw logits, and train a logistic-regression/MLP head in plain PyTorch on CPU. No fine-tuning of d1's weights; no transformers library involved.

Every number below is reported next to (a) d1 zero-shot on the same images and (b) a style-only control (colour grid, white-background fraction, edge density, aspect ratio; no semantics), because healthy images in PlantDoc are mostly white-background stock photos and would otherwise inflate healthy-vs-diseased accuracy.

All numbers: accuracy with Wilson 95% CI, after removing evaluation images that have a near-duplicate (cosine >= 0.95 on d1 trunk features) in the training pool. Heads use the 3,072-d trunk features ("feat"); the "logit" and "feat+logit" variants are in head_all_dedup.log and were not better. Hyper-parameters were chosen on validation splits only.

task (train -> test) n d1 trunk head d1 zero-shot style-only control majority class
Crop ID, 17 crops, Safari train -> held-out Safari probe (same campaign) 187 91.4% [86.6, 94.7] 18.7% [13.8, 24.9] 27.8% [21.9, 34.6] 0%
Crop ID, Safari train -> Artemis detection images (different campaign/camera; bean, cowpea, sorghum) 299 31.1% [26.1, 36.6] 0.3% 0% 0%
Healthy vs diseased, PlantWild train -> PlantWild test 644 87.4% [84.6, 89.8] 80.3% [77.0, 83.2] 70.7% [67.0, 74.0] 59.0%
Healthy vs diseased, PlantWild train -> PlantDoc (cross-source) 2,523 90.8% [89.6, 91.9] 84.7% [83.3, 86.1] 72.1% [70.4, 73.9] 66.7%
Condition (9 coarse classes), PlantWild+PlantSeg train -> PlantWild test 639 68.1% [64.4, 71.6] 43.5% 30.8% 41.3%
Condition, -> PlantSeg test 478 60.5% [56.0, 64.7] 11.5% 21.5% 0%
Condition, -> PlantDoc (cross-source) 2,521 63.4% [61.5, 65.2] 36.6% [34.8, 38.5] 26.7% 33.3%
Pest species (IP102, 102 classes), official split sample 534 40.1% [36.0, 44.3] (chance ~1%) n/a (no species options) 10.9% 0.6%

Honest reading:

  • The heads help a lot over d1 zero-shot, and the style-only control (colours, white background, edges) is far below them, so they are not just learning photo style. For healthy-vs-diseased the gain over zero-shot is modest (+6 points on PlantDoc) because d1's "healthy green leaves" concept already works reasonably zero-shot.
  • Crop ID does not generalise across campaigns. The 91% is in-distribution. On the Artemis images the head is right for beans (85 of 100) but 1 of 100 for cowpea and 8 of 100 for sorghum (checked on the pre-dedup test set): it calls almost everything "common bean" (an always-bean classifier would score ~33%, the head scores 31%). Treat crop ID as unsolved outside the Safari collection.
  • Duplicates were real. The same photos appear in PlantDoc, PlantWild and PlantSeg (65 of 2,578 PlantDoc images matched at cosine >= 0.98, several verified by eye as identical). Removing near-duplicates changed accuracies by up to ~4 points (largest: PlantSeg test 64.6% -> 60.5%), and without that check the cross-source numbers would have been optimistic. The 0.95 threshold only catches near-identical images; re-cropped copies can slip through.
  • Style confound partly present. For the feat+logit variant of the healthy-vs-diseased head, PlantDoc accuracy is 96.1% on white-background images (n=362) vs 90.3% on the rest (n=2,027, balanced accuracy 0.84), so stock-photo style does help, but the head is clearly above the style-only control (72%) on the non-white subset too.
  • Label quality caps these numbers. PlantWild is web-scraped and noisy; the coarse condition classes are my keyword mapping of 74 raw disease names (e.g. "leaf mold" -> mildew class; "leaf curl" dropped as ambiguous); cassava is evaluated separately below.
  • Small test sets (e.g. ~5 images per pest species); intervals above are wide where n is small.

Cassava (Uganda field photos; the closest proxy we have for the African target) — features extracted, evaluated separately (eval_cassava.log, eval_multisource.log):

test n result comparison
5-class cassava head (bacterial blight / brown streak / green mottle / mosaic / healthy), trained on cassava 300 60.3% [54.7, 65.7] style-only 39.0%, majority 20%
PlantWild-trained healthy-vs-diseased head applied cross-domain to all cassava (400 healthy / 1,600 diseased) 2,000 balanced accuracy 0.605 (healthy recall 71%, diseased recall 50%)
saved condition9 head, grouped healthy / blight / virus 2,000 46.4% (balanced 0.553) zero-shot 24.9% (balanced 0.414); majority 60%
healthy-vs-diseased head trained on PlantWild + cassava train 300 83.3% (balanced 0.771) PlantWild-only head on the same test: 63.0% (balanced 0.625); PlantDoc 89.7% vs 90.8%, PlantWild test unchanged

Reading: heads trained on web/lab-style leaf photos do not transfer to field canopy photos (cross-domain cassava is no better than d1 zero-shot); adding in-domain data fixes most of it without hurting the other test sets. The cassava test is a random split of one Kaggle collection (no capture dates), so in-domain numbers are optimistic, and the 300-image test is 80% diseased, so use balanced accuracy. A single end-to-end example (diagnose.py) on a cassava mosaic image: the condition head said mosaic/yellowing virus (74.6%, correct), but the detector called the leaf "sorghum" (0.68) and the crop head said yam/common bean: both are out-of-distribution there.

Saved heads (artifacts/heads_v1/*.npz, plain NumPy: standardise with mu/sd, then x @ W.T + b; 50 KB - 1.2 MB each, classes in the .json): crop17, healthy_vs_diseased, condition9, pest102. Re-scored from the saved files on the same deduplicated test sets: crop17 probe 92.5% / Artemis 30.8%; healthy_vs_diseased PlantDoc 90.4%; condition9 PlantDoc 63.4%; pest102 42.7%. They were refit on train+val; test sets were never used for fitting.

4b. Tiny CNN diagnoser (end-to-end fine-tuned; added 2026-10-10)

A second, much smaller classifier that needs no d1 and no llama.cpp: MobileNetV3-Large (ImageNet-pretrained, fine-tuned end to end in plain PyTorch on the same disease sources) with two heads, healthy vs diseased and a 12-way coarse condition (same 12 concept labels as the d1 heads). 224x224 whole-image input. ONNX: 11.9 MB fp32, 3.4 MB INT8. On a busy 2014-era Xeon core it ran at ~21 ms/image (fp32, 1 thread, ONNX Runtime); never run on a phone. Release candidate: artifacts/cnn_tiny_v1/ (trained on PlantWild, PlantSeg, PlantDoc, Cassava, KaraAgro, Makerere, Pretoria and CCMT train splits; SHA256SUMS included). infer_tiny.py runs detector -> per-plant crop -> classifier with an "unsure" band, ONNX Runtime only.

How to read these numbers. In-source test splits of KaraAgro, Makerere and CCMT come from random/hash splits over consecutive shots of the same fields and read ~0.97-0.99 (Pretoria's in-source split is unusable: 0.49-0.56, below its majority baseline); the same CCMT images score 0.84-0.87 accuracy (balanced 0.83-0.88) when CCMT was never trained on. Use the held-out-source rows below, not the in-source ones.

question test (never trained on) n CNN comparison
web -> web: healthy vs diseased, trained on PlantWild+PlantSeg PlantDoc (deduped) 2,523 0.943 (3 seeds: 0.939, 0.947, 0.942) d1 trunk head, same rows: 0.915 [0.904, 0.925]
12-way condition, same training PlantDoc 2,523 0.780 (1 seed) d1 head: 0.640
web only -> tropical field: healthy vs diseased KaraAgro (Ghana maize) 768 0.535-0.555 (3 seeds) below the 0.694 majority baseline style-only 0.483
+ other field sources in training, KaraAgro held out KaraAgro 768 0.801 [0.771, 0.828], balanced 0.805 d1 head 0.784, balanced 0.805; style-only 0.483
+ other field sources, Makerere held out Makerere (Uganda maize) 2,113 0.969 [0.961, 0.976] (1 seed; web-only 0.805-0.841) d1 head on 861 rows: 0.905; style-only 0.634
external, never trained on CCMT raw (Ghana; cashew/cassava/maize/tomato; 87% diseased) 25,122 balanced accuracy 0.83-0.88 (accuracy 0.84-0.87, equal to majority) maize is weakest (0.63-0.76)
pest species, 97 IP102 classes (19k images, official split, pHash-deduped test) IP102 test 2,969 top-1 0.726 [0.710, 0.742], top-5 0.914, balanced top-1 0.537 small-sample comparison on the d1 test rows: CNN 0.530 vs d1 head 0.401
crop ID, 18 crops, site-disjoint folds Safari sites 2,824 0.523 [0.504, 0.541] d1 head 0.518; style-only 0.127
Amazonian/tropical useful-plant species ID, 11 taxa from GBIF photos, observer-disjoint folds GBIF 2,238 0.818 [0.801, 0.833] majority 0.134 (internal experiment)

What the CNN work showed (all logged in LAB_NOTEBOOK.md sections 26-46):

  • Adding field data from other farms/countries is what moves field numbers; photometric augmentation, test-time augmentation, a 3-seed ensemble and EfficientNet-B0 did not (all within seed noise or worse).
  • Abstention helps, calibration does not transfer. Taking the 80% most confident KaraAgro predictions raises balanced accuracy 0.805 -> 0.867; temperature scaling fitted on source validation data made held-out calibration worse (ECE 0.095 -> 0.117).
  • INT8 (ORT static QDQ) keeps accuracy (KaraAgro-held-out model 0.814 vs 0.801 fp32) only with percentile calibration; MinMax/Entropy calibration collapsed to chance. No speed claim.
  • Error analysis, KaraAgro held out: maize streak virus 0.90 correct, healthy 0.82, fall-armyworm damage 0.65.
  • Seed-to-seed spread is 1 point on web sets but 2-4 points on Makerere/KaraAgro and up to 14 on the small Pretoria split, so single-seed field differences below that are not evidence.
  • A d1 head and the CNN tie on KaraAgro and on site-grouped crop ID; the CNN is better on web cross-source and on Makerere. Neither is deployment-grade for farmer-facing diagnosis.

5. Data and provenance

source use size used
Artemis / Image Safari (CIAT/Alliance Bioversity, AWS Open Data alliance-artemis-imagesafari) detector training/eval; crop-ID head; d1 probe 8,160 detection images; 1,653 crop-ID images; 204 probe images
PlantDoc disease head train/eval (cross-source test) 2,578
PlantWild (original release) disease head 3,560 sampled of 18,542
PlantSeg disease head 2,792 sampled of 11,458
IP102 (Zenodo 15546419 COCO re-upload by a third party, not the original authors) pest-species head (d1: 2,308 sampled) and CNN (all 18,969 images, 97 classes present) 2,308 / 18,969
Cassava Leaf Disease (HF mirror Engineer101/...) disease head / CNN 2,000 (400/class)
KaraAgro AI Maize (Ghana; Harvard Dataverse 10.7910/DVN/CXUMDS v4) CNN and d1 head (healthy / fall armyworm / maize streak virus), held-out-source tests 5,172 unique photos (one per base image, label derived from the VOC boxes, not folder names)
Maize photos, Makerere AI Lab (Uganda; Harvard Dataverse 10.7910/DVN/LPGHKK) CNN and d1 head (healthy / leaf blight / maize streak virus) 7,421 photos, 9 capture days (class is tied to collection day: confounded)
Pretoria "Diseases of maize in the field" (Figshare 20237613) CNN and d1 head 2,355 photos; split unreliable (five camera-image groups hold most images)
CCMT, Mendeley Data bwh3zbpkpv (Ghana; cashew, cassava, maize, tomato) CNN external test and training (raw .jpg only; the 105k augmented copies were not used) 25,215 raw images (93 unreadable)
GBIF occurrence photos (iNaturalist-contributed) for 11 Amazonian/tropical useful-plant taxa internal species-ID experiment only 2,238 photos

Not obtained: DLCPD-25 (Baidu only), AgriPest (never released), INSECT204. Hemp: no labelled public set was obtainable (notebook section 43). Images are not redistributed; manifests and scripts are in the repo, so the data can be re-fetched. (An early version of this card said CCMT was blocked; it is now fetched, see notebook sections 31-32.)

7. Limitations (summary)

3 of 18 crops covered by the detector; tiny-plant recall is poor (tiled inference trades precision for recall; no setting beat full-image F1); disease labels are mostly web-scraped or lab-style and noisy (e.g. a cannabis leaf appeared under "maple, healthy" in PlantWild); PlantDoc healthy/diseased differ in photographic style; cassava disease classes are visually subtle; no on-device latency measured (the patched llama.cpp server cross-compiles for Android arm64 with NDK r27c, but has never been run on a phone); TFLite export was attempted four times and failed (onnx2tf errors), so ONNX is the only mobile format.

8. Files

best.onnx, best.pt, results.csv, args.yaml, detector_eval_yolo11n_v1.json, SHA256SUMS (copies verified byte-identical to the training outputs), infer_onnx.py, dataset builders and manifests, train_detector.py, eval_detector.py, train_cnn.py, export_cnn_onnx.py, quantize_cnn_onnx.py, infer_tiny.py, artifacts/cnn_tiny_v1/, extract_features.py, train_head.py, export_heads.py, eval_cassava.py, eval_multisource.py, diagnose.py (end-to-end demo), and the llama.cpp patch (see HOWTO.md).

Citation / credits

Artemis & Image Safari: Mutuvi S. et al., agriRxiv 2026.00491 (CIAT/Alliance of Bioversity International). d1-omni: Liquid AI. YOLO11: Ultralytics. PlantDoc, PlantWild, PlantSeg, IP102, Cassava Leaf Disease: cite per their papers. Trained on a single Tesla M40 (sm_52) with torch 1.13.1. [ORG / authors / contact: fill in]

4c. 18-crop identifier with web photos (added 2026-10-25)

artifacts/crop18_web_v1/ (MobileNetV3-L, 224 px, fp32 ONNX 11.9 MB, 18 Safari crops). Trained on Safari field photos plus 2,235 GBIF/iNaturalist photos (16 of 18 crops; the images themselves are not redistributed). Held-out estimate = site-disjoint 5-fold CV of the same recipe, 0.595-0.605 pooled over 3 seeds (n=2,824) vs 0.503-0.533 without web photos; the shipped model is fit on all rows so has no test set of its own. Expect about 0.60 or lower at a new site; crops without web photos (african yam, lentil) and finger millet (38 photos) are weaker by construction. Not INT8-quantised, not run on a phone, labels not audited by an expert. Details: LAB_NOTEBOOK sections 49-50.