# HOWTO: reproduce, run, and extend Everything here was run on one Dell T7610 with Tesla M40s (sm_52), torch 1.13.1, on 2026-10-08. Paths below are the ones used; change `ROOT` as needed. Scripts live in this repo; bulky data lives under `/mnt/store1/agri` (not in git). ## 0. Ground rules learned the hard way - **No Hugging Face transformers `generate()` on old GPUs** (it has crashed this box). Nothing here uses transformers: the detector is a CNN (Ultralytics) and d1 runs in llama.cpp. - On a shared GPU box, reserve first (`gpusched reserve --gpu 0 ...`), pin to one GPU/NUMA node (`gpu0run`), watch temperature with a guard scoped to **your own PID**, and release when done. - Never `pkill -f`/`pgrep -f` a pattern that your own shell command contains (it kills your shell). Match on `readlink /proc/PID/exe`, a PID you saved, or a listening port. - Large downloads: single-stream `curl -L -C -` with retries (HF xet paths failed on this box). Check completeness by size/MD5, not exit code. Re-download if a "CSV" is a few hundred bytes (it was a redirect page: use `curl -L`). - Disk filled to 98%: download, downscale (max side 640), record in a manifest, delete the raw zip, repeat. ## 1. Environments ```bash python3 -m venv ~/venvs/agri-vision && ~/venvs/agri-vision/bin/pip install boto3 pycocotools pillow numpy requests tqdm # data + analysis, no torch # detector training: reuse an existing torch 1.13.1+cu117 venv through a .pth file (so pip never touches torch), then add torchvision + ultralytics python -m venv ~/venvs/agri-yolo && echo "/research-pt113/lib/python3.10/site-packages" > ~/venvs/agri-yolo/lib/python3.10/site-packages/zz_pt113.pth ~/venvs/agri-yolo/bin/pip install "torchvision==0.14.1" --index-url https://download.pytorch.org/whl/cu117 ~/venvs/agri-yolo/bin/pip install "ultralytics==8.3.0" opencv-python-headless onnx onnxruntime ``` Confirm after installing: `pip list | grep -i '^torch '` still shows 1.13.1 in the shared venv. ## 2. Artemis / Image Safari data (anonymous S3) ```bash python s3_list.py ImageSafari/ / 50 # list crops; bucket alliance-artemis-imagesafari, us-west-2, unsigned python enum_annotations.py # -> annotation_index.json (all COCO JSONs + image dirs per crop) python fetch_coco.py # downloads box-bearing COCO files (<130 MB each) to /mnt/store1/agri/coco, prints classes ``` Facts to know: the real prefix layout differs from the README (`/annotations//standard///`); only bean, cowpea and sorghum have true box-detection sets; several annotation sets are duplicates (Safari bush bean == Artemis bush bean, two identical cowpea JSONs, two overlapping sorghum JSONs). `cowpea_plant` boxes include tiny noisy boxes. Always draw boxes on a few images (`overlay.py`) before training. ## 3. Detector ```bash python build_dataset.py /mnt/store1/agri/ds_v1 # YOLO-format dataset (3 classes), EXIF-free raw pixels, longest side <=1280 python resplit.py # per-crop, date/batch-grouped 70/15/15 split (the first greedy split was class-skewed; check class counts per split!) gpusched reserve --gpu 0 --duration 5h --session --harness claude-code --purpose "YOLO11n detector" ./run_train.sh yolo11n_v1 60 1.0 640 16 # smoke test first: ./run_train.sh smoke 1 0.05 640 16 (fraction takes the first N files alphabetically) python eval_detector.py yolo11n_v1 # CPU eval on val+test, torch vs ONNX -> detector_eval_yolo11n_v1.json ``` Settings: YOLO11n pretrained, imgsz 640, batch 16, `amp=False` (M40 has no fast fp16), patience 15. ~2.2 min/epoch; early-stopped at epoch 55 (best 40). GPU peaked ~83 C. `thermal_guard.sh PID GPU LIMIT LOGFILE` kills only that PID after three consecutive reads at/above the limit. ### Run the exported model ```bash python infer_onnx.py best.onnx photo.jpg --conf 0.3 --iou 0.5 --save out.jpg # onnxruntime + numpy + PIL only ``` Output tensor `(1, 7, 8400)`: cx, cy, w, h (letterboxed 640 px) + scores for `common_bean, cowpea, sorghum`. For Android use ONNX Runtime Mobile (not tested here). TFLite: **does not work yet.** `onnx2tf` 1.22.3 + TF 2.15 failed on YOLO11n (detect head, then C2PSA attention) and YOLOv8n (a Conv op). Don't spend long on it without a newer toolchain. ## 4. Disease / pest data ```bash python disease_manifest.py plantdoc|plantwild|plantseg|ip102|cassava # per-source parser: downscale to max 640, append rows to disease_manifest.jsonl python make_crop_set.py # ~100 Safari field images per crop for crop-ID (excludes probe images by exact key AND by image-hash stem) python canon.py # prints how raw condition strings map to the 12 coarse concepts ``` Sources and exact URLs: PlantDoc `codeload.github.com/pratikkayal/PlantDoc-Dataset/zip/refs/heads/master`; PlantWild `huggingface.co/datasets/uqtwei2/PlantWild/resolve/main/plantwild.zip`; PlantSeg `zenodo.org/api/records/16665770/files/Plantseg.zip/content` (MD5 23385d877bc84bab51b54630055c75b8); IP102 COCO `zenodo.org/api/records/15546419/files/ip102.zip/content`; Cassava via HF `Engineer101/cassava-leaf-disease-classification` (individual files; use the repo's file list to find `train_images/part_xxx/`). **Audit labels by eye** (we did: contact sheets with the mapped label printed on each tile). Findings: PlantWild is noisy; PlantDoc healthy = white-background stock images (style confound). Leakage checks that mattered: the same photo appeared under several annotation sets (8 training images were probe images; 14 within-set duplicates) — compare image-hash stems, not only keys. ## 5. d1-omni-600M locally (llama.cpp) and the trunk-feature patch ```bash git clone https://github.com/ggml-org/llama.cpp /mnt/store1/agri/llama.cpp-d1 # tested at de7fa0a (2026-10-08); older checkouts lack /v1/systemone cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=52 -DLLAMA_CURL=OFF && cmake --build build-cuda -j6 --target llama-server # models: curl -L -C - the d1-omni-600M-{Q8_0,F16}.gguf and mmproj-d1-omni-600M-{Q8_0,F16}.gguf files from LiquidAI/d1-omni-600M-GGUF D1_EXPORT_TRUNK=1 D1_DUMP_PATH=/tmp/dump.bin gpu0run ./build-cuda/bin/llama-server -m d1-omni-600M-F16.gguf --mmproj mmproj-d1-omni-600M-F16.gguf \ --port 8097 -b 4096 -ub 4096 -ngl 99 -np 1 --image-max-tokens 256 ``` **The patch** (two small edits, kept outside upstream; `patches/d1-trunk-export.patch`, 17 added lines, verified to apply cleanly to llama.cpp commit de7fa0a): (1) `src/models/lfm2.cpp` — when `D1_EXPORT_TRUNK` is set, `n_embd_out` grows from 3 to 3+1024 and the post-`output_norm` trunk state at the output rows is concatenated after the 3 score columns; (2) `tools/server/server-context.cpp` — when `D1_DUMP_PATH` is set, append a record `[int32 n_markers, int32 n_embd_out, float32 x n_markers*n_embd_out]` per request. Requires `-np 1` (records are matched to requests by order). `extract_features.py` asserts the record count/order and that `softmax(dumped logits)` reproduces the server's returned probabilities. Questions are sent to `POST /v1/systemone` with `files:[data-URL image]` and `questions:{id:{type:choice|score|noul, instructions, criteria}}`. `noul` returns P(true). Throughput on one M40: ~0.45 s/image at 448 px. CPU-only is ~190 s for a 1080p image: use a GPU. ### Probe and feature extraction ```bash python make_probe_set.py && python sheets.py # 204-image probe + numbered contact sheets (look at them, then fill probe_flags.json) python probe_run.py http://127.0.0.1:8099 probe_results_f16.jsonl 896 && python probe_analyze.py probe_results_f16.jsonl python build_extract_lists.py && for n in probe artemis_det safari_crop plantdoc cassava plantwild plantseg ip102; do python style_feats.py $n; done ./run_extraction.sh # unattended: reserves GPU, starts the patched server under a thermal guard, extracts each list, tears down ``` ## 6. Heads ```bash python train_head.py --tasks crop,healthy,cond,pest # plain PyTorch on CPU (~2 min); every number next to d1 zero-shot, a style-only control and majority class, with Wilson 95% CIs python export_heads.py # refit on train+val, save artifacts/heads_v1/*.npz (+json), verify numpy forward == torch ``` **Dedup is mandatory.** PlantDoc, PlantWild and PlantSeg contain the same photos (65 of 2,578 PlantDoc images match at cosine >= 0.98 in d1 trunk-feature space; verified by eye). `train_head.py` flags any evaluation image with a >= 0.95 near-duplicate in the training pool (`flagdups`) and drops it from evaluation only. `train_head_v1_before_dedup.py` is the pre-dedup version, kept so the effect is auditable (up to ~4 points). The saved heads are plain NumPy: `logits = ((x - mu) / sd) @ W.T + b` on the 3,072-d feature vector. Rules we hold ourselves to: report n and a Wilson interval for every rate; compare with zero-shot and with a no-semantics control; evaluate on a different source/campaign than training (PlantDoc is the cross-source test; Artemis detection images are the out-of-domain crop test); hand-check flagged samples before quoting a count. ## 7. Backups Features, manifests and weights are not "recreatable": they depend on the exact d1 build, patch and hardware. Keep a private copy with a SHA-256 manifest and re-hash the copy. `artifacts/yolo11n_v1/SHA256SUMS` is an example (copies verified byte-identical). ## 8. Android (status: builds only) ```bash cmake -B build-android -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-28 -DCMAKE_BUILD_TYPE=Release \ -DGGML_OPENMP=OFF -DGGML_NATIVE=OFF -DLLAMA_CURL=OFF -DLLAMA_OPENSSL=OFF -DBUILD_SHARED_LIBS=OFF && cmake --build build-android -j3 --target llama-server ``` Produced a 185 MB (unstripped) aarch64 ELF with NDK r27c. **Never executed on a device**; latency and memory on a phone are unmeasured. Expect CPU-only (no GPU offload) and test the Q8_0 model (407 MB + 263 MB mmproj). ## 9. End-to-end demo (CPU only, no GPU needed) ```bash # CPU-only build of the patched server (never loads CUDA): cmake -B build-cpu -DGGML_CUDA=OFF -DLLAMA_CURL=OFF && cmake --build build-cpu --target llama-server D1_EXPORT_TRUNK=1 D1_DUMP_PATH=/tmp/dump.bin ./build-cpu/bin/llama-server -m d1-omni-600M-F16.gguf --mmproj mmproj-d1-omni-600M-F16.gguf --port 8097 -np 1 -t 6 --image-max-tokens 256 & python diagnose.py photo.jpg --server http://127.0.0.1:8097 --dump /tmp/dump.bin --save out.jpg # detector boxes + crop / healthy / condition head outputs as JSON, with warnings ``` ~6-9 s per image on 6 CPU threads (d1 dominates; the ONNX detector is fast). Features from this CPU path matched the stored GPU-extracted features (cosine 1.00000, relative L2 difference ~0.001 on 3 images), so CPU and GPU extraction are interchangeable here. `diagnose.py` is a research demo: the detector only knows bean/cowpea/sorghum seedlings in Artemis-style photos and will mislabel other crops (it called a cassava leaf "sorghum"). ## 10. Tiny CNN diagnoser (no d1, no llama.cpp) — added 2026-10-10 Everything below is plain PyTorch/torchvision/ONNX Runtime (no transformers), run on the same M40 box via the same `gpusched` + `gpu{0,1}run` + scoped thermal-guard protocol as section 0 (`run_cnn.sh GPU NAME args...` does the pinning and guard). ```bash # 1. caches (uint8 224x224 squash-resize, row order == meta order; ~8 GB for the disease manifest, ~4 GB CCMT) python build_img_cache.py 224 # disease_manifest.jsonl minus IP102 -> imgcache/img224.npy, meta.jsonl python ccmt_manifest.py && python build_cache_generic.py imgcache/ccmt_meta.jsonl imgcache/ccmt 224 # CCMT raw .jpg (see notebook 31: names collide across folders, fetch with uuid suffix) python ip102full_manifest.py && python build_cache_generic.py imgcache/ip102full_meta.jsonl imgcache/ip102full 224 # 2. train (MobileNetV3-L from torchvision weights; heads: healthy/diseased + masked 12-way condition); measured on one M40: E1 (21k training images, 14 epochs) ~24 min, E5 (53k images, 14 epochs) ~65 min ./run_cnn.sh 0 e5_all_ccmt --train plantwild,plantseg,plantdoc,cassava,karaagro,makerere,pretoria,ccmt --epochs 14 [--seed N] [--arch effb0] [--aug strong] # leave-one-source-out: simply leave a source out of --train; every source not in --train is evaluated as an external test (test split; PlantDoc/CCMT all rows), near-dups excluded # 3. other tasks DS=ip102full ./run_cnn_pest.sh 0 pest_ip102full 30 # 97-class pest species, pHash-deduped test ./run_cnn_crop.sh 0 crop_sites 15 # site-disjoint 5-fold crop ID (Safari) ./run_cnn_ident.sh 1 species_gbif 15 gbif_amazon taxon group # observer-grouped species ID (GBIF) # 4. analysis python calibrate_cnn.py RUN train_srcs eval_srcs # temperature + abstention curves (T fitted on source val only) python eval_cnn_ccmt.py RUN all # external CCMT evaluation, per class / crop python eval_tta_ens.py run1,run2,run3 plantdoc,karaagro,ccmt # TTA / seed-ensemble check python per_condition_eval.py RUN SRC # per-condition error analysis # 5. export + INT8 (opset 13 needed for per-channel QDQ; PERCENTILE calibration is required - MinMax/Entropy collapse to chance) OPSET=13 python export_cnn_onnx.py RUN && CALIB=Percentile python quantize_cnn_onnx.py RUN karaagro && python export_cnn_onnx.py RUN # last call: opset-12 fp32 # 6. run it (onnxruntime + PIL + numpy only) python infer_tiny.py artifacts/yolo11n_ft1280/best.onnx artifacts/cnn_tiny_v1/cnn_tiny_int8_percentile.onnx photo.jpg --unsure 0.75 --save out.jpg ``` Notes: (a) the detector input size is read from the ONNX (1280 for `yolo11n_ft1280`, which must be run at 1280; 960 for v2/v1res960); (b) `infer_tiny.py` runs the classifier on each detected plant crop **and** on the whole image, because the classifier was trained on leaf/close-up and whole-plant photos, not on tiny Safari crops: crops below 24 px are reported as `too_small`; (c) confidences are raw softmax, not calibrated (notebook 33); (d) the trained model has never seen a hemp image and was not evaluated on Amazonian crops for disease. Detector fine-tune used for the 1280 model: `./run_train.sh yolo11n_ft1280 25 1.0 1280 8 0.002` (GPU0, 3.5 h). Evaluate with `DEV=0 python eval_detector.py yolo11n_ft1280 1280 ft1280` and `DEV=0 python size_recall.py 1280 ft1280` (model must be a path). ## 11. Crop ID with web photos (added 2026-10-25) ```bash python3 -I gbif_fetch.py /mnt/store1/agri/dl/gbif_crops 150 "Zea mays" "Sorghum bicolor" ... # GBIF/iNaturalist photos # build imgcache/gbif_crops_meta.jsonl (add a "crop" field per taxon), then: python build_cache_generic.py imgcache/gbif_crops_meta.jsonl imgcache/gbif_crops 224 ./run_web_chain.sh # matched web vs control (MAXW=0), seeds 0-2, GPU0, ~25 min each; hold a gpusched reservation first SEED=0 gpu0run python -I train_crop_final.py crop18_web_final 15 # final all-data model (~7 min) python -I export_crop18_onnx.py artifacts/crop18_web_v1 # ONNX opset 12 + parity check (CPU) ``` Web rows are train-only (fold=-1); the test set is computed from Safari rows so it is identical with and without web photos.