Solshine's picture
Self-contained release: phone_site with runtime, RUN.md, d1 GGUFs, scripts, weights, licences by upstream terms
f79b7a1 verified
|
Raw History Blame Contribute Delete
15.6 kB

HOWTO: reproduce, run, and extend

Everything here was run on one Dell T7610 with Tesla M40s (sm_52), torch 1.13.1, on 2026-10-08. Paths below are the ones used; change ROOT as needed. Scripts live in this repo; bulky data lives under /mnt/store1/agri (not in git).

0. Ground rules learned the hard way

  • No Hugging Face transformers generate() on old GPUs (it has crashed this box). Nothing here uses transformers: the detector is a CNN (Ultralytics) and d1 runs in llama.cpp.
  • On a shared GPU box, reserve first (gpusched reserve --gpu 0 ...), pin to one GPU/NUMA node (gpu0run), watch temperature with a guard scoped to your own PID, and release when done.
  • Never pkill -f/pgrep -f a pattern that your own shell command contains (it kills your shell). Match on readlink /proc/PID/exe, a PID you saved, or a listening port.
  • Large downloads: single-stream curl -L -C - with retries (HF xet paths failed on this box). Check completeness by size/MD5, not exit code. Re-download if a "CSV" is a few hundred bytes (it was a redirect page: use curl -L).
  • Disk filled to 98%: download, downscale (max side 640), record in a manifest, delete the raw zip, repeat.

1. Environments

python3 -m venv ~/venvs/agri-vision && ~/venvs/agri-vision/bin/pip install boto3 pycocotools pillow numpy requests tqdm      # data + analysis, no torch
# detector training: reuse an existing torch 1.13.1+cu117 venv through a .pth file (so pip never touches torch), then add torchvision + ultralytics
python -m venv ~/venvs/agri-yolo && echo "<path-to>/research-pt113/lib/python3.10/site-packages" > ~/venvs/agri-yolo/lib/python3.10/site-packages/zz_pt113.pth
~/venvs/agri-yolo/bin/pip install "torchvision==0.14.1" --index-url https://download.pytorch.org/whl/cu117
~/venvs/agri-yolo/bin/pip install "ultralytics==8.3.0" opencv-python-headless onnx onnxruntime

Confirm after installing: pip list | grep -i '^torch ' still shows 1.13.1 in the shared venv.

2. Artemis / Image Safari data (anonymous S3)

python s3_list.py ImageSafari/ / 50          # list crops; bucket alliance-artemis-imagesafari, us-west-2, unsigned
python enum_annotations.py                   # -> annotation_index.json (all COCO JSONs + image dirs per crop)
python fetch_coco.py                         # downloads box-bearing COCO files (<130 MB each) to /mnt/store1/agri/coco, prints classes

Facts to know: the real prefix layout differs from the README (<crop>/annotations/<variety>/standard/<task>/<set>/); only bean, cowpea and sorghum have true box-detection sets; several annotation sets are duplicates (Safari bush bean == Artemis bush bean, two identical cowpea JSONs, two overlapping sorghum JSONs). cowpea_plant boxes include tiny noisy boxes. Always draw boxes on a few images (overlay.py) before training.

3. Detector

python build_dataset.py /mnt/store1/agri/ds_v1        # YOLO-format dataset (3 classes), EXIF-free raw pixels, longest side <=1280
python resplit.py                                     # per-crop, date/batch-grouped 70/15/15 split (the first greedy split was class-skewed; check class counts per split!)
gpusched reserve --gpu 0 --duration 5h --session <you> --harness claude-code --purpose "YOLO11n detector"
./run_train.sh yolo11n_v1 60 1.0 640 16               # smoke test first: ./run_train.sh smoke 1 0.05 640 16  (fraction takes the first N files alphabetically)
python eval_detector.py yolo11n_v1                    # CPU eval on val+test, torch vs ONNX -> detector_eval_yolo11n_v1.json

Settings: YOLO11n pretrained, imgsz 640, batch 16, amp=False (M40 has no fast fp16), patience 15. ~2.2 min/epoch; early-stopped at epoch 55 (best 40). GPU peaked ~83 C. thermal_guard.sh PID GPU LIMIT LOGFILE kills only that PID after three consecutive reads at/above the limit.

Run the exported model

python infer_onnx.py best.onnx photo.jpg --conf 0.3 --iou 0.5 --save out.jpg    # onnxruntime + numpy + PIL only

Output tensor (1, 7, 8400): cx, cy, w, h (letterboxed 640 px) + scores for common_bean, cowpea, sorghum. For Android use ONNX Runtime Mobile (not tested here). TFLite: does not work yet. onnx2tf 1.22.3 + TF 2.15 failed on YOLO11n (detect head, then C2PSA attention) and YOLOv8n (a Conv op). Don't spend long on it without a newer toolchain.

4. Disease / pest data

python disease_manifest.py plantdoc|plantwild|plantseg|ip102|cassava   # per-source parser: downscale to max 640, append rows to disease_manifest.jsonl
python make_crop_set.py                                                # ~100 Safari field images per crop for crop-ID (excludes probe images by exact key AND by image-hash stem)
python canon.py                                                        # prints how raw condition strings map to the 12 coarse concepts

Sources and exact URLs: PlantDoc codeload.github.com/pratikkayal/PlantDoc-Dataset/zip/refs/heads/master; PlantWild huggingface.co/datasets/uqtwei2/PlantWild/resolve/main/plantwild.zip; PlantSeg zenodo.org/api/records/16665770/files/Plantseg.zip/content (MD5 23385d877bc84bab51b54630055c75b8); IP102 COCO zenodo.org/api/records/15546419/files/ip102.zip/content; Cassava via HF Engineer101/cassava-leaf-disease-classification (individual files; use the repo's file list to find train_images/part_xxx/). Audit labels by eye (we did: contact sheets with the mapped label printed on each tile). Findings: PlantWild is noisy; PlantDoc healthy = white-background stock images (style confound). Leakage checks that mattered: the same photo appeared under several annotation sets (8 training images were probe images; 14 within-set duplicates) — compare image-hash stems, not only keys.

5. d1-omni-600M locally (llama.cpp) and the trunk-feature patch

git clone https://github.com/ggml-org/llama.cpp /mnt/store1/agri/llama.cpp-d1      # tested at de7fa0a (2026-10-08); older checkouts lack /v1/systemone
cmake -B build-cuda -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=52 -DLLAMA_CURL=OFF && cmake --build build-cuda -j6 --target llama-server
# models: curl -L -C - the d1-omni-600M-{Q8_0,F16}.gguf and mmproj-d1-omni-600M-{Q8_0,F16}.gguf files from LiquidAI/d1-omni-600M-GGUF
D1_EXPORT_TRUNK=1 D1_DUMP_PATH=/tmp/dump.bin gpu0run ./build-cuda/bin/llama-server -m d1-omni-600M-F16.gguf --mmproj mmproj-d1-omni-600M-F16.gguf \
    --port 8097 -b 4096 -ub 4096 -ngl 99 -np 1 --image-max-tokens 256

The patch (two small edits, kept outside upstream; patches/d1-trunk-export.patch, 17 added lines, verified to apply cleanly to llama.cpp commit de7fa0a): (1) src/models/lfm2.cpp — when D1_EXPORT_TRUNK is set, n_embd_out grows from 3 to 3+1024 and the post-output_norm trunk state at the output rows is concatenated after the 3 score columns; (2) tools/server/server-context.cpp — when D1_DUMP_PATH is set, append a record [int32 n_markers, int32 n_embd_out, float32 x n_markers*n_embd_out] per request. Requires -np 1 (records are matched to requests by order). extract_features.py asserts the record count/order and that softmax(dumped logits) reproduces the server's returned probabilities.

Questions are sent to POST /v1/systemone with files:[data-URL image] and questions:{id:{type:choice|score|noul, instructions, criteria}}. noul returns P(true). Throughput on one M40: ~0.45 s/image at 448 px. CPU-only is ~190 s for a 1080p image: use a GPU.

Probe and feature extraction

python make_probe_set.py && python sheets.py        # 204-image probe + numbered contact sheets (look at them, then fill probe_flags.json)
python probe_run.py http://127.0.0.1:8099 probe_results_f16.jsonl 896 && python probe_analyze.py probe_results_f16.jsonl
python build_extract_lists.py && for n in probe artemis_det safari_crop plantdoc cassava plantwild plantseg ip102; do python style_feats.py $n; done
./run_extraction.sh                                  # unattended: reserves GPU, starts the patched server under a thermal guard, extracts each list, tears down

6. Heads

python train_head.py --tasks crop,healthy,cond,pest   # plain PyTorch on CPU (~2 min); every number next to d1 zero-shot, a style-only control and majority class, with Wilson 95% CIs
python export_heads.py                                # refit on train+val, save artifacts/heads_v1/*.npz (+json), verify numpy forward == torch

Dedup is mandatory. PlantDoc, PlantWild and PlantSeg contain the same photos (65 of 2,578 PlantDoc images match at cosine >= 0.98 in d1 trunk-feature space; verified by eye). train_head.py flags any evaluation image with a >= 0.95 near-duplicate in the training pool (flagdups) and drops it from evaluation only. train_head_v1_before_dedup.py is the pre-dedup version, kept so the effect is auditable (up to ~4 points). The saved heads are plain NumPy: logits = ((x - mu) / sd) @ W.T + b on the 3,072-d feature vector.

Rules we hold ourselves to: report n and a Wilson interval for every rate; compare with zero-shot and with a no-semantics control; evaluate on a different source/campaign than training (PlantDoc is the cross-source test; Artemis detection images are the out-of-domain crop test); hand-check flagged samples before quoting a count.

7. Backups

Features, manifests and weights are not "recreatable": they depend on the exact d1 build, patch and hardware. Keep a private copy with a SHA-256 manifest and re-hash the copy. artifacts/yolo11n_v1/SHA256SUMS is an example (copies verified byte-identical).

8. Android (status: builds only)

cmake -B build-android -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-28 -DCMAKE_BUILD_TYPE=Release \
      -DGGML_OPENMP=OFF -DGGML_NATIVE=OFF -DLLAMA_CURL=OFF -DLLAMA_OPENSSL=OFF -DBUILD_SHARED_LIBS=OFF && cmake --build build-android -j3 --target llama-server

Produced a 185 MB (unstripped) aarch64 ELF with NDK r27c. Never executed on a device; latency and memory on a phone are unmeasured. Expect CPU-only (no GPU offload) and test the Q8_0 model (407 MB + 263 MB mmproj).

9. End-to-end demo (CPU only, no GPU needed)

# CPU-only build of the patched server (never loads CUDA): cmake -B build-cpu -DGGML_CUDA=OFF -DLLAMA_CURL=OFF && cmake --build build-cpu --target llama-server
D1_EXPORT_TRUNK=1 D1_DUMP_PATH=/tmp/dump.bin ./build-cpu/bin/llama-server -m d1-omni-600M-F16.gguf --mmproj mmproj-d1-omni-600M-F16.gguf --port 8097 -np 1 -t 6 --image-max-tokens 256 &
python diagnose.py photo.jpg --server http://127.0.0.1:8097 --dump /tmp/dump.bin --save out.jpg   # detector boxes + crop / healthy / condition head outputs as JSON, with warnings

~6-9 s per image on 6 CPU threads (d1 dominates; the ONNX detector is fast). Features from this CPU path matched the stored GPU-extracted features (cosine 1.00000, relative L2 difference ~0.001 on 3 images), so CPU and GPU extraction are interchangeable here. diagnose.py is a research demo: the detector only knows bean/cowpea/sorghum seedlings in Artemis-style photos and will mislabel other crops (it called a cassava leaf "sorghum").

10. Tiny CNN diagnoser (no d1, no llama.cpp) — added 2026-10-10

Everything below is plain PyTorch/torchvision/ONNX Runtime (no transformers), run on the same M40 box via the same gpusched + gpu{0,1}run + scoped thermal-guard protocol as section 0 (run_cnn.sh GPU NAME args... does the pinning and guard).

# 1. caches (uint8 224x224 squash-resize, row order == meta order; ~8 GB for the disease manifest, ~4 GB CCMT)
python build_img_cache.py 224                                      # disease_manifest.jsonl minus IP102 -> imgcache/img224.npy, meta.jsonl
python ccmt_manifest.py && python build_cache_generic.py imgcache/ccmt_meta.jsonl imgcache/ccmt 224      # CCMT raw .jpg (see notebook 31: names collide across folders, fetch with uuid suffix)
python ip102full_manifest.py && python build_cache_generic.py imgcache/ip102full_meta.jsonl imgcache/ip102full 224
# 2. train (MobileNetV3-L from torchvision weights; heads: healthy/diseased + masked 12-way condition); measured on one M40: E1 (21k training images, 14 epochs) ~24 min, E5 (53k images, 14 epochs) ~65 min
./run_cnn.sh 0 e5_all_ccmt --train plantwild,plantseg,plantdoc,cassava,karaagro,makerere,pretoria,ccmt --epochs 14 [--seed N] [--arch effb0] [--aug strong]
#    leave-one-source-out: simply leave a source out of --train; every source not in --train is evaluated as an external test (test split; PlantDoc/CCMT all rows), near-dups excluded
# 3. other tasks
DS=ip102full ./run_cnn_pest.sh 0 pest_ip102full 30                  # 97-class pest species, pHash-deduped test
./run_cnn_crop.sh 0 crop_sites 15                                  # site-disjoint 5-fold crop ID (Safari)
./run_cnn_ident.sh 1 species_gbif 15 gbif_amazon taxon group      # observer-grouped species ID (GBIF)
# 4. analysis
python calibrate_cnn.py RUN train_srcs eval_srcs                   # temperature + abstention curves (T fitted on source val only)
python eval_cnn_ccmt.py RUN all                                    # external CCMT evaluation, per class / crop
python eval_tta_ens.py run1,run2,run3 plantdoc,karaagro,ccmt       # TTA / seed-ensemble check
python per_condition_eval.py RUN SRC                               # per-condition error analysis
# 5. export + INT8 (opset 13 needed for per-channel QDQ; PERCENTILE calibration is required - MinMax/Entropy collapse to chance)
OPSET=13 python export_cnn_onnx.py RUN && CALIB=Percentile python quantize_cnn_onnx.py RUN karaagro && python export_cnn_onnx.py RUN   # last call: opset-12 fp32
# 6. run it (onnxruntime + PIL + numpy only)
python infer_tiny.py artifacts/yolo11n_ft1280/best.onnx artifacts/cnn_tiny_v1/cnn_tiny_int8_percentile.onnx photo.jpg --unsure 0.75 --save out.jpg

Notes: (a) the detector input size is read from the ONNX (1280 for yolo11n_ft1280, which must be run at 1280; 960 for v2/v1res960); (b) infer_tiny.py runs the classifier on each detected plant crop and on the whole image, because the classifier was trained on leaf/close-up and whole-plant photos, not on tiny Safari crops: crops below 24 px are reported as too_small; (c) confidences are raw softmax, not calibrated (notebook 33); (d) the trained model has never seen a hemp image and was not evaluated on Amazonian crops for disease.

Detector fine-tune used for the 1280 model: ./run_train.sh yolo11n_ft1280 25 1.0 1280 8 <v1res960/best.pt> 0.002 (GPU0, 3.5 h). Evaluate with DEV=0 python eval_detector.py yolo11n_ft1280 1280 ft1280 and DEV=0 python size_recall.py <weights.pt> 1280 ft1280 (model must be a path).

11. Crop ID with web photos (added 2026-10-25)

python3 -I gbif_fetch.py /mnt/store1/agri/dl/gbif_crops 150 "Zea mays" "Sorghum bicolor" ...      # GBIF/iNaturalist photos
# build imgcache/gbif_crops_meta.jsonl (add a "crop" field per taxon), then:
python build_cache_generic.py imgcache/gbif_crops_meta.jsonl imgcache/gbif_crops 224
./run_web_chain.sh                                   # matched web vs control (MAXW=0), seeds 0-2, GPU0, ~25 min each; hold a gpusched reservation first
SEED=0 gpu0run python -I train_crop_final.py crop18_web_final 15   # final all-data model (~7 min)
python -I export_crop18_onnx.py artifacts/crop18_web_v1            # ONNX opset 12 + parity check (CPU)

Web rows are train-only (fold=-1); the test set is computed from Safari rows so it is identical with and without web photos.