scorevision: push artifact

training/DATASET.md

@@ -0,0 +1,133 @@
+# Detect-crime — Dataset Sources & Pipeline
+
+The king's training set is small and synthetic — that's why per-class recalls collapse on
+balaclava (3.4%), glove (6.4%), and spray paint (16.1%). The single biggest score lever is
+**more, real, in-domain data**, especially for the four catastrophic classes.
+
+## Target classes (target order — must match `class_names.txt`)
+
+| id | name | king recall | priority |
+|---:|---|---:|---|
+| 0 | balaclava | 0.034 | **critical** |
+| 1 | bat | 0.143 | high |
+| 2 | glove | 0.064 | **critical** |
+| 3 | graffiti | 0.321 | medium |
+| 4 | hoodie | 0.274 | medium (carries the IoU mean alone) |
+| 5 | spray paint | 0.161 | high |
+
+## Sources
+
+### Tier 1 — in-domain (gold)
+
+**`manak0/Detect-crime` `latestAnnotatedChallenge` API.** Same pattern as petrol-station: the
+console at `console.scorevision.io/api/v2/elements/manak0%2FDetect-crime?lookback_days=30`
+returns one `latestAnnotatedChallenge` record at a time. The record rotates as the runner
+emits fresh challenges, so polling at ~120s for several hours / days accumulates distinct
+in-domain frames *with the king's predictions attached*. Use those predictions as silver
+labels (king is 0.576, so they're noisy but in-domain).
+
+Use `poll_manako.py` to collect; use `build_dataset.py --manako` to integrate.
+
+### Tier 2 — public detection datasets (silver / gold mix)
+
+| class | dataset | notes |
+|---|---|---|
+| **balaclava** | Roboflow Universe `balaclava-detection` (multiple variants, ~1k–5k labeled imgs) | search "balaclava" on universe.roboflow.com; download YOLO format |
+| balaclava | MAFA — masked-face dataset (35k labeled faces, occluded) | ~3k images contain ski-mask / balaclava-style coverage; relabel as "balaclava" only when the mouth+nose+ears are all occluded |
+| **bat** | COCO 2017, class 35 "baseball bat" (~3.3k train + ~150 val instances) | already labeled YOLO-compatible after `pycocotools` re-export; map cls 35 -> our cls 1 |
+| bat | Open Images V7, class `Baseball bat` | larger, but noisier labels |
+| **glove** | Roboflow Universe `gloves-detection` / `ppe-detection` | ~5k–10k labeled; merges leather, latex, work gloves |
+| glove | COCO 2017 has no "glove" class; ImageNet n02883344 "ski glove" / n03775546 "boxing glove" — classification only, but useful for crops |
+| **graffiti** | Roboflow Universe `graffiti-detection` | several variants 500–3k imgs |
+| graffiti | STORM-IRMA Graffiti Dataset (Bardienus et al., academic release) | annotated graffiti tags + region bboxes |
+| **hoodie** | DeepFashion2 (item id `short_sleeve_top + long_sleeve_outwear`) | very large; relabel "hooded" subset only |
+| hoodie | Roboflow Universe `clothing-detection` | hooded sweatshirt subset |
+| **spray paint** | Roboflow Universe `spray-paint-can`/`graffiti-tools` | small, 200–800 imgs each |
+| spray paint | ImageNet n04270147 "spray can" | classification only — useful as copy-paste source crops |
+
+Roboflow Universe links rotate; the build script uses the public Roboflow REST API
+(`https://universe.roboflow.com/...`) and falls back to anonymous downloads for permissive-
+licensed datasets. **Always check each dataset's license before use** — if a dataset is
+non-redistributable, paste it into `--extra-dir` from a local copy instead.
+
+### Tier 3 — synthetic & augmentation
+
+- **Copy-paste augmentation** (built into ultralytics: `copy_paste=0.30`). Disproportionately
+  helps rare classes by pasting their crops into other training images.
+- **Mosaic + mixup** (also built-in): standard for YOLOv11.
+- **Photometric augmentation**: HSV jitter, brightness scaling. CCTV crime footage is
+  high-variance lighting (flashbang, IR night vision, daylight) — push `hsv_v` to 0.5.
+
+## Pipeline
+
+```
+sources                                              → KingLabeler (silver) ──┐
+  ├─ manako_pool/   (poll_manako.py)                   king's preds attached  │
+  ├─ roboflow/      (download.py per-class)                                   │
+  ├─ coco_bat/      (subset extractor)                                        ├─ build_dataset.py
+  ├─ openimages/    (`oidv6` CLI, optional)                                   │  → YOLO data.yaml
+  └─ extra/         (manual / verified frames)                                │
+                                                                              ┘
+```
+
+## Stage A vs Stage B
+
+- **Stage A — silver pretrain**: train on the **assembled silver corpus** (Roboflow + COCO
+  + king-labeled manako + ImageNet copy-paste source). Targets: `200 epochs` at `imgsz=1280`,
+  YOLOv11s. Use heavy augmentation. Save EMA. Expect val mAP@50 around 0.30–0.45 — well
+  above king's 0.14, but synthetic-eval-flattering.
+- **Stage B — clean fine-tune**: hand-verify (or LLM-verify with Qwen2-VL) ~300 manako frames
+  drawn from the latest 7-day window. This is the small-but-clean set that aligns most
+  closely with the live evaluator. Train `50 epochs` at `imgsz=1280` on Stage A weights with
+  reduced augmentation. Expect val mAP@50 to recede slightly (cleaner GT, harder eval) but
+  live-element score to *rise*, because we've shifted the distribution toward what's actually
+  scored on subnet.
+
+## Class balancing
+
+The king's set is heavily skewed toward hoodie (~half of GT boxes). Without rebalancing, our
+model would inherit that. Two interventions:
+
+1. **Weighted sampling** in dataloader (`sampler=` weighted by 1/sqrt(class_count)).
+2. **Copy-paste augmentation** sourced from a curated rare-class crop bank
+   (`crime_miner/training/_rare_crops/{balaclava, glove, spray_paint}/`). The build script
+   produces this bank as a byproduct.
+
+## What the build script writes
+
+```
+data/
+├── data.yaml                # ultralytics-style; names + train/val paths
+├── images/
+│   ├── train/{src}_{sha}.jpg  # one per source image
+│   └── val/...
+└── labels/
+    ├── train/{src}_{sha}.txt  # YOLO format: cls cx cy w h (normalized)
+    └── val/...
+```
+
+Manako frames are forced into `val` so the held-out set lives in the real evaluation domain.
+
+## Hard-class strategy: balaclava
+
+Balaclava is the lowest-recall class. Two failure modes from the king's perspective:
+
+1. **Box too small at 640 input** (king's stretch resize destroys 30 px head crops).
+   *Solved* by 1280 letterboxed input + multi-scale TTA. No retraining required.
+2. **Few training examples**. Mitigations:
+   - Pull MAFA (~3k masked-face images) and select the ~30% subset where the mask covers the
+     entire head (i.e., balaclava, not surgical mask).
+   - Roboflow `balaclava-detection-v3` has ~1.5k labeled.
+   - Synthesize via copy-paste of balaclava crops onto random `hoodie` GT boxes (we have many
+     hoodie images; pasting a balaclava crop on top is plausible-looking).
+
+Target: **5,000 balaclava-positive training images** going into Stage A.
+
+## Notes on label vocabulary
+
+Both the manak0 baseline and our model emit class names matching `class_names.txt` exactly.
+**Do not rename or reorder** — class id is what's compared on-subnet.
+
+If the live element uses SAM3 pseudo-GT (not real GT), SAM3 will be prompted with these
+exact strings as text labels (`scorevision/vlm_pipeline/sam3/detect_objects.py:36`). So the
+class names in `class_names.txt` *are* the SAM3 prompts.