docs/delhi_eval/README.md

Delhi Evaluation Set

Real before/after image pairs (+ hand-labeled ground-truth masks) used to measure change-detection accuracy against actual Delhi imagery, instead of LEVIR-CD tiles or synthetic cases. This is the primary metric for the accuracy-improvement sprint (see Accuracy_Improvement_Plan.xlsx); LEVIR and the synthetic benchmark (scripts/validate_detection.py --benchmark) remain regression gates only.

Layout

docs/delhi_eval/
  manifest.json       # pair registry — see schema below
  labels/              # binary GT change masks, one PNG per pair (added Day 2-3)
    <pair_id>.png

manifest.json schema

{
  "pairs": [
    {
      "pair_id": "delhi_0001",
      "before_path": "library_sources/2024/site_a.tif",
      "after_path": "library_sources/2026/site_a.tif",
      "date_before": "2024-03-10",
      "date_after": "2026-02-18",
      "gsd": 0.3,
      "zone": "South Delhi",
      "change_types": ["building"],
      "gt_mask": null,
      "notes": ""
    }
  ]
}

Field	Required	Notes
pair_id	yes	unique, auto-assigned as delhi_%04d
before_path / after_path	yes	path relative to repo root, must exist on disk
date_before / date_after	no	YYYY-MM-DD if known
gsd	no	ground sample distance in meters, if known
zone	no	free-text area/locality label
change_types	yes	subset of building, road, vegetation, mixed_gsd, other — drives coverage checks
gt_mask	yes (auto)	set once docs/delhi_eval/labels/<pair_id>.png exists; null until labeled
notes	no	anything unusual about the pair (misalignment, cloud cover, etc.)

Current status (2026-07-13)

32 pairs logged via scripts/build_delhi_pairs_sentinel2.py, sourced from free Sentinel-2 L2A imagery (Copernicus/AWS Open Data, MGRS tile 43RFM), covering Delhi's western/southwestern periphery (the tile doesn't reach the far-east Trans-Yamuna area). 2019-06-29 vs 2026-06-17 — deliberately season-matched (same time of year, ~7 years apart) so diffs reflect real structural change rather than monsoon/crop-calendar swings.

This satisfies the pair-count target but not full category coverage:

• mixed_gsd and vegetation: covered (32 pairs each)
• building and road: not covered — Sentinel-2 is 10m GSD, too coarse to resolve individual buildings/roads reliably. Real building/road-level pairs still need either DDA's own GeoTIFFs (see docs/IMPLEMENTATION_PLAN_DDA.md, "Blocked until DDA provides...") or another higher-resolution source.
• GT masks: not generated — hand-labeling in QGIS/LabelMe is still a separate step (see Workflow below). Diff-based scores were only used to select likely-changed locations, not as ground truth.

Regenerate/extend with:

python scripts/build_delhi_pairs_sentinel2.py --count 30

Workflow

# one-time
python scripts/build_delhi_manifest.py --init

# see what imagery is on disk to help pick pairs
python scripts/build_delhi_manifest.py --scan

# log a pair once you've picked it
python scripts/build_delhi_manifest.py --add \
    --before library_sources/2024/site_a.tif --after library_sources/2026/site_a.tif \
    --zone "South Delhi" --gsd 0.3 --change-types building,road

# check progress against the >=30 pairs / coverage target
python scripts/build_delhi_manifest.py --validate

# sanity-run the detection harness across every pair in the manifest
# (no GT needed yet — this is the Day 1 "runs end-to-end" check)
python scripts/compare_methods.py --manifest docs/delhi_eval/manifest.json --out runs/manifest_scan

Ground-truth masks (labels/<pair_id>.png) are added in the next pass (hand-labeled in QGIS/LabelMe); --validate will flag pairs still missing one.