docs/large_geotiff_findings.md

Large GeoTIFF Detection — Verification Findings

Verifies the RCA (Root Cause Analysis) claims against the actual codebase and a real large GeoTIFF, since production input = large single-GeoTIFF uploads (not the 300x300px Delhi tiles used in the Day 1-7 calibration work).

Test setup

• Real Sentinel-2 imagery, 8192x8192px pair (2019-06-29 vs 2026-06-17, Delhi MGRS tile 43RFM), pulled via windowed COG read into data/large_geotiff_test/ (~180 MB each, gitignored).
• Run through the actual run_detection() pipeline, AI-Based Deep Learning method, sensitivity 0.5.

RCA issue verification (code + empirical)

#	RCA Issue	Verified status
1	Default mode downscaled destroys detail	CONFIRMED. Default is downscaled, DETECTION_MAX_SIDE=4096. A 50x30px building on a 50,000px-wide GeoTIFF -> 4x2px after downscale. fullres_tiled mode exists but is off by default.
2	AdaptFormer/LEVIR-CD domain mismatch	CONFIRMED (Day 3, model_changed_px≈0). Fix = fine-tuning (Uday's track).
3	Vegetation suppression kills real change	NOT AN ISSUE. git blame shows the correct asymmetric suppress/boost logic was already merged 2026-04-11. RCA describes an older code snapshot.
4	Registration 0.45x penalty on SSIM/edge	Code real, impact negligible. A/B tested on 4 real pairs: <0.001% difference in final score (downstream normalization cancels the uniform scaling).
5-7	Windowed threshold / tile seams / min_region_area scaling	Downstream of Issue 1; require the fullres path to be exercised. Partially addressed by the timing finding below.

Empirical performance finding (the key production blocker)

CPU-only inference at production scale is impractically slow:

Mode	Resolution	change%	Regions	Completed on CPU?
downscaled (default)	4096px	0.614%	11	Yes — ~61 min
fullres_tiled (capped)	5120px	0.464%	18	Yes — but needed overnight (machine idled/throttled; ~46 min active CPU observed early, then stalled)
fullres_tiled (true native)	8192px	—	—	No — >5h across two attempts, never completed

Two findings:

1. The fix works. fullres_tiled produces more granular detections (18 regions vs 11) — higher resolution breaks up what downscaling merges into fewer coarse blobs, so smaller/finer changes are caught. Issue 1's fix path is functional end-to-end, not just theoretical. (Lower total change% with more regions = finer, more localized detections rather than a few over-merged coarse ones.)

2. But it's impractical on CPU. Even the current default downscaled mode takes ~1 hour per pair on an 8192px input; true native fullres_tiled never completed in >5h. This is the same root constraint crashing Uday's fine-tuning (CPU can't handle larger datasets) — GPU is required for both the detection and training tracks at production scale, not optional.

Recommendation

1. Issue 1 fix (highest leverage): auto-detect fullres_tiled for GeoTIFF inputs, OR change the default — BUT this is only usable with GPU inference, given the timing above.
2. GPU provisioning is the gating dependency for the whole large-GeoTIFF goal — surfaced independently by both the detection timing and the training crashes.
3. The Day 1-7 Delhi calibration (cl_q=0.90, +53% F1 on small tiles) remains valid for small/pre-tiled inputs but is unverified at production GeoTIFF scale — do not present it as production-proven without that caveat.

Generated during Day 7+ large-GeoTIFF verification. Raw run logs under runs/.