docs/IMPLEMENTATION_PLAN_DDA.md

DDA Change Detection — Implementation Plan (Dev Space Only)

Document: Scope of Work → Engineering Plan
Client: DDA (Delhi Development Authority)
Target environment: Hugging Face satdetect-dev only (hf-dev remote, master branch)
Production (satdetect) must not receive these changes until explicit sign-off.

---

1. Executive summary

The SOW transforms the current “upload two PNGs → run detection → view history” demo into a departmental change-monitoring system with:

• A persistent image library (GeoTIFF, hierarchical metadata, search)
• Library-driven comparison (Base T1 vs Comparison T2)
• Async AI detection with geo-referenced outputs
• Reports (browser + PDF + email)
• Geo-navigation and clickable regions
• Human review (Confirmed / False Positive) and export to departmental systems

The existing codebase is a strong Phase-0 foundation: AdaptFormer-based detection, region classification, overlay UI, email notifier, and history. Roughly 40% of FR-07 exists; FR-01–FR-06 and FR-08 are largely net-new.

Recommended approach: Implement behind a APP_MODE=dda feature flag on dev, extend the database and API vertically, and keep production on the current simplified flow indefinitely.

---

2. SOW requirements → current state

ID	Requirement	Current state	Gap severity
FR-01	Centralized image library (Zone → Village → Area → Year or Grid)	Zone/Village are free-text dropdowns on upload; images discarded after run except thumbnails	Large
FR-02	GeoTIFF upload + validation + manual georef fallback	PNG/JPEG only, 20 MB, Pillow RGB load	Large
FR-03	Pick Base/Comparison from library or drag from tree	Local file upload only	Medium (depends on FR-01)
FR-04	AI detection as background job; simplified change types; lat/lng per region	Synchronous /api/detect; rich internal taxonomy; pixel coords only	Medium–Large
FR-05	In-app notification + report link; PDF export; persistent reports	Email optional; HTML modal report; no PDF; history exists	Medium
FR-06	Navigate to change by lat/lng; Locate action	No geospatial metadata	Large (depends on FR-02)
FR-07	Clickable regions auto-pan viewer; side-by-side Base/Comparison	Region row hover + compare slider partially exist	Small–Medium
FR-08	Confirm / False Positive; submit to dept API or export	Not implemented	Medium

SOW change-type mapping (canonical output)

Map internal detection_engine labels → DDA report types:

DDA type	Internal sources (examples)
New Construction	New Construction/Building, Temporary Structure, Road/Pavement Change (New)
Demolition	Demolition/Clearing, Partial/Full Demolition
Extension	Expansion, Widening, Renovation
Vegetation Change	Vegetation Change (+ sub-types)
Other	Water Body, Bare Land/Soil, Unclassified

Store both dda_change_type (report) and internal_object_type (model/debug) on each region.

---

3. Deployment & branching strategy

master (DDA features)  ──push──►  hf-dev  →  coderuday21/satdetect-dev
production             ──push──►  hf       →  coderuday21/satdetect  (frozen)

Dev-only safeguards

1. APP_MODE env var on satdetect-dev: dda enables library, GeoTIFF, jobs, RBAC, review UI. Default / production: legacy.
2. Separate DB file on dev: data/dda_app.db (avoid mixing guest history with library assets).
3. All new routes under /api/dda/* or gated by APP_MODE in main.py.
4. CI checklist before any production promote: diff must not touch DDA modules unless mode flag documented.

---

4. Target architecture

flowchart TB
    subgraph ui [Frontend SPA]
        LIB[Image Library Tree + Search]
        CMP[Comparison Picker T1/T2]
        JOB[Job Status / Notifications]
        RPT[Report Viewer + PDF]
        REV[Review Confirmed / FP]
    end

    subgraph api [FastAPI]
        AUTH[RBAC Auth]
        IMG[Image Library API]
        DET[Detection Job API]
        RPTAPI[Report + Export API]
        DEPT[Department Export Adapter]
    end

    subgraph data [Storage]
        DB[(SQLite / PostgreSQL)]
        FS[File Store: GeoTIFF + pyramids + overlays]
        JQ[Job Queue Table]
    end

    subgraph proc [Processing]
        GEO[rasterio ingest + CRS normalize]
        PIPE[detection_engine.run_detection]
        GEOOUT[Region centroid → lat/lng]
    end

    LIB --> IMG
    CMP --> DET
    DET --> JQ
    JQ --> proc
    proc --> DB
    proc --> FS
    JOB --> DET
    RPT --> RPTAPI
    REV --> DEPT
    AUTH --> api

---

5. Data model (new & extended)

5.1 Hierarchy (FR-01)

Two organization modes (user selects at library root):

A. Administrative (default for DDA Delhi)
Zone → Village → Area → Year → [ImageAsset…]

B. Grid
GridId (e.g. G-12-04) → Year → [ImageAsset…]

Implement as normalized tables (not hardcoded JS):

zones(id, name, mode)           -- mode: admin | grid_parent
villages(id, zone_id, name)     -- nullable for grid mode
areas(id, village_id, name)
library_years(id, area_id, year) -- optional grouping node; images also store year directly

For MVP, Area can be a free-text field on ImageAsset if DDA taxonomy is not finalized.

5.2 ImageAsset (FR-01, FR-02)

Column	Purpose
id, uuid	Primary identifiers
zone_id, village_id, area_name, year, grid_id	Hierarchy
capture_date	Required metadata
source	satellite | drone
format	geotiff
file_path, thumb_path, preview_path	Storage
crs, bounds_json	Geo metadata (WGS84 bbox)
width, height, file_size_bytes	Validation
has_georef	bool
manual_location_json	Fallback corner coords + notes if no embedded georef
uploaded_by, created_at	Audit

5.3 DetectionJob (FR-04 async)

Column	Purpose
id, status	queued | running | completed | failed
base_image_id, comparison_image_id	FK → ImageAsset
method, params_json	Detection settings
run_id	FK → DetectionRun when done
error_message, started_at, completed_at	Job lifecycle
notify_email, notify_in_app	FR-05

5.4 Extend DetectionRun

Add: base_image_id, comparison_image_id, job_id, report_pdf_path, status (draft | published).

5.5 Extend region JSON schema

Each region gains:

{
  "ddaChangeType": "New Construction",
  "confidence": 0.87,
  "areaSqM": 142.5,
  "centroid": { "x": 412, "y": 288 },
  "latLng": { "lat": 28.6139, "lng": 77.2090 },
  "bbox": { "x": 380, "y": 260, "w": 64, "h": 56 },
  "reviewStatus": "pending",
  "reviewedBy": null,
  "reviewedAt": null
}

5.6 RegionReview + export batch (FR-08)

region_reviews(id, run_id, region_id, status, reviewer_id, notes, submitted_at)
export_batches(id, run_id, format, payload_path, dept_api_response, created_at)

5.7 RBAC (FR-01, FR-02)

Extend User:

role: viewer | uploader | analyst | admin

Role	View library	Upload	Run detection	Review/submit
viewer	✓
uploader	✓	✓
analyst	✓	✓	✓	✓
admin	✓	✓	✓	✓ + manage hierarchy

Dev Space: Re-enable login UI when APP_MODE=dda. Production stays guest/no-login.

---

6. API design (dev / DDA mode)

6.1 Image library

Method	Path	Description
GET	/api/dda/hierarchy	Tree: zones → villages → areas → years (+ counts)
GET	/api/dda/images	Filter: zone, village, area, year, date_from, date_to, grid_id, q
POST	/api/dda/images/upload	Multipart GeoTIFF + metadata form
GET	/api/dda/images/{id}	Metadata + thumb URL
GET	/api/dda/images/{id}/preview	Downsampled PNG/Web tile for viewer
DELETE	/api/dda/images/{id}	Admin only

Upload validation (FR-02):

1. Extension .tif / .tiff
2. Max size: 500 MB dev (configurable; DDA responsible for suitable resolution per SOW)
3. rasterio.open() — verify readable, extract CRS/transform
4. If no georef: require manual_bounds (SW/NE lat-lng) + capture_date before commit
5. Generate 512px thumbnail + 2048px preview for UI
6. Return { id, status: "success" | "failed", errors[] }

6.2 Comparison & detection jobs

Method	Path	Description
POST	/api/dda/jobs	Body: { baseImageId, comparisonImageId, method?, notifyEmail? } → { jobId }
GET	/api/dda/jobs/{id}	Status + progress + runId when complete
GET	/api/dda/jobs	User's recent jobs (in-app notification feed)
POST	/api/dda/detect	Legacy sync path for small previews only (keep for HF debugging)

Job runner logic:

POST /jobs → insert DetectionJob(queued)
           → return 202 + jobId
Background worker thread:
  1. Load ImageAsset paths via rasterio
  2. Align pair (same CRS; reproject if needed)
  3. Windowed read → RGB numpy arrays (max 1600px for DL)
  4. run_detection(...)
  5. For each region: pixel centroid → lat/lng via transform
  6. Map object types → ddaChangeType
  7. Save DetectionRun + overlay + update job completed
  8. Fire notification (in-app row + optional email with report link)

HF timeout mitigation: Job endpoint returns immediately; worker runs in-process with status polling. For GeoTIFFs >60s, use pyramid preview for detection (DDA supplies suitable resolution) and store full metadata separately. Document in UI: “Analysis uses optimized resolution; full GeoTIFF retained in library.”

6.3 Reports (FR-05)

Method	Path	Description
GET	/api/dda/reports/{run_id}	Full report JSON (existing shape + geo fields)
GET	/api/dda/reports/{run_id}/pdf	Generate/download PDF
POST	/api/dda/reports/{run_id}/notify	Email with link to report

PDF engine: weasyprint or reportlab — HTML template reuse from templates/ChangeDetection.html.

In-app notification: Bell icon + /api/dda/jobs?status=completed&unseen=true.

6.4 Geo-navigation (FR-06, FR-07)

No external map SDK required for MVP — image viewer geo mode:

• Report shows lat, lng per region
• Locate button: pan/zoom canvas to bbox center at 200% zoom, pulse highlight
• Optional Phase 2: Leaflet minimap with footprint rectangle

Frontend already has region hover → bbox highlight; extend to click row → scroll viewer + zoom.

Side-by-side: keep existing compare slider; add locked T1 | T2 | Overlay tabs when images loaded from library.

6.5 Review & departmental export (FR-08)

Method	Path	Description
PATCH	/api/dda/reports/{run_id}/regions/{region_id}	{ reviewStatus: confirmed | false_positive, notes? }
POST	/api/dda/reports/{run_id}/submit	Submit confirmed only
GET	/api/dda/reports/{run_id}/export.{csv|xlsx|pdf}	Structured export

Department adapter pattern:

class DepartmentExporter(Protocol):
    def submit(self, run: DetectionRun, regions: list) -> ExportResult: ...

class ApiExporter(DepartmentExporter):      # DEPT_API_URL + DEPT_API_KEY
class FileExporter(DepartmentExporter):     # CSV/XLSX fallback

False positives: persist in region_reviews with status=false_positive for future training export (/api/dda/training/export — admin only, Phase 2).

---

7. GeoTIFF processing pipeline (accurate logic)

7.1 Ingest

def ingest_geotiff(path: Path) -> GeoImageMeta:
    with rasterio.open(path) as src:
        crs = src.crs
        bounds = transform_bounds(src.crs, "EPSG:4326", *src.bounds)
        has_georef = crs is not None and src.transform != rasterio.identity()
        # Prefer RGB or first 3 bands; handle 1-band grayscale
        preview = read_downsampled(src, max_side=2048)
    return meta

Dependencies to add (dev only branch): rasterio, pyproj (GDAL wheels on Docker).

7.2 Pair compatibility check (before job starts)

Check	Action
Same CRS or reproject T2 → T1 CRS	Reproject comparison raster
Overlapping bounds	Warn if <50% overlap
Band count	Harmonize to RGB
Resolution ratio	Warn if >2× GSD difference

7.3 Pixel → lat/lng

def region_centroid_latlng(region_bbox, transform, crs):
    cx = region_bbox.x + region_bbox.w / 2
    cy = region_bbox.y + region_bbox.h / 2
    lng, lat = rasterio.transform.xy(transform, cy, cx, offset="center")
    if crs != EPSG:4326:
        lng, lat = transform_coords(crs, 4326, lng, lat)
    return lat, lng

7.4 Area in m²

Use pixel area × GSD² from transform (for projected CRS in meters).

---

8. Frontend plan (dev UI)

Replace single upload card with tabbed workflow when APP_MODE=dda:

Tab 1 — Image Library

• Left: collapsible tree (Zone → Village → Area → Year)
• Right: grid of thumbnails + metadata
• Upload modal: GeoTIFF + form (hierarchy, capture date, source, manual bounds if needed)
• Search bar + filters

Tab 2 — Change Detection

• Two slots: Base (T1) | Comparison (T2)
• Drag image from library OR picker dialog
• Show thumb + capture date for each
• Run → job submitted → progress spinner → auto-open report on complete

Tab 3 — Reports & Review

• List of completed runs
• Open report → regions table with DDA types, lat/lng, confidence, area
• Locate per row; click row → pan viewer
• Toggle Confirmed / False Positive
• Submit to department / Download export

Tab 4 — Notifications

• Job completion feed (FR-05 in-app)

Auth screens: Login/register return on dev only (APP_MODE=dda).

---

9. Implementation phases

Phase 0 — Foundation (3–4 days)

• APP_MODE flag + route gating in main.py
• Alembic-style migrations or existing startup migrations for new tables
• rasterio in requirements.txt + Docker GDAL base image update (dev Dockerfile branch)
• Seed script: import current DELHI_ZONES into DB
• Dev README section

Phase 1 — Image library (FR-01, FR-02) (5–7 days)

• Models: hierarchy + ImageAsset
• Upload API + validation + thumbnail generation
• Library tree + search UI
• RBAC: uploader vs viewer

Phase 2 — Library-based comparison (FR-03) (2–3 days)

• T1/T2 picker from library
• Drag-and-drop from tree to comparison slots
• Preview + capture date display

Phase 3 — Async detection + geo output (FR-04, FR-06) (5–6 days)

• DetectionJob worker + polling API
• GeoTIFF → RGB pipeline integration with detection_engine
• DDA change type mapping + lat/lng + area m² on regions
• Job failure handling + user messaging

Phase 4 — Reports & notifications (FR-05) (3–4 days)

• In-app notification feed
• Report page (browser)
• PDF export
• Email with report deep link

Phase 5 — Interactive viewer (FR-07) (2–3 days)

• Locate button + click-to-pan/zoom
• Side-by-side T1/T2/overlay modes
• Region highlight sync (extend existing hover logic)

Phase 6 — Review & export (FR-08) (4–5 days)

• Per-region review state
• CSV/XLSX/PDF export of confirmed changes
• Department API adapter (config-driven)
• False-positive archive for training

Phase 7 — Hardening & UAT (3–5 days)

• Load test with sample GeoTIFFs from DDA
• Alignment QA on drone vs satellite pairs
• Security review (auth, upload limits, path traversal)
• Deploy checklist for satdetect-dev

Total estimate: ~25–35 working days (1 senior dev), assuming DDA provides sample GeoTIFFs and hierarchy taxonomy early.

---

10. Hugging Face dev constraints & mitigations

Constraint	Impact	Mitigation
~60s HTTP timeout (cpu-basic)	Sync detect fails on large TIFF	Async jobs + immediate 202 response; detection on downsampled preview
Ephemeral disk	Library loss on rebuild	HF persistent storage volume; document backup to S3 optional
No GPU	Slow DL	Keep AdaptFormer on CPU tiles; cap image size; queue one job at a time
50 GB space limit	Large GeoTIFF library	Per-file quota; admin purge; external blob store Phase 2
Single worker	No true queue	SQLite job lock; one running job at a time

Dockerfile (dev): Switch base to osgeo/gdal:ubuntu-small or install libgdal in existing image.

---

11. File / module layout (new code)

app/
├── dda/
│   ├── __init__.py
│   ├── config.py              # APP_MODE, limits, DEPT_API_*
│   ├── models_library.py      # ImageAsset, hierarchy ORMs
│   ├── models_jobs.py         # DetectionJob, RegionReview, ExportBatch
│   ├── geotiff_io.py          # rasterio ingest, reproject, preview
│   ├── geo_regions.py         # pixel→lat/lng, area m²
│   ├── change_type_map.py     # internal → DDA types
│   ├── job_runner.py          # async detection worker
│   ├── library_routes.py      # FR-01, FR-02 API
│   ├── jobs_routes.py         # FR-04 API
│   ├── reports_routes.py      # FR-05, PDF
│   ├── review_routes.py       # FR-08
│   └── dept_export.py         # adapter pattern
├── static/js/dda/
│   ├── library.js
│   ├── comparison.js
│   ├── jobs.js
│   └── review.js
└── templates/
    ├── index.html             # legacy (production)
    └── index_dda.html         # dev SPA shell

Wire routers in main.py:

if os.getenv("APP_MODE") == "dda":
    app.include_router(library_routes.router, prefix="/api/dda")
    ...

Set on satdetect-dev Space: APP_MODE=dda.
Leave satdetect unset → legacy behavior.

---

12. Testing strategy

Layer	Tests
GeoTIFF ingest	Unit: georef / no-georef / 1-band / 4-band samples
Pair validation	CRS mismatch, non-overlap
Job lifecycle	queued → running → completed/failed
lat/lng accuracy	Known corner coords → centroid within tolerance
DDA type mapping	Snapshot tests on classified regions
RBAC	Role forbidden on upload/delete
PDF	Report renders all mandatory fields
E2E	Upload 2 TIFFs → job → report → confirm → CSV export

Use scripts/validate_detection.py extended with GeoTIFF fixtures.

---

13. Risks & dependencies

Risk	Owner	Mitigation
DDA GeoTIFF quality / GSD	DDA (per SOW)	Pre-upload validation messages
No departmental API spec	DDA IT	File export fallback (FR-08)
Area taxonomy not finalized	DDA	Configurable hierarchy admin UI
Model accuracy on drone imagery	Shared	Alignment warnings; false-positive review loop
HF CPU timeout	Engineering	Async jobs + downsample policy

Blocked until DDA provides:

1. Sample GeoTIFF pairs (georeferenced + one without georef)
2. Zone/Village/Area master list (or confirm Delhi list)
3. Department API spec OR confirm export-only
4. Grid definition if grid mode required at launch

---

14. Out of scope (explicit)

• Training/fine-tuning new DL models (use existing AdaptFormer + review feedback archive for Phase 2)
• Full GIS desktop analysis (QGIS replacement)
• Real-time satellite ingestion feeds
• Mobile native apps
• Production deployment of DDA features (until UAT sign-off)

---

15. Immediate next steps

1. Create satdetect-dev Space env: APP_MODE=dda
2. Phase 0 PR to master → git push hf-dev master:main
3. Request from DDA: 3 sample GeoTIFF pairs + hierarchy spreadsheet
4. Implement Phase 1 (library + upload) — highest dependency for all other FRs

---

Appendix A — Legacy endpoint compatibility

Keep existing endpoints working in legacy mode:

Legacy	DDA equivalent
POST /api/detect (multipart files)	POST /api/dda/jobs (library IDs)
GET /api/history	GET /api/dda/reports
Guest user	RBAC users with roles

No breaking changes to production API contract.

---

Appendix B — Sample job response

{
  "jobId": 42,
  "status": "completed",
  "runId": 108,
  "reportUrl": "/api/dda/reports/108",
  "summary": {
    "regionsCount": 7,
    "changePercentage": 2.34,
    "byType": {
      "New Construction": 3,
      "Vegetation Change": 2,
      "Other": 2
    }
  }
}

---

Plan version 1.0 — derived from Scope of Work_Change Detection-For Team.docx (FR-01 through FR-08).