docs/POPULATION_SCALE_ROADMAP.md

# Population-Scale Evaluation — Roadmap

Trustworthy foundation for population-level intelligence on Sensorflow Studio.  
**Rule:** extend MegaEval + LabelEval; no parallel stores, auth, or browser ML.

Canonical code: `hf-space/`. Shared perception modules also live under root `sensorflow/`.

---

## Non-goals (all phases)

- Rebuild from scratch or replace MegaEval cube/runs
- New microservice fleet or second database
- Second auth system
- Sync browser-side ML inference
- Hard-coded safety thresholds (use config / policy JSON)
- Silent overwrite of evaluation / regression history
- Full ODD combinatorics, VLM mining, or distributed executors before their phase

---

## Phase 1 — Foundation (this session)

**Extend:**
- `sensorflow/evaluation/records.py` concepts (do not fork store)
- `sensorflow/megaeval/{runs,analysis,api,cube}.py`
- `quality_gate.py`, `launch_gate_evaluator.py`
- React MegaEval / Evaluation / Quality pages

**Create:**
| Artifact | Path |
|----------|------|
| Inventory | `docs/PLATFORM_INVENTORY.md` |
| This roadmap | `docs/POPULATION_SCALE_ROADMAP.md` |
| Platform package | `hf-space/sensorflow/platform/` |
| UI hooks | `hf-space/src/components/platform/`, service `platform.ts` |
| Tests | `hf-space/tests/test_platform/` |

**Deliverables:**
1. Schema entities with versioning/provenance
2. `AggregateLevel` evaluation abstraction (frame→…→population)
3. Metric engine: container PRF/IoU + verification rates
4. Container quality profile API
5. Model compare API (A/B/C) wrapping MegaEval compare
6. Routes: `/api/evaluations`, `/api/containers`, `/api/models/compare`, `/api/gates`
7. UI: container quality table, model compare panel, multi-gate skeleton
8. Evidence package JSON export stub
9. Tests run green

**Hooks left for later (TODO markers):** multi-grader vectors, embedding/VLM mining, SSAM conflict engine, ODD coverage, distributed reduce.

---

## Phase 2 — Multi-grader consensus & label trust

**Extend:** `evaluation/graders.py`, `triage.py`, `reporting.py`  
**Add:** consensus vectors, disputed/auto-accept rates into container profiles  
**UI:** GraderDisagreement + Command Center quality tab  
**Do not:** replace triage policy store

---

## Phase 3 — Rare-event & embedding mining

**Extend:** `evaluation/rare_events.py`, MegaEval `embeddings.npz` / `sketches.py`  
**Add:** VLM/embedding similarity search APIs (async jobs)  
**UI:** RareEventDashboard + InvestigationTab  
**Do not:** sync browser embeddings

---

## Phase 4 — Safety / SSAM conflict engine

**Extend:** SSAM routes in `app_backend.py`, `SSAMSafetyDashboard`  
**Add:** DRAC / DeltaS conflict entities (`Conflict`), EvidencePackage links  
**Config-driven thresholds only**  
**Do not:** hard-code severity cutoffs in UI

---

## Phase 5 — ODD combinatorial coverage

**Extend:** `ODDDefinition` / `ODDObservation` stubs from Phase 1  
**Add:** coverage matrix over dims (weather×lighting×road×scenario…) via MegaEval cube  
**UI:** Coverage gate fills multi-gate skeleton  
**Do not:** new OLAP store — use cube queries

---

## Phase 6 — Release / regression / HITL orchestration

**Extend:** `evaluation/regression.py`, review queue, launch gate  
**Wire:** Regression + Safety + Release gates to real metrics  
**Preserve:** full regression history (append-only)

---

## Phase 7 — Distributed scale

**Replace stand-ins** documented in `hf-space/ARCHITECTURE.md`:
| Today | Target |
|-------|--------|
| npz partitions | Parquet / Iceberg |
| ThreadPoolExecutor | Spark / Flink / Ray |
| QueryRouter cache | Pinot / Druid / ClickHouse |
| JSON EvalStore | Keep for control-plane metadata; lakehouse for facts |

Keep API contracts from Phases 1–6 stable.

---

## Aggregate ladder (canonical)

```
frame → clip/sequence → scene → drive → container → dataset/population → cohort → fleet/population aggregate
```

Implemented in Phase 1 as `AggregateLevel` + `EvaluationScope`; MegaEval already materializes population/cohort/container; LabelEval owns frame/annotation.

---

## Observability checklist (carry forward)

- Every query: `source`, `cache_hit`, `latency_ms` (MegaEval pattern)
- Every gate: config path + threshold values in result
- Every evidence package: model/dataset versions + compute usage + provenance
- No silent history overwrite