ARCHITECTURE.md

Sensorflow Studio — Aggregate-First Evaluation Architecture (megaeval)

The sensorflow/megaeval/ layer replaces the "evaluation-record browser" model (annotation as the primary unit) with an aggregate-first, risk-first, cohort-first, drill-down-on-demand design.

Mental model → code

Concept	Meaning	Module / storage
Dataset	immutable source population	population.py → runs/megaeval/populations/{pop}/part-*.npz
Evaluation Run	model+config being evaluated (async job, first-class)	runs.py → runs/megaeval/runs/{run}/run.json
Population	all objects evaluated in a run	headline row of the metric cube
Cohort	slice by class/weather/lighting/road/scenario/sensor/distance/speed/occlusion	cube cells + POST /api/evaluations/query
Container	scene/frame/segment holding related objects	containers.npz aggregate table
Annotation	individual object — forensic view only	per-object objects-part-*.npz, loaded one partition at a time

Data flow

generate population (vectorized numpy, partitioned npz)
        │
create run (queued) ──► worker pool (4 threads) processes partitions:
        │                 model simulation → PARTIAL sufficient statistics
        │                 (n, tp, fp, fn, sum_iou, conf sums, histograms …)
        │                 grouped by the 9 cube dimensions, per partition
        ▼
reduce (concat + groupby-sum) ──► materialize METRIC CUBE (cube.npz)
        │                          + error index, container table, sketches,
        │                          container embeddings
        ▼
publish ──► queries served by QueryRouter: cache → cube → (rare) record scan

Aggregation never scans raw records at query time; queries like "pedestrian recall at night in rain for model v42" are cube lookups.

Single-node stand-ins for distributed infrastructure

This repo runs on one laptop; the seams are shaped so each piece maps to the production-scale equivalent:

Here (single node)	Production analogue
npz column partitions (part-0000.npz…, hash-partitioned by container)	Parquet files in an Iceberg/Delta table, partitioned by scene
ThreadPoolExecutor workers over partitions	Spark executors / Flink task slots
partial stats per partition + reduce_partials	map-side combine + shuffle reduce
QueryRouter (cache → cube → scan) with latency+source reporting	OLAP engine (Druid/Pinot/ClickHouse) over the lakehouse
QueryCache keyed by hash(dataset+model+filters+group_by+metrics)	result cache / materialized views
in-process SSE progress events	job-status topic (Kafka) + progress service
numpy cosine similarity over container embeddings	vector DB (hybrid retrieval)

Honesty rules encoded in the API

• Cube counts and derived ratios are exact (meta.exact = true).
• Percentiles/distributions come from fixed-bin quantile histograms; cardinalities from HyperLogLog — always labeled approximate.
• Precision/recall from human review are sampling estimates with 95% CIs (stratified risk-weighted design: Wilson per stratum, weighted normal-approx combination), reported with n reviewed / frame size / method.
• Every query response reports source (cache/cube/scan), cache_hit, and latency_ms so aggregate-first behavior is observable in the UI.
• Evaluation runs are reproducible: lineage (dataset, model+checkpoint, label version, evaluator code version, metric version, thresholds, sampling config, seed, hardware) is stored on the run and derives the deterministic seed.

Performance SLOs (measured locally, 320k-object population, 67k cube cells)

Query	SLO	Measured
aggregate dashboard (group by class)	< 2 s	~75 ms
filtered cohort query	< 3 s	~350 ms first / sub-ms cached
cached query	< 100 ms	~0.1 ms
object investigation (one container, forensic)	< 1 s	~50–200 ms

The legacy sensorflow/evaluation/ layer (label-evaluation pipeline, triage, HITL, training flywheel) remains unchanged; megaeval integrates with its regression tracker and alerting rather than duplicating them.