docs/prd/vitis-temporal-stability.md

PRD: Temporal & Stereo Stability Profiling

Feature Name: Temporal & Stereo Stability Profiling Package: sensorflow/vitis (temporal.py) — API under /api/vitis/temporal/* Status: Implemented on the emulated backend. No FPGA hardware is attached; the fixed-point optical-flow/stereo path runs on the constraint-faithful CPU emulator.

Value Proposition

Frame-level mAP hides the failures that hurt downstream planners: detections that flicker, boxes that jitter against real motion, tracks that fragment and swap IDs. This feature builds a MODEL-INDEPENDENT motion baseline — pyramidal dense Lucas-Kanade optical flow over the scene, an op FPGAs accelerate well — and scores every perception engine against it. Because ground truth for motion comes from the flow field rather than from any engine, the metrics expose instability without circularity, penalize engines that drop tracking where the flow proves the object stayed observable, and forgive ID switches that coincide with genuine flow discontinuities. A synthetic stereo pair derived from the exact scene geometry closes the loop on 3D: block-matching disparity is converted to metric depth and checked against scene geometry. Finally, the whole profile runs twice — float32 flow vs fixed-point flow — so the report certifies whether the accelerated metric itself can be trusted.

Architecture & Workflow

  +--------------------------------------------------------------------+
  |                      Cloud evaluation platform                      |
  |  bevfusion scene DB + engines            engine leaderboards       |
  |  (GT tracks, planted occlusions,         (stability score joins    |
  |   perception-v1 / perception-v3)          accuracy metrics)        |
  +-----------+-----------------------------------------^--------------+
              | sequences + engine detections           | stability report
              v                                         |
  +-----------+-----------------------------------------+--------------+
  |            sensorflow/vitis/temporal.py profiler                    |
  |                                                                     |
  |  BEV rasters --> dense pyramidal LK flow --> flow-predicted         |
  |                  (per backend)               object positions       |
  |                                                                     |
  |  scene geometry --> synthetic stereo pair --> SAD disparity         |
  |                                               --> metric depth      |
  |                                                                     |
  |  +----------------+          +------------------------------+       |
  |  | reference flow |  meta-   | vitis_emulated flow          |       |
  |  | (float32)      |  check   | (ap_fixed, LUT recip)        |       |
  |  +----------------+  <-----> +--------------+---------------+       |
  |                                             |  << EMULATION        |
  |                              +--------------+---------------+       |
  |                              | (future) vitis_hw: xf::cv    |       |
  |                              | pyrLK + stereoBM on PL/AIE   |       |
  |                              +------------------------------+       |
  +---------------------------------------------------------------------+
              | runs/vitis/temporal/*.json
              v
      dashboard (engine comparison, cohorts, timeline strip)

Key Metrics Tracked

From POST /api/vitis/temporal/run (per engine, per backend, with cohorts breakdown over day/clear, night/clear, day/rain, occluded):

• flicker_rate — present→absent→present events on flow-continuous tracks, per opportunity
• mean_jitter — |engine displacement − flow-predicted displacement| / (|flow displacement| + 0.5 m)
• fragmentation_per_track — distinct engine track IDs per GT instance over flow-continuous spans, minus one
• id_switches, id_switches_at_flow_break, unexcused_id_switch_fraction — ID switches not explained by a flow discontinuity
• stability_score — 100 × (0.35·(1−4·flicker) + 0.25·e^(−1.5·jitter) + 0.25·(1−0.6·fragmentation) + 0.15·(1−unexcused))
• stereo.median_abs_disparity_error_px, stereo.median_rel_depth_error, stereo.median_abs_depth_error_near_m (near field < 30 m)
• backend_agreement.ranking_agrees, backend_agreement.stability_score_delta_by_engine, backend_agreement.max_abs_score_delta — the trust meta-check
• timeline_sample — per-frame detected/flow-continuous/occluded strip for the most gap-prone track (drives the dashboard timeline)

Rollout

1. Emulated (now): rank engines (e.g. perception-v1-camera vs perception-v3-bevfusion) by stability; wire the score next to accuracy metrics in engine comparisons.
2. HIL bench: move flow + stereo to xf::cv pyrLK / stereoBM kernels on real PL/AIE via the vitis_hw backend; confirm backend_agreement holds on silicon (the emulator predicts it should — verify).
3. Fleet: run the profiler over recorded fleet sequences as a nightly job; alert when an engine's stability score drops or when the accelerated and reference metrics diverge (hardware fault canary).

Risks

• Flow quality bounds metric quality. Where flow fails (textureless regions, huge displacements), continuity is under-detected and instability is under-penalized. The residual gate (FLOW_RESIDUAL_OK_M) trades recall of opportunities for precision of blame.
• Schematic scenes flatter flow. High-contrast BEV rasters are easy for LK; real imagery will lower flow coverage. Rollout stage 3 must re-tune the residual gate on real sequences.
• Stereo far-field error. Depth error grows quadratically with range at fixed disparity error; the near-field figure is the meaningful one, and the report says so explicitly.
• Score weights are opinionated. The 0.35/0.25/0.25/0.15 weighting is a product decision, documented in the formula above and trivially tunable; rankings should be checked for weight sensitivity before gating releases.