Spaces:
Paused
Paused
File size: 6,186 Bytes
e8829d6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | # PRD: Temporal & Stereo Stability Profiling
**Feature Name:** Temporal & Stereo Stability Profiling
**Package:** `sensorflow/vitis` (`temporal.py`) — API under
`/api/vitis/temporal/*`
**Status:** Implemented on the emulated backend. No FPGA hardware is attached;
the fixed-point optical-flow/stereo path runs on the constraint-faithful CPU
emulator.
## Value Proposition
Frame-level mAP hides the failures that hurt downstream planners: detections
that flicker, boxes that jitter against real motion, tracks that fragment and
swap IDs. This feature builds a MODEL-INDEPENDENT motion baseline — pyramidal
dense Lucas-Kanade optical flow over the scene, an op FPGAs accelerate well —
and scores every perception engine against it. Because ground truth for
motion comes from the flow field rather than from any engine, the metrics
expose instability without circularity, penalize engines that drop tracking
where the flow proves the object stayed observable, and forgive ID switches
that coincide with genuine flow discontinuities. A synthetic stereo pair
derived from the exact scene geometry closes the loop on 3D: block-matching
disparity is converted to metric depth and checked against scene geometry.
Finally, the whole profile runs twice — float32 flow vs fixed-point flow — so
the report certifies whether the accelerated metric itself can be trusted.
## Architecture & Workflow
```
+--------------------------------------------------------------------+
| Cloud evaluation platform |
| bevfusion scene DB + engines engine leaderboards |
| (GT tracks, planted occlusions, (stability score joins |
| perception-v1 / perception-v3) accuracy metrics) |
+-----------+-----------------------------------------^--------------+
| sequences + engine detections | stability report
v |
+-----------+-----------------------------------------+--------------+
| sensorflow/vitis/temporal.py profiler |
| |
| BEV rasters --> dense pyramidal LK flow --> flow-predicted |
| (per backend) object positions |
| |
| scene geometry --> synthetic stereo pair --> SAD disparity |
| --> metric depth |
| |
| +----------------+ +------------------------------+ |
| | reference flow | meta- | vitis_emulated flow | |
| | (float32) | check | (ap_fixed, LUT recip) | |
| +----------------+ <-----> +--------------+---------------+ |
| | << EMULATION |
| +--------------+---------------+ |
| | (future) vitis_hw: xf::cv | |
| | pyrLK + stereoBM on PL/AIE | |
| +------------------------------+ |
+---------------------------------------------------------------------+
| runs/vitis/temporal/*.json
v
dashboard (engine comparison, cohorts, timeline strip)
```
## Key Metrics Tracked
From `POST /api/vitis/temporal/run` (per engine, per backend, with
`cohorts` breakdown over day/clear, night/clear, day/rain, occluded):
- `flicker_rate` — present→absent→present events on flow-continuous tracks,
per opportunity
- `mean_jitter` — |engine displacement − flow-predicted displacement| /
(|flow displacement| + 0.5 m)
- `fragmentation_per_track` — distinct engine track IDs per GT instance over
flow-continuous spans, minus one
- `id_switches`, `id_switches_at_flow_break`,
`unexcused_id_switch_fraction` — ID switches not explained by a flow
discontinuity
- `stability_score` — 100 × (0.35·(1−4·flicker) + 0.25·e^(−1.5·jitter) +
0.25·(1−0.6·fragmentation) + 0.15·(1−unexcused))
- `stereo.median_abs_disparity_error_px`, `stereo.median_rel_depth_error`,
`stereo.median_abs_depth_error_near_m` (near field < 30 m)
- `backend_agreement.ranking_agrees`,
`backend_agreement.stability_score_delta_by_engine`,
`backend_agreement.max_abs_score_delta` — the trust meta-check
- `timeline_sample` — per-frame detected/flow-continuous/occluded strip for
the most gap-prone track (drives the dashboard timeline)
## Rollout
1. **Emulated (now):** rank engines (e.g. perception-v1-camera vs
perception-v3-bevfusion) by stability; wire the score next to accuracy
metrics in engine comparisons.
2. **HIL bench:** move flow + stereo to xf::cv pyrLK / stereoBM kernels on
real PL/AIE via the `vitis_hw` backend; confirm `backend_agreement`
holds on silicon (the emulator predicts it should — verify).
3. **Fleet:** run the profiler over recorded fleet sequences as a nightly
job; alert when an engine's stability score drops or when the accelerated
and reference metrics diverge (hardware fault canary).
## Risks
- **Flow quality bounds metric quality.** Where flow fails (textureless
regions, huge displacements), continuity is under-detected and instability
is under-penalized. The residual gate (`FLOW_RESIDUAL_OK_M`) trades recall
of opportunities for precision of blame.
- **Schematic scenes flatter flow.** High-contrast BEV rasters are easy for
LK; real imagery will lower flow coverage. Rollout stage 3 must re-tune
the residual gate on real sequences.
- **Stereo far-field error.** Depth error grows quadratically with range at
fixed disparity error; the near-field figure is the meaningful one, and
the report says so explicitly.
- **Score weights are opinionated.** The 0.35/0.25/0.25/0.15 weighting is a
product decision, documented in the formula above and trivially tunable;
rankings should be checked for weight sensitivity before gating releases.
|