docs/prd/vitis-isp-preprocessing.md

PRD: Accelerated ISP + Synthetic Edge-Case Generation

Feature Name: Accelerated ISP Preprocessing & Synthetic Edge-Case Generation Package: sensorflow/vitis (isp.py, augment.py) — API under /api/vitis/isp/* and /api/vitis/augment/* Status: Implemented on the emulated backend. No FPGA hardware is attached; all FPGA throughput figures are analytically modeled and labeled modeled_not_measured: true.

Value Proposition

Two problems, one accelerator. First, ISP pipelines (bad-pixel correction → demosaic → HDR tone-map → gain → denoise → resize) are the earliest place a hardware port silently degrades a perception stack — this feature runs the composable ISP on the reference and Vitis-constrained paths simultaneously and reports PSNR/SSIM per stage, so the exact stage where fixed-point or streaming constraints bite is visible, alongside a measured-CPU vs modeled-FPGA throughput comparison that quantifies the acceleration case. Second, the same accelerated op library mints rare-edge-case evaluation data at scale: parameterized sensor-noise/low-light/HDR/distortion/blur/glare augmentations generate stress variants of evaluation frames with complete lineage, guarded as evaluation-only so synthetic stress data can never leak into training.

Architecture & Workflow

   +-----------------------------------------------------------------+
   |                     Cloud evaluation platform                    |
   |   evaluation/megaeval record store        raremine scene bank    |
   |   (eval-set supplements, lineage,         (candidate flow for    |
   |    leakage guard destinations)             costume-ped variants) |
   +--------------------^--------------------------------^-----------+
                        | eval-only variant records      | candidates
                        |  (full lineage, protected)     | (try/except)
   +--------------------+--------------------------------+-----------+
   |               sensorflow/vitis: ISP + augment                    |
   |                                                                  |
   |  RAW Bayer -> bad_pixel -> demosaic -> tone_map -> gain          |
   |            -> denoise -> resize        (per-stage PSNR/SSIM)     |
   |                                                                  |
   |  +----------------+        +-------------------------------+     |
   |  | reference CPU  |  vs    | vitis_emulated (ap_fixed<W,I>,|     |
   |  | float32        |        | XFCVDEPTH, LUT div/sqrt,      |     |
   |  +----------------+        | modeled pixels/cycle @ MHz)   |     |
   |                            +---------------+---------------+     |
   |                                            |  << EMULATION      |
   |                            +---------------+---------------+     |
   |                            | (future) vitis_hw: xf::cv ISP |     |
   |                            | kernels on real PL, measured  |     |
   |                            | throughput via XRT profiling  |     |
   |                            +-------------------------------+     |
   +------------------------------------------------------------------+
                        | runs/vitis/{isp,augment}/*.json
                        v
                 dashboard (stage chain, throughput, gallery)

Key Metrics Tracked

From POST /api/vitis/isp/run:

• stage_report[].psnr_db, stage_report[].ssim — emulated vs reference, per stage
• stage_report[].measured_cpu_ms, stage_report[].modeled_fpga_ms, stage_report[].modeled_speedup_x, stage_report[].modeled_placement (PL/AIE)
• throughput.measured_cpu_fps, throughput.modeled_fpga_fps_pipelined (dataflow overlap: 1 / max stage latency), throughput.modeled_speedup_x_serial — all modeled fields flagged modeled_not_measured
• summary.min_stage_psnr_db — the weakest stage at the configured bit-width

From POST /api/vitis/augment/generate and GET /api/vitis/augment/variants:

• per-variant lineage (recipe with resolved params, seed, source frame, backend config), evaluation_only: true, training_eligible: false, recommended_dataset_destination: REGRESSION_EVALUATION_SET
• raremine_hook.routed_candidates (or the documented fallback note when sensorflow.raremine is not importable)

Rollout

1. Emulated (now): tune stage order, bit-widths, and augmentation recipes against per-stage PSNR; mint evaluation-only stress sets.
2. HIL bench: synthesize the ISP as an xf::cv dataflow pipeline, replace modeled latencies with XRT-profiled measurements, and validate that per-stage PSNR on silicon matches the emulator within its documented error bounds.
3. Fleet: run augmentation generation as a batch service against real fleet evaluation frames; generated variants flow into the protected evaluation destinations with the same lineage records.

Risks

• Modeled throughput is not a benchmark. pixels/cycle × clock is an upper bound that ignores memory bandwidth, AXI stalls, and resource contention. Every figure carries modeled_not_measured; stage-2 exists precisely to replace them.
• Leakage. Synthetic stress variants in training would corrupt the regression signal the variants exist to protect. Records default to training_eligible: false and a protected destination; consumers must respect the flag (enforced convention, mirrored from raremine's PROTECTED_EVAL_DESTINATIONS).
• Schematic realism. Variants derive from schematic BEV renders, not camera frames. The recipes and lineage plumbing are production-shaped; the pixels are placeholders until real frames are wired in.
• In-progress raremine API. The candidate-routing hook degrades to a no-op with a note if the raremine package changes shape mid-flight.