Spaces:
Paused
Paused
File size: 5,862 Bytes
e8829d6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 | # PRD: Accelerated ISP + Synthetic Edge-Case Generation
**Feature Name:** Accelerated ISP Preprocessing & Synthetic Edge-Case Generation
**Package:** `sensorflow/vitis` (`isp.py`, `augment.py`) — API under
`/api/vitis/isp/*` and `/api/vitis/augment/*`
**Status:** Implemented on the emulated backend. No FPGA hardware is attached;
all FPGA throughput figures are analytically modeled and labeled
`modeled_not_measured: true`.
## Value Proposition
Two problems, one accelerator. First, ISP pipelines (bad-pixel correction →
demosaic → HDR tone-map → gain → denoise → resize) are the earliest place a
hardware port silently degrades a perception stack — this feature runs the
composable ISP on the reference and Vitis-constrained paths simultaneously
and reports PSNR/SSIM *per stage*, so the exact stage where fixed-point or
streaming constraints bite is visible, alongside a measured-CPU vs
modeled-FPGA throughput comparison that quantifies the acceleration case.
Second, the same accelerated op library mints rare-edge-case evaluation data
at scale: parameterized sensor-noise/low-light/HDR/distortion/blur/glare
augmentations generate stress variants of evaluation frames with complete
lineage, guarded as evaluation-only so synthetic stress data can never leak
into training.
## Architecture & Workflow
```
+-----------------------------------------------------------------+
| Cloud evaluation platform |
| evaluation/megaeval record store raremine scene bank |
| (eval-set supplements, lineage, (candidate flow for |
| leakage guard destinations) costume-ped variants) |
+--------------------^--------------------------------^-----------+
| eval-only variant records | candidates
| (full lineage, protected) | (try/except)
+--------------------+--------------------------------+-----------+
| sensorflow/vitis: ISP + augment |
| |
| RAW Bayer -> bad_pixel -> demosaic -> tone_map -> gain |
| -> denoise -> resize (per-stage PSNR/SSIM) |
| |
| +----------------+ +-------------------------------+ |
| | reference CPU | vs | vitis_emulated (ap_fixed<W,I>,| |
| | float32 | | XFCVDEPTH, LUT div/sqrt, | |
| +----------------+ | modeled pixels/cycle @ MHz) | |
| +---------------+---------------+ |
| | << EMULATION |
| +---------------+---------------+ |
| | (future) vitis_hw: xf::cv ISP | |
| | kernels on real PL, measured | |
| | throughput via XRT profiling | |
| +-------------------------------+ |
+------------------------------------------------------------------+
| runs/vitis/{isp,augment}/*.json
v
dashboard (stage chain, throughput, gallery)
```
## Key Metrics Tracked
From `POST /api/vitis/isp/run`:
- `stage_report[].psnr_db`, `stage_report[].ssim` — emulated vs reference,
per stage
- `stage_report[].measured_cpu_ms`, `stage_report[].modeled_fpga_ms`,
`stage_report[].modeled_speedup_x`, `stage_report[].modeled_placement`
(PL/AIE)
- `throughput.measured_cpu_fps`, `throughput.modeled_fpga_fps_pipelined`
(dataflow overlap: 1 / max stage latency),
`throughput.modeled_speedup_x_serial` — all modeled fields flagged
`modeled_not_measured`
- `summary.min_stage_psnr_db` — the weakest stage at the configured bit-width
From `POST /api/vitis/augment/generate` and `GET /api/vitis/augment/variants`:
- per-variant `lineage` (recipe with resolved params, seed, source frame,
backend config), `evaluation_only: true`, `training_eligible: false`,
`recommended_dataset_destination: REGRESSION_EVALUATION_SET`
- `raremine_hook.routed_candidates` (or the documented fallback note when
`sensorflow.raremine` is not importable)
## Rollout
1. **Emulated (now):** tune stage order, bit-widths, and augmentation
recipes against per-stage PSNR; mint evaluation-only stress sets.
2. **HIL bench:** synthesize the ISP as an xf::cv dataflow pipeline, replace
modeled latencies with XRT-profiled measurements, and validate that
per-stage PSNR on silicon matches the emulator within its documented
error bounds.
3. **Fleet:** run augmentation generation as a batch service against real
fleet evaluation frames; generated variants flow into the protected
evaluation destinations with the same lineage records.
## Risks
- **Modeled throughput is not a benchmark.** pixels/cycle × clock is an
upper bound that ignores memory bandwidth, AXI stalls, and resource
contention. Every figure carries `modeled_not_measured`; stage-2 exists
precisely to replace them.
- **Leakage.** Synthetic stress variants in training would corrupt the
regression signal the variants exist to protect. Records default to
`training_eligible: false` and a protected destination; consumers must
respect the flag (enforced convention, mirrored from raremine's
`PROTECTED_EVAL_DESTINATIONS`).
- **Schematic realism.** Variants derive from schematic BEV renders, not
camera frames. The recipes and lineage plumbing are production-shaped; the
pixels are placeholders until real frames are wired in.
- **In-progress raremine API.** The candidate-routing hook degrades to a
no-op with a note if the raremine package changes shape mid-flight.
|