| --- |
| license: other |
| tags: |
| - heal |
| - horizon |
| --- |
| |
| # E2E Diffusion Planner + HENet-tiny |
|
|
| An end-to-end driving model for camera-based perception and short-horizon ego trajectory planning on NAVSIM. |
|
|
| E2E Diffusion Planner + HENet-tiny combines an 8-camera BEV perception front-end with a diffusion planner conditioned on BEV features, ego state, and route. Perception and planning are trained and quantized as one graph for Horizon J6M / J6P deployment. |
|
|
| --- |
|
|
| ## Deployment Metrics |
|
|
| ### Model Parameters |
|
|
| | Model | Model Input | Backbone | Neck | Model Output | |
| |---|---|---|---|---| |
| | E2EPlanner (BEVFusionMultitask + Diffusion_Planner) | 8-camera multi-view images `(B,8,3,512,960)` + ego state + route lanes | HENet-tiny | FPN + SingleBevFormerViewTransformer | future ego trajectory `(1,8,4)` via diffusion | |
| |
| ### Accuracy Metrics |
| |
| | March | Metric | float | calibration | hbm | |
| | --- | --- | --- | --- | --- | |
| | J6M | PDMS | 0.8224 | 0.8139 | 0.8162 | |
| |
| > Measured on NAVSIM navtest with `HAT_USE_NEIGHBORS_FUTURE=0` and `dequantize_bev_embed=False`. No QAT stage. |
| > |
| > HEAL version: heal 0.6.0 / hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5 (board UCP 3.16.2). |
|
|
| ### Performance Metrics |
|
|
| > **Performance benchmark**: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage. |
| > Data source: `tools/perf.py` (`--perf-time 1`; DDR sampled by `hrt_ucp_monitor -d 200`). Measured 2026-10-19 (`dequantize_bev_embed=False`). Toolchain: hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5. No `native::` CPU ops. |
|
|
| | March | latency (ms) | fps | Memory Usage (MB) | |
| |---|---|---|---| |
| | J6M | 104.66 (BPU 104.56) | 9.75 | 376.90 | |
| | J6P | 53.25 (BPU 53.19) | 66.26 | 353.70 | |
|
|
| --- |
|
|
| ## Model Overview |
|
|
| ### Core Design |
|
|
| Think of two stages glued on one BEV map. Cameras are encoded by HENet-tiny and lifted with a single-frame BevFormer into a forward-looking BEV grid; the same feature feeds detection, occupancy, and map heads for a consistent scene. A compact diffusion planner then reads that BEV, listens to the route, and denoises a few future ego waypoints—so planning stays conditioned on what perception actually sees, without a separate hand-off stack. |
|
|
| - **Task type**: End-to-end planning (perception + planning joint). |
| - **Perception backbone**: HENet-tiny (`block_nums=[4,3,8,6]`, `embed_dims=[64,128,192,384]`), initialized from the ImageNet-pretrained `henet_tinym` float checkpoint. |
| - **Perception neck**: `FPN` + `SingleBevFormerViewTransformer` (`128×192` BEV, `[-51.2m, 102.4m] × [-51.2m, 51.2m]`). |
| - **Dynamic det head**: `DenseBEVSparseHead` (3 classes; `num_anchor=384`, `MemoryBank` 128 temporal instances). |
| - **Static det head**: `DenseBEVSparseHead` (4 classes; `num_anchor=200`). |
| - **Occupancy head**: `BevformerOccDetDecoder` + `BEVOCCHead2D` (`Dz=16`, 9 classes). |
| - **Discrete map head**: `BEVFormerDet2DDecoder` (2 classes: stop_line/crosswalk). |
| - **Online map head**: `MaptrDetDecoder` + `MapInstanceDetectorHead` (1 class `lanes`, SD-map fusion, 20 points per polyline). |
| - **Planner encoder**: `Diffusion_Planner_Encoder` (`BEVFusionEncoder` ×8 downsample → 384 tokens, 3-layer `FusionEncoder` transformer). |
| - **Planner decoder**: 3-layer `DiT` (`hidden_dim=256`, 8 heads, `RouteEncoder`), `model_type="x_start"`, `diffusion_pred_steps=5`, linear VP noise schedule. |
| - **Loss functions**: perception — `FocalLoss` + `L1Loss` + `CrossEntropyLoss` + `GaussianFocalLoss` / `CrossEntropyLoss` / `MapTRCriterion` / `BevFormer2DCriterion` per head; planner — `DiffusionLoss` (ego 2.0 + collision 1.0). |
| - **Model input**: 8-camera multi-view images `(B,8,3,512,960)` (YUV444 + `Normalize(128,128)`; camera-only) + ego state + route lanes. |
| - **Model output**: 8-step future ego trajectory `(1,8,4)`. |
|
|
| ### Official Repo and Paper |
|
|
| 1. Horizon reference algorithm blog (Diffusion-Planner_v3.0): https://developer.horizon.auto/blog/14190 |
| 2. Diffusion-Planner: https://github.com/ZhengYinan-AIR/Diffusion-Planner (paper: https://arxiv.org/abs/2501.15564) |
| 3. NAVSIM benchmark: https://github.com/autonomousvision/navsim (paper: https://arxiv.org/abs/2406.15349) |
| 4. SparseBEV-style det head repo: https://github.com/MCG-NJU/SparseBEV (paper: https://arxiv.org/abs/2308.09244) |
| 5. MapTR repo: https://github.com/hustvl/MapTR (paper: https://arxiv.org/abs/2208.14437) |
| |
| Note: The camera backbone HENet and the E2E assembly are HEAL in-house implementations; the official repos use different backbones and pipelines. |
| |
