File size: 4,488 Bytes
64009a6 0153ce4 64009a6 cd2b21d 64009a6 cd2b21d 64009a6 0153ce4 64009a6 0153ce4 64009a6 8c2d5d7 64009a6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | ---
license: other
tags:
- heal
- horizon
---
# E2E Diffusion Planner + HENet-tiny
An end-to-end driving model for camera-based perception and short-horizon ego trajectory planning on NAVSIM.
E2E Diffusion Planner + HENet-tiny combines an 8-camera BEV perception front-end with a diffusion planner conditioned on BEV features, ego state, and route. Perception and planning are trained and quantized as one graph for Horizon J6M / J6P deployment.
---
## Deployment Metrics
### Model Parameters
| Model | Model Input | Backbone | Neck | Model Output |
|---|---|---|---|---|
| E2EPlanner (BEVFusionMultitask + Diffusion_Planner) | 8-camera multi-view images `(B,8,3,512,960)` + ego state + route lanes | HENet-tiny | FPN + SingleBevFormerViewTransformer | future ego trajectory `(1,8,4)` via diffusion |
### Accuracy Metrics
| March | Metric | float | calibration | hbm |
| --- | --- | --- | --- | --- |
| J6M | PDMS | 0.8224 | 0.8139 | 0.8162 |
> Measured on NAVSIM navtest with `HAT_USE_NEIGHBORS_FUTURE=0` and `dequantize_bev_embed=False`. No QAT stage.
>
> HEAL version: heal 0.6.0 / hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5 (board UCP 3.16.2).
### Performance Metrics
> **Performance benchmark**: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage.
> Data source: `tools/perf.py` (`--perf-time 1`; DDR sampled by `hrt_ucp_monitor -d 200`). Measured 2026-10-19 (`dequantize_bev_embed=False`). Toolchain: hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5. No `native::` CPU ops.
| March | latency (ms) | fps | Memory Usage (MB) |
|---|---|---|---|
| J6M | 104.66 (BPU 104.56) | 9.75 | 376.90 |
| J6P | 53.25 (BPU 53.19) | 66.26 | 353.70 |
---
## Model Overview
### Core Design
Think of two stages glued on one BEV map. Cameras are encoded by HENet-tiny and lifted with a single-frame BevFormer into a forward-looking BEV grid; the same feature feeds detection, occupancy, and map heads for a consistent scene. A compact diffusion planner then reads that BEV, listens to the route, and denoises a few future ego waypoints—so planning stays conditioned on what perception actually sees, without a separate hand-off stack.
- **Task type**: End-to-end planning (perception + planning joint).
- **Perception backbone**: HENet-tiny (`block_nums=[4,3,8,6]`, `embed_dims=[64,128,192,384]`), initialized from the ImageNet-pretrained `henet_tinym` float checkpoint.
- **Perception neck**: `FPN` + `SingleBevFormerViewTransformer` (`128×192` BEV, `[-51.2m, 102.4m] × [-51.2m, 51.2m]`).
- **Dynamic det head**: `DenseBEVSparseHead` (3 classes; `num_anchor=384`, `MemoryBank` 128 temporal instances).
- **Static det head**: `DenseBEVSparseHead` (4 classes; `num_anchor=200`).
- **Occupancy head**: `BevformerOccDetDecoder` + `BEVOCCHead2D` (`Dz=16`, 9 classes).
- **Discrete map head**: `BEVFormerDet2DDecoder` (2 classes: stop_line/crosswalk).
- **Online map head**: `MaptrDetDecoder` + `MapInstanceDetectorHead` (1 class `lanes`, SD-map fusion, 20 points per polyline).
- **Planner encoder**: `Diffusion_Planner_Encoder` (`BEVFusionEncoder` ×8 downsample → 384 tokens, 3-layer `FusionEncoder` transformer).
- **Planner decoder**: 3-layer `DiT` (`hidden_dim=256`, 8 heads, `RouteEncoder`), `model_type="x_start"`, `diffusion_pred_steps=5`, linear VP noise schedule.
- **Loss functions**: perception — `FocalLoss` + `L1Loss` + `CrossEntropyLoss` + `GaussianFocalLoss` / `CrossEntropyLoss` / `MapTRCriterion` / `BevFormer2DCriterion` per head; planner — `DiffusionLoss` (ego 2.0 + collision 1.0).
- **Model input**: 8-camera multi-view images `(B,8,3,512,960)` (YUV444 + `Normalize(128,128)`; camera-only) + ego state + route lanes.
- **Model output**: 8-step future ego trajectory `(1,8,4)`.
### Official Repo and Paper
1. Horizon reference algorithm blog (Diffusion-Planner_v3.0): https://developer.horizon.auto/blog/14190
2. Diffusion-Planner: https://github.com/ZhengYinan-AIR/Diffusion-Planner (paper: https://arxiv.org/abs/2501.15564)
3. NAVSIM benchmark: https://github.com/autonomousvision/navsim (paper: https://arxiv.org/abs/2406.15349)
4. SparseBEV-style det head repo: https://github.com/MCG-NJU/SparseBEV (paper: https://arxiv.org/abs/2308.09244)
5. MapTR repo: https://github.com/hustvl/MapTR (paper: https://arxiv.org/abs/2208.14437)
Note: The camera backbone HENet and the E2E assembly are HEAL in-house implementations; the official repos use different backbones and pipelines.
|