tt-model push bevfusion-p150 (container)

5528e28 verified 20 days ago

16.5 kB

# Python API: BEVFusion-L (Autoware autoware_bevfusion, lidar-only) on Blackhole
Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the
API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are
the same.
## Install
Install the package on top of an environment that already has `ttnn` (a tt-metal `python_env` at `44d66500520`
with `patches/tt-metal-eth-dispatch.patch`, or the tt-model container). From the root of the model repository (the
directory that holds `pyproject.toml`, `README.md` and `code/`):

```bash
pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests
```

The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_bevfusion` (there is no
`pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree).

The package carries `tt_bevfusion.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner,
decoders, sparse rulebooks, model base class, HTTP app), vendored at the version recorded in
`code/tt_bevfusion/ttaw/VENDORED.json`.

| You want to run | Extras |
|---|---|
| the Python API | none |
| the HTTP server (`tt_bevfusion.server.app`, see `SERVING.md`) | `server` |
| host tests (no device; device tests are skipped): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevfusion/tests` | `server,test` |
| device tests: `python -m pytest -q -s code/tt_bevfusion/tests/test_pcc_device.py code/tt_bevfusion/tests/test_e2e_device.py` | `test` |

## Quickstart

```python
from tt_bevfusion import BEVFusion
with BEVFusion.from_pretrained(device_id=0) as model:
    out = model("code/tt_bevfusion/samples/test_pcd.npz")        # Autoware's test.pcd, Apache-2.0
print(out.to_dict())
```

`examples/quickstart.py` runs this snippet on the shipped sample and also writes `quickstart_bev.png` (the points and
the boxes from above, `tt_bevfusion.viz.render_bev`).

## `BEVFusion.from_pretrained(...)`
```python
BEVFusion.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default AutowareFoundation/bevfusion
    *,
    revision=None,          # default for the default repo: the validated commit e1bf164b909 (tag v2.0)
    variant=None,           # load-time variant: "lidar" (the only one in this release); default $BEVFUSION_VARIANT or "lidar"
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_bevfusion.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $BEVFUSION_DISPATCH or "eth"
    num_command_queues=None,  # default $BEVFUSION_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
    **compile_params,       # none for this model: the device shapes are the three capacity buckets below
) -> BEVFusion
```
What it does: resolves the weights first, so a Hub problem never claims the chip (`weights_dir` >
`$BEVFUSION_WEIGHTS_DIR` > a local `model_id` directory > the HF snapshot at the pinned revision, restricted to
`bevfusion_lidar.onnx, ml_package_bevfusion_lidar.param.yaml, detection_class_remapper.param.yaml, deploy_metadata.yaml`
(34 MB), with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ; the other open parameters
are `DEVICE_DEFAULTS` in `tt_bevfusion/device.py`, overridable with `BEVFUSION_*`), builds the graph, then compiles
and captures the metal traces of every warm-up variant. If ETH dispatch cannot open (tt-metal without the patch), it
warns and falls back to WORKER dispatch.

The camera-lidar profile of the Autoware package (`bevfusion_camera_lidar.onnx` + the Swin-T image backbone) is not
part of this release; it is planned for a later one.

## Warm-up

`from_pretrained` returns a warm model: it builds the graph, runs every trace segment once eagerly (this compiles the
kernels and uploads the weights) and captures **four metal traces**: the sparse encoder once per capacity bucket
(`sparse_s`, `sparse_m`, `sparse_full`) and the dense BEV network (`dense`). Every call then replays the sparse trace
of the frame's bucket and the dense trace; nothing is compiled on a call.

| bucket | active rows per sparse level (l1 / l2 / l3 / l4 / out) | typical frames |
|---|---|---|
| `s` | 32,768 / 40,960 / 24,576 / 8,192 / 8,192 | one sweep of a 64-beam scan (test.pcd: 21,359 voxels), nuScenes 32-beam frames |
| `m` | 131,072 / 147,456 / 81,920 / 32,768 / 24,576 | dense 2-sweep frames (PandaSet: about 95,000 voxels; Autoware's sample rosbag: 98,334) |
| `full` | 262,144 / 262,144 / 131,072 / 65,536 / 65,536 | anything larger, up to Autoware's 256,000-voxel cap |

- The bucket is chosen per frame on the host from the actual level counts (smallest that holds every level). The
  sparse trace's cost follows the bucket more than the frame: test.pcd replays in 3.55 ms in `s`, 7.2 ms in `m`,
  12.5 ms in `full` (every bucket gives the same outputs).
- A frame whose counts exceed `full` is refused before the device runs (`tt_bevfusion.io.InputError`, HTTP 400, the
  message names the levels). Nothing is clipped. None of the 122 measured frames came close (the densest is at about
  37 % of `full`'s level-1 rows), but with the level ratios measured on PandaSet (level 2 can reach 1.43× level 1) a
  frame from about 180,000 voxels on can hit it, below Autoware's 256,000-voxel cap.
- Measured (2026-10-12, ETH dispatch, 1 CQ, 12x10, the shipped sample): with an empty JIT cache the load took
  201.5 s (firmware and every kernel compiled, on a loaded host); with that cache warm, in a new process, 38.0 s (graph
  build 31.6 s, most of it the numba compile of the host stages, which has no on-disk cache; eager warm-up and capture
  of the 4 traces 1.9 s). The traces are captured during the load, so no call compiles anything (first / second call
  on the shipped sample: 40.9 / 33.6 ms and 40.7 / 33.5 ms in two processes; the host stages move with the load of the
  shared host, load average 7-9 during these runs).
- `warmup_variants="none"` (server: `BEVFUSION_WARMUP=none`) skips the capture at load; call `model.warmup()` before
  the first frame (it is idempotent).
- The JIT cache is tt-metal's (`~/.cache/tt-metal-cache`, or `TT_METAL_CACHE`); inside the container it is
  `~/.cache/tt-model/bevfusion-p150/cache` of the host. A process with WORKER dispatch (11x10) compiles its own
  kernels.

## Call: `model(points, **kwargs)`
| Argument | Type | Description |
|---|---|---|
| `points` (first positional) | path, `bytes` + `fmt=`, (N, C) array / tensor, structured array, `PointCloud`, JSON envelope | one LiDAR cloud in `base_link` with x, y, z (see "Input types"); a column named `time_lag` marks Autoware's already densified 2-sweep cloud |
| `sweeps` | list of `{"points", "time_lag_s", "T_current_from_sweep"}` | client-side densification: the past sweeps, moved into the current frame with Autoware's float32 sweep arithmetic |
| `stream` | `{"id", "timestamp_s", "T_world_from_ego", "reset"}` | server-side densification exactly as Autoware's `PointCloudDensification` (`densification_num_past_frames` 1, per stream id, LRU of 16); a frame without its pose is refused and not cached (Autoware's TF failure) |
| `score_threshold` | float in [0, 1], default 0.1 | every class and distance bin (Autoware's `min_confidence_scores`); a decoded score below it is dropped (`<`) |
| `circle_nms_dist_threshold` | float in [0, 10] m, default 0.5 | class-agnostic circle NMS between the score-sorted proposals (<= 0 means 0.5 m, as the node) |
| `iou_nms_threshold` | float in [0, 1], default 0.1 | Autoware's IoU-BEV NMS: suppress when the BEV IoU is above it |
| `iou_nms_search_distance_2d` | float in [0, 200] m, default 10 | the IoU-BEV NMS search radius |
| `remap_classes` | bool, default `True` | Autoware's area class remapper (CAR to TRUCK for 12.1-36 m², CAR / TRUCK / BUS to TRAILER from 36 m²) |
| `max_detections` | int >= 1, default `None` (no cap, as Autoware) | keep the N best after NMS (not an Autoware knob) |
Every knob is host-only (PLAN.md 2.6, RT-host): no device shape changes, and the defaults reproduce Autoware's
`bevfusion_lidar.param.yaml` and the ml_package exactly (the yaw-norm thresholds and the distance bins are not knobs:
the node's values, yaw-norm 0.3 for CAR, TRUCK, BUS, BICYCLE and 0 for PEDESTRIAN; distance bins 50 / 90 / 121 /
200 m). They are validated before the device runs (a bad value raises `tt_bevfusion.io.InputError`, HTTP 400). The
device test `test_stream_and_params` checks `stream=` and non-default values against the CPU reference.
### Input types
- Point clouds: a path (`.bin` raw float32 rows -- pass `fields=` if not `x, y, z, intensity`; `.npy`;
  `.npz`; `.pcd` ascii / binary / binary_compressed), raw bytes with `fmt=`, an (N, C) numpy array or torch tensor,
  a structured array, the JSON envelope of `/predict`, or `tt_bevfusion.io.PointCloud`. Frame: `base_link`, metres.
  An unnamed (N, 4) array is x, y, z, intensity; name the columns (structured array, `PointCloud`, `fields=`) for any
  other layout, e.g. x, y, z, time_lag.
- `intensity` is not a network input (`use_intensity: false` in the ml_package): a column is accepted and ignored.
- Three ways to give Autoware's 2-sweep input: a cloud with a `time_lag` column (already densified, used as is),
  `sweeps=` (densified on the host), or `stream=` (the node's own state machine). A plain cloud is one sweep.
- Rows with a non-finite x / y / z are dropped, and so are rows at exactly (0, 0, 0): the missing returns of organized
  clouds.
- Voxelization: 0.17 × 0.17 × 0.2 m voxels over ±122.4 m (x, y) and -3 to 5 m (z), at most 256,000 voxels
  (Autoware's clip; `meta.voxel_overflow`) and 10 points per voxel: the first 10 in input order (the training-time
  deterministic rule; Autoware keeps whichever 10 its GPU atomics leave).
- Images, calibration and planner inputs are refused (the lidar-only profile).
## Output
`model(...)` returns a `tt_bevfusion.Output` (= `ttaw.outputs.Detections3D`); `out.to_dict()` is exactly the
`POST /predict` body (SERVING.md section 3.2).
| field | type | meaning |
|---|---|---|
| `boxes` | float32 `[N, 7]` | x, y, z (box centre), length, width, height, yaw in `base_link` metres / radians, the convention of Autoware's autoware_bevfusion node (yaw counter-clockwise from +x); sorted by score |
| `scores` | float32 `[N]` | Autoware's `existence_probability` (`query_heat · σ(heatmap)` of the winning class) |
| `label_ids` | int32 `[N]` | `autoware_perception_msgs/ObjectClassification` values (CAR 1, TRUCK 2, BUS 3, TRAILER 4, BICYCLE 6, PEDESTRIAN 7); `labels` holds the names |
| `meta` | dict | `variant`; `points_in`, `points_in_range`, `num_voxels`, `voxels_total`, `voxel_overflow`, `points_dropped_by_cap`, `max_points_in_voxel`; `sparse` (`counts` per level, `pairs` per neighbour map, `bucket`, `overflow`); `skipped` (no voxel: no inference, no objects, as the node); `densify` (mode, points); `counts` (`proposals`, `after_threshold`, `after_circle_nms`, `after_iou_nms`); per detection `orientation_availability`, `label_before_remap` and `proposal` (the top-k slot) |
| `timing_ms` | dict | `preprocess`, `device` (input packing + H2D + both traces + D2H + unpack), `postprocess`, `total` of this call |
`out.to_dicts()` gives `[{"label", "label_id", "score", "center", "size", "yaw"}]`; `len(out)` the number of
detections. No velocity is returned: Autoware decodes the network's velocity rows but does not publish them.
The last step of the head runs on the host in float32 after the one readback: `score = σ(heatmap)[cls] ·
query_heat[cls]`, `label = cls` and `center + bev_pos` (this keeps the BEV cell positions out of bf16). Everything
before it (sparse encoder, SECOND, SECONDFPN, heatmap, local max, top-512, the transformer decoder layer, the box
heads) runs on the device.
## Lifetime and information
- `model.close()` releases the traces and the device tensors and closes the chip if the model opened it; idempotent.
  `with` calls it for you; an unclosed model is closed when Python exits.
- `model.info`: weights (repo, tag, revision, path), device (`dispatch`, `grid`, `cores`, `num_command_queues`,
  `fallback`), variant, warm variants, warm-up times, runtime parameter defaults, the config, the pre-processing and
  the trace descriptions.
- Calls from several threads are safe and pipelined (`BEVFUSION_PIPELINE=2`, the default): host pre-processing, the
  device stage (staging, H2D, replays, D2H) and host post-processing each run under their own lock, so up to three
  frames overlap while the chip runs one at a time; every result equals a serial call's. One model per process per chip.
- `tt_bevfusion.viz.render_bev(points, out.to_dicts(), reference=...)` draws a bird's-eye view (PIL, no OpenCV).
## Speed
Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, the shipped precision (`code/scripts/bench.py`, 20 iterations, two
runs; the numbers of `OPT_BASELINE.md` "After optimization" and `VERIFICATION_OPT_2026-10-12.md`, 2026-10-12, on a
shared host whose load moves the host stages, load average 4-5):
| stage, p50 | test.pcd (shipped, bucket `s`) | PandaSet 019 f40 (not shipped, bucket `m`) |
|---|---:|---:|
| `model(path)` end to end | 32.5-32.6 ms | 76.4-77.5 ms |
| host pre-processing (decode, densify, voxelize, VFE, the sorted level sets) | 14.2 ms | 48.5-49.0 ms |
| input packing (staged VFE rows, level keys) + H2D | 0.64 + 0.51 ms | 2.0-2.1 + 1.01 ms |
| device traces, one frame (sparse, incl. the device rulebooks, + dense) | 3.6 + 12.5 ms | 9.3 + 12.5 ms |
| D2H + unpack | 0.55 + 0.21 ms | 0.61 + 0.25 ms |
| host post-processing (decode, NMS, remap) | 0.64 ms | 1.08-1.11 ms |
- Back-to-back frames take 15.99 ms (test.pcd) and 21.72 ms (PandaSet) of device time: 62.5 and 46.0 frames/s. A
  synchronous call gives 30.7 and 13.0 calls/s; three concurrent callers get a frame every 18.8-18.9 ms (test.pcd) and
  32.2 ms (PandaSet), 53 and 31 frames/s.
- The first release (2026-10-09) took 657.1 / 2,786.0 ms per call (86-88 % of it a numpy rulebook build, now on the
  device) and 53.09 / 112.02 ms of device time per frame. The optimization steps are in `OPT_REPORT.md`.
## Tests
From the repository root, with `code/` on `PYTHONPATH` (or the package installed):
- Host (no device, no ttnn call): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevfusion/tests`. Inside the
  tt-models workspace the suite runs on its fake ttnn (`-p fake_ttnn_plugin`), which also enables `test_tt_host.py`
  (the TT graph emulated on the CPU against the fp32 reference); tests that need workspace-only oracles or data skip
  elsewhere, and a missing sample FAILS inside the workspace.
- Device (one p150): `python -m pytest -q -s code/tt_bevfusion/tests/test_pcc_device.py
  code/tt_bevfusion/tests/test_e2e_device.py` (per-module PCC and the top-500 selection against the fp32 CPU
  reference on replay outputs, end-to-end agreement on the PandaSet sample, 7 development frames and 10 held-out
  frames, `stream=` and non-default parameters, replay == eager, API == `/predict`). The public-frame gates run only
  inside the tt-models workspace, where their data lives: the frames are PandaSet and nuScenes sweeps and Autoware's
  sample rosbag, which this repository does not ship.
- The gates are frozen in `tests/*.gates.json` at the first green run and are never loosened.
## Limits
- Batch 1 on the chip; one model per process; at most 256,000 voxels (Autoware's clip) and 10 points per voxel (the
  first 10 in input order); the sparse encoder's three capacity buckets (a frame beyond `full` is refused, see
  "Warm-up").
- Outputs differ slightly from the fp32 reference: an object near one of Autoware's decision thresholds (score,
  yaw-norm gate, circle NMS, remap area) can appear, disappear or change label (README "Caveats" has the measured
  rates).
- One p150, ETH dispatch (12x10); `dispatch="worker"` is an A/B opt-in (11x10). Not a ROS 2 node. Lidar-only: the
  camera-lidar profile is not in this release.

	# Python API: BEVFusion-L (Autoware autoware_bevfusion, lidar-only) on Blackhole

	Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the
	API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are
	the same.

	## Install

	Install the package on top of an environment that already has `ttnn` (a tt-metal `python_env` at `44d66500520`
	with `patches/tt-metal-eth-dispatch.patch`, or the tt-model container). From the root of the model repository (the
	directory that holds `pyproject.toml`, `README.md` and `code/`):

	```bash
	pip install -e . # the Python API
	pip install -e ".[server,test]" # + the HTTP server and the tests
	```

	The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_bevfusion` (there is no
	`pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree).

	The package carries `tt_bevfusion.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner,
	decoders, sparse rulebooks, model base class, HTTP app), vendored at the version recorded in
	`code/tt_bevfusion/ttaw/VENDORED.json`.

	\| You want to run \| Extras \|
	\|---\|---\|
	\| the Python API \| none \|
	\| the HTTP server (`tt_bevfusion.server.app`, see `SERVING.md`) \| `server` \|
	\| host tests (no device; device tests are skipped): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevfusion/tests` \| `server,test` \|
	\| device tests: `python -m pytest -q -s code/tt_bevfusion/tests/test_pcc_device.py code/tt_bevfusion/tests/test_e2e_device.py` \| `test` \|

	## Quickstart

	```python
	from tt_bevfusion import BEVFusion

	with BEVFusion.from_pretrained(device_id=0) as model:
	out = model("code/tt_bevfusion/samples/test_pcd.npz") # Autoware's test.pcd, Apache-2.0
	print(out.to_dict())
	```

	`examples/quickstart.py` runs this snippet on the shipped sample and also writes `quickstart_bev.png` (the points and
	the boxes from above, `tt_bevfusion.viz.render_bev`).

	## `BEVFusion.from_pretrained(...)`

	```python
	BEVFusion.from_pretrained(
	model_id=None, # HF repo or a local directory with the weights files; default AutowareFoundation/bevfusion
	*,
	revision=None, # default for the default repo: the validated commit e1bf164b909 (tag v2.0)
	variant=None, # load-time variant: "lidar" (the only one in this release); default $BEVFUSION_VARIANT or "lidar"
	device_id=None, # chip to open; default $TT_DEVICE_ID or 0
	device=None, # an already-opened ttnn device (tt_bevfusion.device.open_device); close() does not close it
	dispatch=None, # "eth" (p150 target, 12x10 grid) \| "worker" (A/B only, 11x10) \| "auto"; default $BEVFUSION_DISPATCH or "eth"
	num_command_queues=None, # default $BEVFUSION_NUM_CQS or 1
	weights_dir=None, # explicit local weights directory; no Hub access
	warmup_variants="default", # trace variants to capture now; see "Warm-up"
	verbose=False,
	**compile_params, # none for this model: the device shapes are the three capacity buckets below
	) -> BEVFusion
	```

	What it does: resolves the weights first, so a Hub problem never claims the chip (`weights_dir` >
	`$BEVFUSION_WEIGHTS_DIR` > a local `model_id` directory > the HF snapshot at the pinned revision, restricted to
	`bevfusion_lidar.onnx, ml_package_bevfusion_lidar.param.yaml, detection_class_remapper.param.yaml, deploy_metadata.yaml`
	(34 MB), with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ; the other open parameters
	are `DEVICE_DEFAULTS` in `tt_bevfusion/device.py`, overridable with `BEVFUSION_*`), builds the graph, then compiles
	and captures the metal traces of every warm-up variant. If ETH dispatch cannot open (tt-metal without the patch), it
	warns and falls back to WORKER dispatch.

	The camera-lidar profile of the Autoware package (`bevfusion_camera_lidar.onnx` + the Swin-T image backbone) is not
	part of this release; it is planned for a later one.

	## Warm-up

	`from_pretrained` returns a warm model: it builds the graph, runs every trace segment once eagerly (this compiles the
	kernels and uploads the weights) and captures four metal traces: the sparse encoder once per capacity bucket
	(`sparse_s`, `sparse_m`, `sparse_full`) and the dense BEV network (`dense`). Every call then replays the sparse trace
	of the frame's bucket and the dense trace; nothing is compiled on a call.

	\| bucket \| active rows per sparse level (l1 / l2 / l3 / l4 / out) \| typical frames \|
	\|---\|---\|---\|
	\| `s` \| 32,768 / 40,960 / 24,576 / 8,192 / 8,192 \| one sweep of a 64-beam scan (test.pcd: 21,359 voxels), nuScenes 32-beam frames \|
	\| `m` \| 131,072 / 147,456 / 81,920 / 32,768 / 24,576 \| dense 2-sweep frames (PandaSet: about 95,000 voxels; Autoware's sample rosbag: 98,334) \|
	\| `full` \| 262,144 / 262,144 / 131,072 / 65,536 / 65,536 \| anything larger, up to Autoware's 256,000-voxel cap \|

	- The bucket is chosen per frame on the host from the actual level counts (smallest that holds every level). The
	sparse trace's cost follows the bucket more than the frame: test.pcd replays in 3.55 ms in `s`, 7.2 ms in `m`,
	12.5 ms in `full` (every bucket gives the same outputs).
	- A frame whose counts exceed `full` is refused before the device runs (`tt_bevfusion.io.InputError`, HTTP 400, the
	message names the levels). Nothing is clipped. None of the 122 measured frames came close (the densest is at about
	37 % of `full`'s level-1 rows), but with the level ratios measured on PandaSet (level 2 can reach 1.43× level 1) a
	frame from about 180,000 voxels on can hit it, below Autoware's 256,000-voxel cap.
	- Measured (2026-10-12, ETH dispatch, 1 CQ, 12x10, the shipped sample): with an empty JIT cache the load took
	201.5 s (firmware and every kernel compiled, on a loaded host); with that cache warm, in a new process, 38.0 s (graph
	build 31.6 s, most of it the numba compile of the host stages, which has no on-disk cache; eager warm-up and capture
	of the 4 traces 1.9 s). The traces are captured during the load, so no call compiles anything (first / second call
	on the shipped sample: 40.9 / 33.6 ms and 40.7 / 33.5 ms in two processes; the host stages move with the load of the
	shared host, load average 7-9 during these runs).
	- `warmup_variants="none"` (server: `BEVFUSION_WARMUP=none`) skips the capture at load; call `model.warmup()` before
	the first frame (it is idempotent).
	- The JIT cache is tt-metal's (`~/.cache/tt-metal-cache`, or `TT_METAL_CACHE`); inside the container it is
	`~/.cache/tt-model/bevfusion-p150/cache` of the host. A process with WORKER dispatch (11x10) compiles its own
	kernels.

	## Call: `model(points, **kwargs)`

	\| Argument \| Type \| Description \|
	\|---\|---\|---\|
	\| `points` (first positional) \| path, `bytes` + `fmt=`, (N, C) array / tensor, structured array, `PointCloud`, JSON envelope \| one LiDAR cloud in `base_link` with x, y, z (see "Input types"); a column named `time_lag` marks Autoware's already densified 2-sweep cloud \|
	\| `sweeps` \| list of `{"points", "time_lag_s", "T_current_from_sweep"}` \| client-side densification: the past sweeps, moved into the current frame with Autoware's float32 sweep arithmetic \|
	\| `stream` \| `{"id", "timestamp_s", "T_world_from_ego", "reset"}` \| server-side densification exactly as Autoware's `PointCloudDensification` (`densification_num_past_frames` 1, per stream id, LRU of 16); a frame without its pose is refused and not cached (Autoware's TF failure) \|
	\| `score_threshold` \| float in [0, 1], default 0.1 \| every class and distance bin (Autoware's `min_confidence_scores`); a decoded score below it is dropped (`<`) \|
	\| `circle_nms_dist_threshold` \| float in [0, 10] m, default 0.5 \| class-agnostic circle NMS between the score-sorted proposals (<= 0 means 0.5 m, as the node) \|
	\| `iou_nms_threshold` \| float in [0, 1], default 0.1 \| Autoware's IoU-BEV NMS: suppress when the BEV IoU is above it \|
	\| `iou_nms_search_distance_2d` \| float in [0, 200] m, default 10 \| the IoU-BEV NMS search radius \|
	\| `remap_classes` \| bool, default `True` \| Autoware's area class remapper (CAR to TRUCK for 12.1-36 m², CAR / TRUCK / BUS to TRAILER from 36 m²) \|
	\| `max_detections` \| int >= 1, default `None` (no cap, as Autoware) \| keep the N best after NMS (not an Autoware knob) \|

	Every knob is host-only (PLAN.md 2.6, RT-host): no device shape changes, and the defaults reproduce Autoware's
	`bevfusion_lidar.param.yaml` and the ml_package exactly (the yaw-norm thresholds and the distance bins are not knobs:
	the node's values, yaw-norm 0.3 for CAR, TRUCK, BUS, BICYCLE and 0 for PEDESTRIAN; distance bins 50 / 90 / 121 /
	200 m). They are validated before the device runs (a bad value raises `tt_bevfusion.io.InputError`, HTTP 400). The
	device test `test_stream_and_params` checks `stream=` and non-default values against the CPU reference.

	### Input types

	- Point clouds: a path (`.bin` raw float32 rows -- pass `fields=` if not `x, y, z, intensity`; `.npy`;
	`.npz`; `.pcd` ascii / binary / binary_compressed), raw bytes with `fmt=`, an (N, C) numpy array or torch tensor,
	a structured array, the JSON envelope of `/predict`, or `tt_bevfusion.io.PointCloud`. Frame: `base_link`, metres.
	An unnamed (N, 4) array is x, y, z, intensity; name the columns (structured array, `PointCloud`, `fields=`) for any
	other layout, e.g. x, y, z, time_lag.
	- `intensity` is not a network input (`use_intensity: false` in the ml_package): a column is accepted and ignored.
	- Three ways to give Autoware's 2-sweep input: a cloud with a `time_lag` column (already densified, used as is),
	`sweeps=` (densified on the host), or `stream=` (the node's own state machine). A plain cloud is one sweep.
	- Rows with a non-finite x / y / z are dropped, and so are rows at exactly (0, 0, 0): the missing returns of organized
	clouds.
	- Voxelization: 0.17 × 0.17 × 0.2 m voxels over ±122.4 m (x, y) and -3 to 5 m (z), at most 256,000 voxels
	(Autoware's clip; `meta.voxel_overflow`) and 10 points per voxel: the first 10 in input order (the training-time
	deterministic rule; Autoware keeps whichever 10 its GPU atomics leave).
	- Images, calibration and planner inputs are refused (the lidar-only profile).

	## Output

	`model(...)` returns a `tt_bevfusion.Output` (= `ttaw.outputs.Detections3D`); `out.to_dict()` is exactly the
	`POST /predict` body (SERVING.md section 3.2).

	\| field \| type \| meaning \|
	\|---\|---\|---\|
	\| `boxes` \| float32 `[N, 7]` \| x, y, z (box centre), length, width, height, yaw in `base_link` metres / radians, the convention of Autoware's autoware_bevfusion node (yaw counter-clockwise from +x); sorted by score \|
	\| `scores` \| float32 `[N]` \| Autoware's `existence_probability` (`query_heat · σ(heatmap)` of the winning class) \|
	\| `label_ids` \| int32 `[N]` \| `autoware_perception_msgs/ObjectClassification` values (CAR 1, TRUCK 2, BUS 3, TRAILER 4, BICYCLE 6, PEDESTRIAN 7); `labels` holds the names \|
	\| `meta` \| dict \| `variant`; `points_in`, `points_in_range`, `num_voxels`, `voxels_total`, `voxel_overflow`, `points_dropped_by_cap`, `max_points_in_voxel`; `sparse` (`counts` per level, `pairs` per neighbour map, `bucket`, `overflow`); `skipped` (no voxel: no inference, no objects, as the node); `densify` (mode, points); `counts` (`proposals`, `after_threshold`, `after_circle_nms`, `after_iou_nms`); per detection `orientation_availability`, `label_before_remap` and `proposal` (the top-k slot) \|
	\| `timing_ms` \| dict \| `preprocess`, `device` (input packing + H2D + both traces + D2H + unpack), `postprocess`, `total` of this call \|

	`out.to_dicts()` gives `[{"label", "label_id", "score", "center", "size", "yaw"}]`; `len(out)` the number of
	detections. No velocity is returned: Autoware decodes the network's velocity rows but does not publish them.

	The last step of the head runs on the host in float32 after the one readback: `score = σ(heatmap)[cls] ·
	query_heat[cls]`, `label = cls` and `center + bev_pos` (this keeps the BEV cell positions out of bf16). Everything
	before it (sparse encoder, SECOND, SECONDFPN, heatmap, local max, top-512, the transformer decoder layer, the box
	heads) runs on the device.

	## Lifetime and information

	- `model.close()` releases the traces and the device tensors and closes the chip if the model opened it; idempotent.
	`with` calls it for you; an unclosed model is closed when Python exits.
	- `model.info`: weights (repo, tag, revision, path), device (`dispatch`, `grid`, `cores`, `num_command_queues`,
	`fallback`), variant, warm variants, warm-up times, runtime parameter defaults, the config, the pre-processing and
	the trace descriptions.
	- Calls from several threads are safe and pipelined (`BEVFUSION_PIPELINE=2`, the default): host pre-processing, the
	device stage (staging, H2D, replays, D2H) and host post-processing each run under their own lock, so up to three
	frames overlap while the chip runs one at a time; every result equals a serial call's. One model per process per chip.
	- `tt_bevfusion.viz.render_bev(points, out.to_dicts(), reference=...)` draws a bird's-eye view (PIL, no OpenCV).

	## Speed

	Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, the shipped precision (`code/scripts/bench.py`, 20 iterations, two
	runs; the numbers of `OPT_BASELINE.md` "After optimization" and `VERIFICATION_OPT_2026-10-12.md`, 2026-10-12, on a
	shared host whose load moves the host stages, load average 4-5):

	\| stage, p50 \| test.pcd (shipped, bucket `s`) \| PandaSet 019 f40 (not shipped, bucket `m`) \|
	\|---\|---:\|---:\|
	\| `model(path)` end to end \| 32.5-32.6 ms \| 76.4-77.5 ms \|
	\| host pre-processing (decode, densify, voxelize, VFE, the sorted level sets) \| 14.2 ms \| 48.5-49.0 ms \|
	\| input packing (staged VFE rows, level keys) + H2D \| 0.64 + 0.51 ms \| 2.0-2.1 + 1.01 ms \|
	\| device traces, one frame (sparse, incl. the device rulebooks, + dense) \| 3.6 + 12.5 ms \| 9.3 + 12.5 ms \|
	\| D2H + unpack \| 0.55 + 0.21 ms \| 0.61 + 0.25 ms \|
	\| host post-processing (decode, NMS, remap) \| 0.64 ms \| 1.08-1.11 ms \|

	- Back-to-back frames take 15.99 ms (test.pcd) and 21.72 ms (PandaSet) of device time: 62.5 and 46.0 frames/s. A
	synchronous call gives 30.7 and 13.0 calls/s; three concurrent callers get a frame every 18.8-18.9 ms (test.pcd) and
	32.2 ms (PandaSet), 53 and 31 frames/s.
	- The first release (2026-10-09) took 657.1 / 2,786.0 ms per call (86-88 % of it a numpy rulebook build, now on the
	device) and 53.09 / 112.02 ms of device time per frame. The optimization steps are in `OPT_REPORT.md`.

	## Tests

	From the repository root, with `code/` on `PYTHONPATH` (or the package installed):

	- Host (no device, no ttnn call): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevfusion/tests`. Inside the
	tt-models workspace the suite runs on its fake ttnn (`-p fake_ttnn_plugin`), which also enables `test_tt_host.py`
	(the TT graph emulated on the CPU against the fp32 reference); tests that need workspace-only oracles or data skip
	elsewhere, and a missing sample FAILS inside the workspace.
	- Device (one p150): `python -m pytest -q -s code/tt_bevfusion/tests/test_pcc_device.py
	code/tt_bevfusion/tests/test_e2e_device.py` (per-module PCC and the top-500 selection against the fp32 CPU
	reference on replay outputs, end-to-end agreement on the PandaSet sample, 7 development frames and 10 held-out
	frames, `stream=` and non-default parameters, replay == eager, API == `/predict`). The public-frame gates run only
	inside the tt-models workspace, where their data lives: the frames are PandaSet and nuScenes sweeps and Autoware's
	sample rosbag, which this repository does not ship.
	- The gates are frozen in `tests/*.gates.json` at the first green run and are never loosened.

	## Limits

	- Batch 1 on the chip; one model per process; at most 256,000 voxels (Autoware's clip) and 10 points per voxel (the
	first 10 in input order); the sparse encoder's three capacity buckets (a frame beyond `full` is refused, see
	"Warm-up").
	- Outputs differ slightly from the fp32 reference: an object near one of Autoware's decision thresholds (score,
	yaw-norm gate, circle NMS, remap area) can appear, disappear or change label (README "Caveats" has the measured
	rates).
	- One p150, ETH dispatch (12x10); `dispatch="worker"` is an A/B opt-in (11x10). Not a ROS 2 node. Lidar-only: the
	camera-lidar profile is not in this release.