# Python API: BEVDet (Autoware tensorrt_bevdet) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the
API and the server share the decoders, the device graph and the post-processing, so the outputs are the same bit for
bit (`tests/test_e2e_device.py::test_api_equals_server`) and the speed is the same.

## Install

Install the package on top of an environment that already has `ttnn` (a tt-metal `python_env` at `44d66500520`
with `patches/tt-metal-eth-dispatch.patch`, or the tt-model container). From the root of the model repository (the
directory that holds `pyproject.toml`, `README.md` and `code/`):

```bash
pip install -e .                        # the Python API (numpy<2, pillow, pyyaml, onnx, huggingface_hub, opencv-python-headless<4.12)
pip install -e ".[server,test]"         # + the HTTP server and the tests
```

The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_bevdet` (there
is no `pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree). ttnn and
torch come from tt-metal and are not declared, so pip never replaces tt-metal's torch.

The package carries `tt_bevdet.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner,
decoders, model base class, HTTP app), vendored at the version recorded in `code/tt_bevdet/ttaw/VENDORED.json`.

| You want to run | Extras |
|---|---|
| the Python API | none |
| the HTTP server (`tt_bevdet.server.app`, see `SERVING.md`) | `server` |
| host tests (no device; device tests are skipped): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevdet/tests` | `server,test` |
| device tests: `python -m pytest -q -s code/tt_bevdet/tests/test_pcc_device.py code/tt_bevdet/tests/test_e2e_device.py code/tt_bevdet/tests/test_streams_device.py` | `test` (most gates need the tt-models workspace data; outside it they skip) |

## Quickstart

```python
from tt_bevdet import BEVDet, load_sample

with BEVDet.from_pretrained(device_id=0) as model:
    out = model(**load_sample("code/tt_bevdet/samples/synthetic_6cam.json"))   # six cameras + calibration preset

for d in out.to_dicts():
    print(d["label"], d["autoware_label"], d["score"], d["center"], d["size"], d["yaw"], d["velocity"])
```

The relative path assumes the current directory is the repository root. `examples/quickstart.py` finds the sample
relative to its own location (it runs from any directory) and also writes the `/predict` JSON and a bird's-eye view
PNG to `examples/output/`. The shipped sample is a synthetic test pattern (eight objects placed by its generator:
`code/tt_bevdet/samples/README.md`); use your own rig's images and calibration for anything else.

`from_pretrained` returns a model that is ready: the load ends with one synthetic frame through the whole path (see
"Warm-up"). The first call on a new rig calibration builds that calibration's tables, and the first image of a new
non-1600×900 size checks the host resize against OpenCV once, so it takes longer: 0.18 s (nuScenes), 0.54 s (the
shipped sample) and 1.6 s (PandaSet 1920×1080) measured; the later calls take the times under "Speed". A load takes
about 14-15 s with a warm kernel cache on the measurement host. The first load on an empty kernel cache also compiles
the kernels (a few minutes). `verbose=True` shows the progress.

## `BEVDet.from_pretrained(...)`

```python
BEVDet.from_pretrained(
    model_id=None,          # HF repo or a local directory holding bevdet_one_lt_d.onnx; default AutowareFoundation/tensorrt_bevdet
    *,
    revision=None,          # default for the default repo: the validated commit 391466349a1 (tag v1.0)
    variant=None,           # "default" (the only variant)
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_bevdet.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $BEVDET_DISPATCH or "eth"
    num_command_queues=None,  # default $BEVDET_NUM_CQS or 1 (2 is selectable: uploads on CQ1; no gain for one stream)
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # "default" captures every trace now; "none" defers it to model.warmup()
    verbose=False,
    # load-time knobs (default: $BEVDET_<NAME>, else the value below; SERVING.md section 3.5):
    max_streams=4,          # stream ids that keep a BEV history at once (LRU eviction; one 25 MB device ring bank each)
    history_stride=1,       # frames per history push (1 = the Autoware node; > 1 is not device-validated yet)
    iou_mode="vendor",      # rotated IoU of the Scale-NMS: "vendor" (deployed) | "exact" (polygon clipping)
    precision="HiFi4+fp32+l1acc:w=fp32",  # conv / linear precision (the release policy: fp32 weights, bf16 activations)
) -> BEVDet
```

What it does:

1. It finds the weights before it claims the chip: `weights_dir` > `$BEVDET_WEIGHTS_DIR` > a local `model_id`
   directory > the Hugging Face snapshot at the pinned revision, restricted to `bevdet_one_lt_d.onnx` (305 MB), with
   an offline fallback to the cache. The repo is public: if your HF cache holds a token, set
   `HF_HUB_DISABLE_IMPLICIT_TOKEN=1` so none is sent (the container does).
2. It opens the chip as every published number was measured: ETH dispatch, 1 command queue, the 12×10 compute grid
   (read from the device, never hard-coded); L1_SMALL 32 KiB and a 256 MiB trace region (`DEVICE_DEFAULTS` in
   `tt_bevdet/device.py`, overridable with `BEVDET_*`). If ETH dispatch cannot open (tt-metal without the patch), it
   warns and falls back to WORKER dispatch (11×10); the card's numbers do not apply to that mode.
3. It reads the ONNX initializers as data (no ONNX Runtime), folds BatchNorm into the convs and applies the port's
   exact rewrites (ASPP dilation-18 branch as a 1×3 conv, the camera-aware MLP + SE folded into per-calibration gates,
   a merged CenterHead), then builds the device graph: two trace segments around the host BEVPool (`encode`, `decode`)
   and the history-ring updates (`push`, `push_x`, `fill`).

## Warm-up

`warmup_variants="default"` compiles every program, captures the seven trace variants (`encode`; `decode` and its `decode_fresh` / `decode_move` forms; the ring updates `push`, `push_x`, `fill`) and runs one synthetic frame through
the whole host + device path (the warm-up stream is forgotten afterwards), so no real call compiles anything; only
the first call on a new rig calibration builds that calibration's tables (above). The shapes are fixed (six cameras, 256×704 network input, 128×128 BEV), so there is nothing else to
warm: any input image size works without a new compile (non-1600×900 images are resized on the host, like the node).
`model.warmup()` is idempotent; with `warmup_variants="none"` call it before the first frame (a call without captured
traces raises).

## Call: `model(...)`

| Argument | Type | Description |
|---|---|---|
| `images` | list or dict | The six cameras `CAM_FRONT_LEFT`, `CAM_FRONT`, `CAM_FRONT_RIGHT`, `CAM_BACK_LEFT`, `CAM_BACK`, `CAM_BACK_RIGHT`, in any order: a list of `{"camera": name, "image": <image>, "intrinsics": K, "T_ref_from_camera": T}` dicts (or `"data"`: base64, the `/predict` spelling), a `{name: image}` mapping, or `tt_bevdet.io.CameraImage`s. Each camera exactly once. |
| `calibration` | dict | `{"cameras": {name: {"intrinsics", "T_ref_from_camera"}}}` or `{"preset": "<name>"}` (files in `code/tt_bevdet/calib/`); values given with an image win. `"scale_intrinsics": False` keeps the raw K when an image is resized to 1600×900 (the node's quirk); the default `True` scales it. |
| `stream` | dict | `{"id": "default", "reset": False, "timestamp_s": None, "T_world_from_ego": None}`: the BEV history (below) |
| `score_threshold` | float 0..1 | 0.2: the node's publication filter |
| `decode_score_threshold` | float 0..1 | 0.1: the vendor decode keeps cells with sigmoid(max logit) above it |
| `nms_iou_threshold` | float 0..1 | 0.2: the Scale-NMS rotated BEV IoU |
| `max_pre_nms`, `max_detections` | int 1..500 | 500, 500: the pre-NMS cap and the post-NMS cap |
| `nms_rescale_factor` | list of 10 or dict | the Scale-NMS factor per class (default `[1.0, 0.7, 0.7, 0.4, 0.55, 1.1, 1.0, 1.0, 1.5, 3.5]`, class order of `BEVDet.LABELS`), or `{"pedestrian": 2.0}` to override some; each in (0, 100] |

Unknown keyword arguments or values out of range raise `tt_bevdet.io.InputError` (the server's 400).

### Input types

- Images: a path, PNG / JPEG bytes, a `PIL.Image`, a uint8 HxWx3 RGB array or 3xHxW tensor, or a float array in
  [0, 1]. Any size: images that are not 1600×900 are resized to it with OpenCV `INTER_LINEAR` (the node's
  `cv::resize`), and their K is scaled with them unless `scale_intrinsics=False`.
- Intrinsics: 3×3 K, 3×4 P or `{"fx", "fy", "cx", "cy"}` of the image **as sent** (rectified rigs: P[0:3, 0:3]).
- Extrinsics `T_ref_from_camera`: camera optical frame (x right, y down, z forward) -> `base_link`, as a 4×4 / 3×4
  matrix, `{"translation", "rotation_wxyz"}` (nuScenes / bevdet_vendor yaml) or `{x, y, z, roll, pitch, yaw}`
  (Autoware `sensor_kit_calibration.yaml`).
- `tt_bevdet.load_sample(path, with_stream=False)` reads a sample manifest (`{"images": {camera: relative path},
  "calibration": {...} | {"preset": name}, "stream": {...}}`) into these keyword arguments; `with_stream=True` adds
  the manifest's stream (ego pose).

### Streams: the BEV history

- A new `stream["id"]` starts like a freshly launched Autoware node: zero history on its first frame.
- `reset=True` is a scene change: every history slot is replaced by the current frame.
- `T_world_from_ego` (4×4 pose of `base_link` in a fixed world frame) turns on motion compensation: each history
  slot is warped into the current ego frame. Without it the history is never aligned, exactly as in the deployed
  node. Pass the pose whenever you have a localisation: on nuScenes mini_val it is worth +0.11 mAP / +0.16 NDS
  (README "Demo & Performances").
- `max_streams` (default 4) ids keep their history at once; a further id evicts the least recently used one, whose
  next frame starts fresh. Switching the active stream swaps a 25 MB device bank (save + load: 1.07 ms on the
  device, measured).
- A frame that raises after the device ring changed restarts its stream (next frame: zero history), so the device
  ring and the host bookkeeping never disagree; a frame that raises earlier leaves the history untouched.
- `model.stream_info()` returns the table (`max_streams`, `history_stride`, policy, the ids that hold a history).

## Output

`model(...)` returns a `tt_bevdet.host.outputs.BEVDetDetections` (a `ttaw.outputs.Detections3D`):

| attribute | meaning |
|---|---|
| `boxes` | float32 [N, 7]: x, y, z (box centre; z = gravity centre), length, width, height, yaw (CCW from +x), `base_link` metres / radians |
| `scores` | float32 [N], sorted descending |
| `label_ids` | int32 [N], index into `labels` |
| `labels` | the 10 network class names (`car`, `truck`, `construction_vehicle`, `bus`, `trailer`, `barrier`, `motorcycle`, `bicycle`, `pedestrian`, `traffic_cone`) |
| `autoware_labels` | per class, the label the node publishes (UNKNOWN for construction_vehicle, barrier, traffic_cone); `autoware_label_names` gives it per row |
| `velocities` | float32 [N, 2]: (vx, vy) m/s in `base_link` axes |
| `twists()` | float64 [N, 2]: the node's (twist.linear.x, twist.angular.z) |
| `timing_ms` | `preprocess`, `encode` (H2D + `encode` trace + D2H), `bevpool` (host BEVPool + AlignBEV grid), `bev_decode` (H2D + `decode` trace + D2H + ring update), `device` (the three), `postprocess`, `total` |
| `meta` | `class_names`, `autoware_labels`, `score_threshold`, `stream` (`id`, `frame`, `flag`) |

`out.to_dicts()` gives one dict per detection (`label`, `label_id`, `autoware_label`, `score`, `center`, `size`,
`yaw`, `velocity`, `twist`); `out.to_dict()` gives the `/predict` body, `out.to_dict("npz")` adds lossless arrays.

## Lifetime and information

- `model.close()` releases the traces and the device tensors and closes the chip if the model opened it; idempotent.
  `with` calls it for you; an unclosed model is closed when Python exits.
- `model.info`: weights (repo, tag, revision, path), device (dispatch, grid, CQs, fallback), variant, warm variants,
  warm-up times, runtime parameter defaults, the trace layout, the knobs and the stream table.
- Calls from several threads are safe: the device calls are serialised. One model per process per chip.

## Speed

Measured with `code/scripts/bench.py` (OPT_REPORT.md, VERIFICATION_OPT_2026-10-14.md; ETH dispatch, 1 CQ, 12×10
grid, warm, batch 1; decoded images in, so JPEG / PNG decoding is not included):

| | p50 |
|---|---:|
| device, one steady frame (`encode` + `decode` + `push` back to back) | about 10.0 ms (about 100 frames/s device-bound; 14.7 ms with ego poses) |
| `model(...)` on 1600×900 images (synthetic, Autoware mode) | about 31 ms |
| `model(...)` on nuScenes 1600×900 with ego poses | about 39-40 ms |
| `model(...)` on PandaSet 1920×1080 (host resize) | about 34-36 ms |

The first release took 34.15 ms on the device and 173 / 220 / 342 ms per `model(...)` call on the same inputs. The
host stages (pre-processing, BEVPool, Scale-NMS) are now numba twins and the images go up as uint8 crops, so the
host and the device share the time about equally; the host stages move with the load of the shared measurement host.
Decode your images in a separate thread to hide the JPEG cost.

## Tests

- Host (no device): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevdet/tests` (with the fake ttnn of the
  workspace: `-p fake_ttnn_plugin`, SERVING.md section 1).
- Device: `python -m pytest -q -s code/tt_bevdet/tests/test_pcc_device.py code/tt_bevdet/tests/test_e2e_device.py
  code/tt_bevdet/tests/test_streams_device.py`. The gates (`tests/*.gates.json`) were frozen at the first green run and
  are never loosened. The public-data gates need the tt-models workspace (nuScenes, PandaSet); outside it they skip,
  and `test_shipped_sample_agreement` (the container smoke's comparison) runs anywhere.

## Limits

- Batch 1 on the chip; one model per process; six cameras, 256×704 network input, 118 depth bins, 128×128 BEV grid
  (±51.2 m at 0.8 m), 8 history frames, 10 classes. A rig whose BEVPool tables exceed M 370,000 / N 14,000 is
  refused (`InputError`), as TensorRT would.
- BEVPool and the Scale-NMS run on the host (numba, bit-identical to the reference order); a device BEVPool
  (`BEVDET_BEVPOOL=device`) is experimental, not served, and hung the chip in one of its forms (OPT_REPORT.md round 7).