code/PYTHON.md

Python API: BEVDet (Autoware tensorrt_bevdet) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the API and the server share the decoders, the device graph and the post-processing, so the outputs are the same bit for bit (tests/test_e2e_device.py::test_api_equals_server) and the speed is the same.

Install

Install the package on top of an environment that already has ttnn (a tt-metal python_env at 44d66500520 with patches/tt-metal-eth-dispatch.patch, or the tt-model container). From the root of the model repository (the directory that holds pyproject.toml, README.md and code/):

pip install -e .                        # the Python API (numpy<2, pillow, pyyaml, onnx, huggingface_hub, opencv-python-headless<4.12)
pip install -e ".[server,test]"         # + the HTTP server and the tests

The pip project is the repository's top-level pyproject.toml; it installs the package from code/tt_bevdet (there is no pyproject.toml inside code/, because the container build copies code/ over the tt-metal tree). ttnn and torch come from tt-metal and are not declared, so pip never replaces tt-metal's torch.

The package carries tt_bevdet.ttaw, the shared code of the Autoware ports to Blackhole (device open, trace runner, decoders, model base class, HTTP app), vendored at the version recorded in code/tt_bevdet/ttaw/VENDORED.json.

You want to run	Extras
the Python API	none
the HTTP server (tt_bevdet.server.app, see SERVING.md)	server
host tests (no device; device tests are skipped): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevdet/tests	server,test
device tests: python -m pytest -q -s code/tt_bevdet/tests/test_pcc_device.py code/tt_bevdet/tests/test_e2e_device.py code/tt_bevdet/tests/test_streams_device.py	test (most gates need the tt-models workspace data; outside it they skip)

Quickstart

from tt_bevdet import BEVDet, load_sample

with BEVDet.from_pretrained(device_id=0) as model:
    out = model(**load_sample("code/tt_bevdet/samples/synthetic_6cam.json"))   # six cameras + calibration preset

for d in out.to_dicts():
    print(d["label"], d["autoware_label"], d["score"], d["center"], d["size"], d["yaw"], d["velocity"])

The relative path assumes the current directory is the repository root. examples/quickstart.py finds the sample relative to its own location (it runs from any directory) and also writes the /predict JSON and a bird's-eye view PNG to examples/output/. The shipped sample is a synthetic test pattern (eight objects placed by its generator: code/tt_bevdet/samples/README.md); use your own rig's images and calibration for anything else.

from_pretrained returns a model that is ready: the load ends with one synthetic frame through the whole path (see "Warm-up"). The first call on a new rig calibration builds that calibration's tables, and the first image of a new non-1600×900 size checks the host resize against OpenCV once, so it takes longer: 0.18 s (nuScenes), 0.54 s (the shipped sample) and 1.6 s (PandaSet 1920×1080) measured; the later calls take the times under "Speed". A load takes about 14-15 s with a warm kernel cache on the measurement host. The first load on an empty kernel cache also compiles the kernels (a few minutes). verbose=True shows the progress.

BEVDet.from_pretrained(...)

BEVDet.from_pretrained(
    model_id=None,          # HF repo or a local directory holding bevdet_one_lt_d.onnx; default AutowareFoundation/tensorrt_bevdet
    *,
    revision=None,          # default for the default repo: the validated commit 391466349a1 (tag v1.0)
    variant=None,           # "default" (the only variant)
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_bevdet.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $BEVDET_DISPATCH or "eth"
    num_command_queues=None,  # default $BEVDET_NUM_CQS or 1 (2 is selectable: uploads on CQ1; no gain for one stream)
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # "default" captures every trace now; "none" defers it to model.warmup()
    verbose=False,
    # load-time knobs (default: $BEVDET_<NAME>, else the value below; SERVING.md section 3.5):
    max_streams=4,          # stream ids that keep a BEV history at once (LRU eviction; one 25 MB device ring bank each)
    history_stride=1,       # frames per history push (1 = the Autoware node; > 1 is not device-validated yet)
    iou_mode="vendor",      # rotated IoU of the Scale-NMS: "vendor" (deployed) | "exact" (polygon clipping)
    precision="HiFi4+fp32+l1acc:w=fp32",  # conv / linear precision (the release policy: fp32 weights, bf16 activations)
) -> BEVDet

What it does:

1. It finds the weights before it claims the chip: weights_dir > $BEVDET_WEIGHTS_DIR > a local model_id directory > the Hugging Face snapshot at the pinned revision, restricted to bevdet_one_lt_d.onnx (305 MB), with an offline fallback to the cache. The repo is public: if your HF cache holds a token, set HF_HUB_DISABLE_IMPLICIT_TOKEN=1 so none is sent (the container does).
2. It opens the chip as every published number was measured: ETH dispatch, 1 command queue, the 12×10 compute grid (read from the device, never hard-coded); L1_SMALL 32 KiB and a 256 MiB trace region (DEVICE_DEFAULTS in tt_bevdet/device.py, overridable with BEVDET_*). If ETH dispatch cannot open (tt-metal without the patch), it warns and falls back to WORKER dispatch (11×10); the card's numbers do not apply to that mode.
3. It reads the ONNX initializers as data (no ONNX Runtime), folds BatchNorm into the convs and applies the port's exact rewrites (ASPP dilation-18 branch as a 1×3 conv, the camera-aware MLP + SE folded into per-calibration gates, a merged CenterHead), then builds the device graph: two trace segments around the host BEVPool (encode, decode) and the history-ring updates (push, push_x, fill).

Warm-up

warmup_variants="default" compiles every program, captures the seven trace variants (encode; decode and its decode_fresh / decode_move forms; the ring updates push, push_x, fill) and runs one synthetic frame through the whole host + device path (the warm-up stream is forgotten afterwards), so no real call compiles anything; only the first call on a new rig calibration builds that calibration's tables (above). The shapes are fixed (six cameras, 256×704 network input, 128×128 BEV), so there is nothing else to warm: any input image size works without a new compile (non-1600×900 images are resized on the host, like the node). model.warmup() is idempotent; with warmup_variants="none" call it before the first frame (a call without captured traces raises).

Call: model(...)

Argument	Type	Description
images	list or dict	The six cameras CAM_FRONT_LEFT, CAM_FRONT, CAM_FRONT_RIGHT, CAM_BACK_LEFT, CAM_BACK, CAM_BACK_RIGHT, in any order: a list of {"camera": name, "image": <image>, "intrinsics": K, "T_ref_from_camera": T} dicts (or "data": base64, the /predict spelling), a {name: image} mapping, or tt_bevdet.io.CameraImages. Each camera exactly once.
calibration	dict	{"cameras": {name: {"intrinsics", "T_ref_from_camera"}}} or {"preset": "<name>"} (files in code/tt_bevdet/calib/); values given with an image win. "scale_intrinsics": False keeps the raw K when an image is resized to 1600×900 (the node's quirk); the default True scales it.
stream	dict	{"id": "default", "reset": False, "timestamp_s": None, "T_world_from_ego": None}: the BEV history (below)
score_threshold	float 0..1	0.2: the node's publication filter
decode_score_threshold	float 0..1	0.1: the vendor decode keeps cells with sigmoid(max logit) above it
nms_iou_threshold	float 0..1	0.2: the Scale-NMS rotated BEV IoU
max_pre_nms, max_detections	int 1..500	500, 500: the pre-NMS cap and the post-NMS cap
nms_rescale_factor	list of 10 or dict	the Scale-NMS factor per class (default [1.0, 0.7, 0.7, 0.4, 0.55, 1.1, 1.0, 1.0, 1.5, 3.5], class order of BEVDet.LABELS), or {"pedestrian": 2.0} to override some; each in (0, 100]

Unknown keyword arguments or values out of range raise tt_bevdet.io.InputError (the server's 400).

Input types

• Images: a path, PNG / JPEG bytes, a PIL.Image, a uint8 HxWx3 RGB array or 3xHxW tensor, or a float array in [0, 1]. Any size: images that are not 1600×900 are resized to it with OpenCV INTER_LINEAR (the node's cv::resize), and their K is scaled with them unless scale_intrinsics=False.
• Intrinsics: 3×3 K, 3×4 P or {"fx", "fy", "cx", "cy"} of the image as sent (rectified rigs: P[0:3, 0:3]).
• Extrinsics T_ref_from_camera: camera optical frame (x right, y down, z forward) -> base_link, as a 4×4 / 3×4 matrix, {"translation", "rotation_wxyz"} (nuScenes / bevdet_vendor yaml) or {x, y, z, roll, pitch, yaw} (Autoware sensor_kit_calibration.yaml).
• tt_bevdet.load_sample(path, with_stream=False) reads a sample manifest ({"images": {camera: relative path}, "calibration": {...} | {"preset": name}, "stream": {...}}) into these keyword arguments; with_stream=True adds the manifest's stream (ego pose).

Streams: the BEV history

• A new stream["id"] starts like a freshly launched Autoware node: zero history on its first frame.
• reset=True is a scene change: every history slot is replaced by the current frame.
• T_world_from_ego (4×4 pose of base_link in a fixed world frame) turns on motion compensation: each history slot is warped into the current ego frame. Without it the history is never aligned, exactly as in the deployed node. Pass the pose whenever you have a localisation: on nuScenes mini_val it is worth +0.11 mAP / +0.16 NDS (README "Demo & Performances").
• max_streams (default 4) ids keep their history at once; a further id evicts the least recently used one, whose next frame starts fresh. Switching the active stream swaps a 25 MB device bank (save + load: 1.07 ms on the device, measured).
• A frame that raises after the device ring changed restarts its stream (next frame: zero history), so the device ring and the host bookkeeping never disagree; a frame that raises earlier leaves the history untouched.
• model.stream_info() returns the table (max_streams, history_stride, policy, the ids that hold a history).

Output

model(...) returns a tt_bevdet.host.outputs.BEVDetDetections (a ttaw.outputs.Detections3D):

attribute	meaning
boxes	float32 [N, 7]: x, y, z (box centre; z = gravity centre), length, width, height, yaw (CCW from +x), base_link metres / radians
scores	float32 [N], sorted descending
label_ids	int32 [N], index into labels
labels	the 10 network class names (car, truck, construction_vehicle, bus, trailer, barrier, motorcycle, bicycle, pedestrian, traffic_cone)
autoware_labels	per class, the label the node publishes (UNKNOWN for construction_vehicle, barrier, traffic_cone); autoware_label_names gives it per row
velocities	float32 [N, 2]: (vx, vy) m/s in base_link axes
twists()	float64 [N, 2]: the node's (twist.linear.x, twist.angular.z)
timing_ms	preprocess, encode (H2D + encode trace + D2H), bevpool (host BEVPool + AlignBEV grid), bev_decode (H2D + decode trace + D2H + ring update), device (the three), postprocess, total
meta	class_names, autoware_labels, score_threshold, stream (id, frame, flag)

out.to_dicts() gives one dict per detection (label, label_id, autoware_label, score, center, size, yaw, velocity, twist); out.to_dict() gives the /predict body, out.to_dict("npz") adds lossless arrays.

Lifetime and information

• model.close() releases the traces and the device tensors and closes the chip if the model opened it; idempotent. with calls it for you; an unclosed model is closed when Python exits.
• model.info: weights (repo, tag, revision, path), device (dispatch, grid, CQs, fallback), variant, warm variants, warm-up times, runtime parameter defaults, the trace layout, the knobs and the stream table.
• Calls from several threads are safe: the device calls are serialised. One model per process per chip.

Speed

Measured with code/scripts/bench.py (OPT_REPORT.md, VERIFICATION_OPT_2026-10-14.md; ETH dispatch, 1 CQ, 12×10 grid, warm, batch 1; decoded images in, so JPEG / PNG decoding is not included):

	p50
device, one steady frame (encode + decode + push back to back)	about 10.0 ms (about 100 frames/s device-bound; 14.7 ms with ego poses)
model(...) on 1600×900 images (synthetic, Autoware mode)	about 31 ms
model(...) on nuScenes 1600×900 with ego poses	about 39-40 ms
model(...) on PandaSet 1920×1080 (host resize)	about 34-36 ms

The first release took 34.15 ms on the device and 173 / 220 / 342 ms per model(...) call on the same inputs. The host stages (pre-processing, BEVPool, Scale-NMS) are now numba twins and the images go up as uint8 crops, so the host and the device share the time about equally; the host stages move with the load of the shared measurement host. Decode your images in a separate thread to hide the JPEG cost.

Tests

• Host (no device): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevdet/tests (with the fake ttnn of the workspace: -p fake_ttnn_plugin, SERVING.md section 1).
• Device: python -m pytest -q -s code/tt_bevdet/tests/test_pcc_device.py code/tt_bevdet/tests/test_e2e_device.py code/tt_bevdet/tests/test_streams_device.py. The gates (tests/*.gates.json) were frozen at the first green run and are never loosened. The public-data gates need the tt-models workspace (nuScenes, PandaSet); outside it they skip, and test_shipped_sample_agreement (the container smoke's comparison) runs anywhere.

Limits

• Batch 1 on the chip; one model per process; six cameras, 256×704 network input, 118 depth bins, 128×128 BEV grid (±51.2 m at 0.8 m), 8 history frames, 10 classes. A rig whose BEVPool tables exceed M 370,000 / N 14,000 is refused (InputError), as TensorRT would.
• BEVPool and the Scale-NMS run on the host (numba, bit-identical to the reference order); a device BEVPool (BEVDET_BEVPOOL=device) is experimental, not served, and hung the chip in one of its forms (OPT_REPORT.md round 7).