code/PYTHON.md

Python API: BEVFusion-L (Autoware autoware_bevfusion, lidar-only) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are the same.

Install

Install the package on top of an environment that already has ttnn (a tt-metal python_env at 44d66500520 with patches/tt-metal-eth-dispatch.patch, or the tt-model container). From the root of the model repository (the directory that holds pyproject.toml, README.md and code/):

pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests

The pip project is the repository's top-level pyproject.toml; it installs the package from code/tt_bevfusion (there is no pyproject.toml inside code/, because the container build copies code/ over the tt-metal tree).

The package carries tt_bevfusion.ttaw, the shared code of the Autoware ports to Blackhole (device open, trace runner, decoders, sparse rulebooks, model base class, HTTP app), vendored at the version recorded in code/tt_bevfusion/ttaw/VENDORED.json.

You want to run	Extras
the Python API	none
the HTTP server (tt_bevfusion.server.app, see SERVING.md)	server
host tests (no device; device tests are skipped): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevfusion/tests	server,test
device tests: python -m pytest -q -s code/tt_bevfusion/tests/test_pcc_device.py code/tt_bevfusion/tests/test_e2e_device.py	test

Quickstart

from tt_bevfusion import BEVFusion

with BEVFusion.from_pretrained(device_id=0) as model:
    out = model("code/tt_bevfusion/samples/test_pcd.npz")        # Autoware's test.pcd, Apache-2.0
print(out.to_dict())

examples/quickstart.py runs this snippet on the shipped sample and also writes quickstart_bev.png (the points and the boxes from above, tt_bevfusion.viz.render_bev).

BEVFusion.from_pretrained(...)

BEVFusion.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default AutowareFoundation/bevfusion
    *,
    revision=None,          # default for the default repo: the validated commit e1bf164b909 (tag v2.0)
    variant=None,           # load-time variant: "lidar" (the only one in this release); default $BEVFUSION_VARIANT or "lidar"
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_bevfusion.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $BEVFUSION_DISPATCH or "eth"
    num_command_queues=None,  # default $BEVFUSION_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
    **compile_params,       # none for this model: the device shapes are the three capacity buckets below
) -> BEVFusion

What it does: resolves the weights first, so a Hub problem never claims the chip (weights_dir > $BEVFUSION_WEIGHTS_DIR > a local model_id directory > the HF snapshot at the pinned revision, restricted to bevfusion_lidar.onnx, ml_package_bevfusion_lidar.param.yaml, detection_class_remapper.param.yaml, deploy_metadata.yaml (34 MB), with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ; the other open parameters are DEVICE_DEFAULTS in tt_bevfusion/device.py, overridable with BEVFUSION_*), builds the graph, then compiles and captures the metal traces of every warm-up variant. If ETH dispatch cannot open (tt-metal without the patch), it warns and falls back to WORKER dispatch.

The camera-lidar profile of the Autoware package (bevfusion_camera_lidar.onnx + the Swin-T image backbone) is not part of this release; it is planned for a later one.

Warm-up

from_pretrained returns a warm model: it builds the graph, runs every trace segment once eagerly (this compiles the kernels and uploads the weights) and captures four metal traces: the sparse encoder once per capacity bucket (sparse_s, sparse_m, sparse_full) and the dense BEV network (dense). Every call then replays the sparse trace of the frame's bucket and the dense trace; nothing is compiled on a call.

bucket	active rows per sparse level (l1 / l2 / l3 / l4 / out)	typical frames
s	32,768 / 40,960 / 24,576 / 8,192 / 8,192	one sweep of a 64-beam scan (test.pcd: 21,359 voxels), nuScenes 32-beam frames
m	131,072 / 147,456 / 81,920 / 32,768 / 24,576	dense 2-sweep frames (PandaSet: about 95,000 voxels; Autoware's sample rosbag: 98,334)
full	262,144 / 262,144 / 131,072 / 65,536 / 65,536	anything larger, up to Autoware's 256,000-voxel cap

• The bucket is chosen per frame on the host from the actual level counts (smallest that holds every level). The sparse trace's cost follows the bucket more than the frame: test.pcd replays in 3.55 ms in s, 7.2 ms in m, 12.5 ms in full (every bucket gives the same outputs).
• A frame whose counts exceed full is refused before the device runs (tt_bevfusion.io.InputError, HTTP 400, the message names the levels). Nothing is clipped. None of the 122 measured frames came close (the densest is at about 37 % of full's level-1 rows), but with the level ratios measured on PandaSet (level 2 can reach 1.43× level 1) a frame from about 180,000 voxels on can hit it, below Autoware's 256,000-voxel cap.
• Measured (2026-10-12, ETH dispatch, 1 CQ, 12x10, the shipped sample): with an empty JIT cache the load took 201.5 s (firmware and every kernel compiled, on a loaded host); with that cache warm, in a new process, 38.0 s (graph build 31.6 s, most of it the numba compile of the host stages, which has no on-disk cache; eager warm-up and capture of the 4 traces 1.9 s). The traces are captured during the load, so no call compiles anything (first / second call on the shipped sample: 40.9 / 33.6 ms and 40.7 / 33.5 ms in two processes; the host stages move with the load of the shared host, load average 7-9 during these runs).
• warmup_variants="none" (server: BEVFUSION_WARMUP=none) skips the capture at load; call model.warmup() before the first frame (it is idempotent).
• The JIT cache is tt-metal's (~/.cache/tt-metal-cache, or TT_METAL_CACHE); inside the container it is ~/.cache/tt-model/bevfusion-p150/cache of the host. A process with WORKER dispatch (11x10) compiles its own kernels.

Call: model(points, **kwargs)

Argument	Type	Description
points (first positional)	path, bytes + fmt=, (N, C) array / tensor, structured array, PointCloud, JSON envelope	one LiDAR cloud in base_link with x, y, z (see "Input types"); a column named time_lag marks Autoware's already densified 2-sweep cloud
sweeps	list of {"points", "time_lag_s", "T_current_from_sweep"}	client-side densification: the past sweeps, moved into the current frame with Autoware's float32 sweep arithmetic
stream	{"id", "timestamp_s", "T_world_from_ego", "reset"}	server-side densification exactly as Autoware's PointCloudDensification (densification_num_past_frames 1, per stream id, LRU of 16); a frame without its pose is refused and not cached (Autoware's TF failure)
score_threshold	float in [0, 1], default 0.1	every class and distance bin (Autoware's min_confidence_scores); a decoded score below it is dropped (<)
circle_nms_dist_threshold	float in [0, 10] m, default 0.5	class-agnostic circle NMS between the score-sorted proposals (<= 0 means 0.5 m, as the node)
iou_nms_threshold	float in [0, 1], default 0.1	Autoware's IoU-BEV NMS: suppress when the BEV IoU is above it
iou_nms_search_distance_2d	float in [0, 200] m, default 10	the IoU-BEV NMS search radius
remap_classes	bool, default True	Autoware's area class remapper (CAR to TRUCK for 12.1-36 m², CAR / TRUCK / BUS to TRAILER from 36 m²)
max_detections	int >= 1, default None (no cap, as Autoware)	keep the N best after NMS (not an Autoware knob)

Every knob is host-only (PLAN.md 2.6, RT-host): no device shape changes, and the defaults reproduce Autoware's bevfusion_lidar.param.yaml and the ml_package exactly (the yaw-norm thresholds and the distance bins are not knobs: the node's values, yaw-norm 0.3 for CAR, TRUCK, BUS, BICYCLE and 0 for PEDESTRIAN; distance bins 50 / 90 / 121 / 200 m). They are validated before the device runs (a bad value raises tt_bevfusion.io.InputError, HTTP 400). The device test test_stream_and_params checks stream= and non-default values against the CPU reference.

Input types

• Point clouds: a path (.bin raw float32 rows -- pass fields= if not x, y, z, intensity; .npy; .npz; .pcd ascii / binary / binary_compressed), raw bytes with fmt=, an (N, C) numpy array or torch tensor, a structured array, the JSON envelope of /predict, or tt_bevfusion.io.PointCloud. Frame: base_link, metres. An unnamed (N, 4) array is x, y, z, intensity; name the columns (structured array, PointCloud, fields=) for any other layout, e.g. x, y, z, time_lag.
• intensity is not a network input (use_intensity: false in the ml_package): a column is accepted and ignored.
• Three ways to give Autoware's 2-sweep input: a cloud with a time_lag column (already densified, used as is), sweeps= (densified on the host), or stream= (the node's own state machine). A plain cloud is one sweep.
• Rows with a non-finite x / y / z are dropped, and so are rows at exactly (0, 0, 0): the missing returns of organized clouds.
• Voxelization: 0.17 × 0.17 × 0.2 m voxels over ±122.4 m (x, y) and -3 to 5 m (z), at most 256,000 voxels (Autoware's clip; meta.voxel_overflow) and 10 points per voxel: the first 10 in input order (the training-time deterministic rule; Autoware keeps whichever 10 its GPU atomics leave).
• Images, calibration and planner inputs are refused (the lidar-only profile).

Output

model(...) returns a tt_bevfusion.Output (= ttaw.outputs.Detections3D); out.to_dict() is exactly the POST /predict body (SERVING.md section 3.2).

field	type	meaning
boxes	float32 [N, 7]	x, y, z (box centre), length, width, height, yaw in base_link metres / radians, the convention of Autoware's autoware_bevfusion node (yaw counter-clockwise from +x); sorted by score
scores	float32 [N]	Autoware's existence_probability (query_heat · σ(heatmap) of the winning class)
label_ids	int32 [N]	autoware_perception_msgs/ObjectClassification values (CAR 1, TRUCK 2, BUS 3, TRAILER 4, BICYCLE 6, PEDESTRIAN 7); labels holds the names
meta	dict	variant; points_in, points_in_range, num_voxels, voxels_total, voxel_overflow, points_dropped_by_cap, max_points_in_voxel; sparse (counts per level, pairs per neighbour map, bucket, overflow); skipped (no voxel: no inference, no objects, as the node); densify (mode, points); counts (proposals, after_threshold, after_circle_nms, after_iou_nms); per detection orientation_availability, label_before_remap and proposal (the top-k slot)
timing_ms	dict	preprocess, device (input packing + H2D + both traces + D2H + unpack), postprocess, total of this call

out.to_dicts() gives [{"label", "label_id", "score", "center", "size", "yaw"}]; len(out) the number of detections. No velocity is returned: Autoware decodes the network's velocity rows but does not publish them.

The last step of the head runs on the host in float32 after the one readback: score = σ(heatmap)[cls] · query_heat[cls], label = cls and center + bev_pos (this keeps the BEV cell positions out of bf16). Everything before it (sparse encoder, SECOND, SECONDFPN, heatmap, local max, top-512, the transformer decoder layer, the box heads) runs on the device.

Lifetime and information

• model.close() releases the traces and the device tensors and closes the chip if the model opened it; idempotent. with calls it for you; an unclosed model is closed when Python exits.
• model.info: weights (repo, tag, revision, path), device (dispatch, grid, cores, num_command_queues, fallback), variant, warm variants, warm-up times, runtime parameter defaults, the config, the pre-processing and the trace descriptions.
• Calls from several threads are safe and pipelined (BEVFUSION_PIPELINE=2, the default): host pre-processing, the device stage (staging, H2D, replays, D2H) and host post-processing each run under their own lock, so up to three frames overlap while the chip runs one at a time; every result equals a serial call's. One model per process per chip.
• tt_bevfusion.viz.render_bev(points, out.to_dicts(), reference=...) draws a bird's-eye view (PIL, no OpenCV).

Speed

Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, the shipped precision (code/scripts/bench.py, 20 iterations, two runs; the numbers of OPT_BASELINE.md "After optimization" and VERIFICATION_OPT_2026-10-12.md, 2026-10-12, on a shared host whose load moves the host stages, load average 4-5):

stage, p50	test.pcd (shipped, bucket s)	PandaSet 019 f40 (not shipped, bucket m)
model(path) end to end	32.5-32.6 ms	76.4-77.5 ms
host pre-processing (decode, densify, voxelize, VFE, the sorted level sets)	14.2 ms	48.5-49.0 ms
input packing (staged VFE rows, level keys) + H2D	0.64 + 0.51 ms	2.0-2.1 + 1.01 ms
device traces, one frame (sparse, incl. the device rulebooks, + dense)	3.6 + 12.5 ms	9.3 + 12.5 ms
D2H + unpack	0.55 + 0.21 ms	0.61 + 0.25 ms
host post-processing (decode, NMS, remap)	0.64 ms	1.08-1.11 ms

• Back-to-back frames take 15.99 ms (test.pcd) and 21.72 ms (PandaSet) of device time: 62.5 and 46.0 frames/s. A synchronous call gives 30.7 and 13.0 calls/s; three concurrent callers get a frame every 18.8-18.9 ms (test.pcd) and 32.2 ms (PandaSet), 53 and 31 frames/s.
• The first release (2026-10-09) took 657.1 / 2,786.0 ms per call (86-88 % of it a numpy rulebook build, now on the device) and 53.09 / 112.02 ms of device time per frame. The optimization steps are in OPT_REPORT.md.

Tests

From the repository root, with code/ on PYTHONPATH (or the package installed):

• Host (no device, no ttnn call): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevfusion/tests. Inside the tt-models workspace the suite runs on its fake ttnn (-p fake_ttnn_plugin), which also enables test_tt_host.py (the TT graph emulated on the CPU against the fp32 reference); tests that need workspace-only oracles or data skip elsewhere, and a missing sample FAILS inside the workspace.
• Device (one p150): python -m pytest -q -s code/tt_bevfusion/tests/test_pcc_device.py code/tt_bevfusion/tests/test_e2e_device.py (per-module PCC and the top-500 selection against the fp32 CPU reference on replay outputs, end-to-end agreement on the PandaSet sample, 7 development frames and 10 held-out frames, stream= and non-default parameters, replay == eager, API == /predict). The public-frame gates run only inside the tt-models workspace, where their data lives: the frames are PandaSet and nuScenes sweeps and Autoware's sample rosbag, which this repository does not ship.
• The gates are frozen in tests/*.gates.json at the first green run and are never loosened.

Limits

• Batch 1 on the chip; one model per process; at most 256,000 voxels (Autoware's clip) and 10 points per voxel (the first 10 in input order); the sparse encoder's three capacity buckets (a frame beyond full is refused, see "Warm-up").
• Outputs differ slightly from the fp32 reference: an object near one of Autoware's decision thresholds (score, yaw-norm gate, circle NMS, remap area) can appear, disappear or change label (README "Caveats" has the measured rates).
• One p150, ETH dispatch (12x10); dispatch="worker" is an A/B opt-in (11x10). Not a ROS 2 node. Lidar-only: the camera-lidar profile is not in this release.