code/PYTHON.md

Python API: BEVFormer-small (Autoware tensorrt_bevformer) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are the same.

Install

Install the package on top of an environment that already has ttnn (a tt-metal python_env at 44d66500520 with patches/tt-metal-eth-dispatch.patch, or the tt-model container). From the root of the model repository (the directory that holds pyproject.toml, README.md and code/):

pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests

The pip project is the repository's top-level pyproject.toml; it installs the package from code/tt_bevformer (there is no pyproject.toml inside code/, because the container build copies code/ over the tt-metal tree).

The package carries tt_bevformer.ttaw, the shared code of the Autoware ports to Blackhole (device open, trace runner, decoders, model base class, HTTP app), vendored at the version recorded in code/tt_bevformer/ttaw/VENDORED.json.

You want to run	Extras
the Python API	none
the HTTP server (tt_bevformer.server.app, see SERVING.md)	server
host tests (no device; device tests are skipped): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevformer/tests	server,test
device tests: python -m pytest -q -s code/tt_bevformer/tests/test_pcc_device.py code/tt_bevformer/tests/test_e2e_device.py	test

Quickstart

from tt_bevformer import BEVFormer, load_sample

with BEVFormer.from_pretrained(device_id=0) as model:
    out = model(**load_sample("code/tt_bevformer/samples/synthetic_6cam.json"))   # from the repo root
print(out.to_dict())

load_sample reads a sample manifest (six image files, a calibration or a preset, the stream with the ego pose) into the call's keyword arguments. The shipped sample is a synthetic test pattern (code/tt_bevformer/samples/README.md); examples/quickstart.py runs it and writes the result and a bird's-eye-view PNG.

BEVFormer.from_pretrained(...)

BEVFormer.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default changh95/bevformer-small-weights
    *,
    revision=None,          # default for the default repo: the validated commit 0e6d9d9c7b8 (branch main; no tags)
    variant=None,           # load-time variant (default); default $BEVFORMER_VARIANT or "default"
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_bevformer.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $BEVFORMER_DISPATCH or "eth"
    num_command_queues=None,  # default $BEVFORMER_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
    **compile_params,       # shape-defining options (documented per model below)
) -> BEVFormer

What it does: resolves the weights first, so a Hub problem never claims the chip (weights_dir > $BEVFORMER_WEIGHTS_DIR > a local model_id directory > the HF snapshot at the pinned revision, restricted to bevformer_small_epoch_24.safetensors, provenance.json, with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ; the other open parameters are DEVICE_DEFAULTS in tt_bevformer/device.py, overridable with BEVFORMER_*), builds the graph, then compiles and captures the metal trace of every warm-up variant. If ETH dispatch cannot open (tt-metal without the patch), it warns and falls back to WORKER dispatch.

Warm-up

from_pretrained captures the two trace variants of the frame (image: ResNet-101-DCN + FPN + the spatial cross-attention value maps; bev: BEV queries, history, 3 encoder layers, 6 decoder layers, heads), then runs one synthetic frame through the whole host + device path, so the first 1600×900 call compiles nothing. A camera size other than 1600×900 (e.g. 1920×1080) compiles its host resize kernel in a background thread on first sight (about 6 s); until it is ready those calls use the slower generic resize kernel. A new calibration pays its host geometry once (cached per lidar2img). There is one variant (default); the steady-state frame (use_prev_bev == 1) has its own bev trace. warmup_variants="none" skips the captures (the first call then fails: capture explicitly with model.warmup()). The first boot on a machine also pays the ttnn kernel JIT (minutes, cached in ~/.cache/tt-metal-cache afterwards); a warm load takes about 10 s (10.4 s measured on 2026-10-13: weights 0.3 s, graph 1.3 s, warm-up 4.5 s, the rest the device open). The warm-up frame uses the shipped synthetic_6cam calibration preset.

Precision: fast (default) and accurate

The load-time BEVFORMER_* knobs (read once by from_pretrained, from the environment or from keyword arguments with the knob's lower-case name, e.g. precision="HiFi4+fp32+l1acc:w=fp32") select the precision path. With no variable set you get the server's default profile fast (PLAN D20 reduced precision: HiFi2, bf16 weights, bfp8 layer-3 weights, a bfp8 residual stream in the backbone, flash-SDPA decoder self-attention; judged by task-level gates on nuScenes mini_val). For the full-precision accurate path (fp32 weights, HiFi4, the first release's policy; the frozen per-stage PCC gates pass) set the 13 variables of the accurate serve profile in tt-model.yaml (serve_profiles) before loading; SERVING.md section 3.4 lists them. model.info reports the knob values in use.

Call: model(...)

Argument	Type	Description
images	the six cameras	CAM_FRONT, CAM_FRONT_RIGHT, CAM_FRONT_LEFT, CAM_BACK, CAM_BACK_LEFT, CAM_BACK_RIGHT (any order; the network order is baked into its camera embeddings): a list of tt_bevformer.io.CameraImage / {"camera", "image" | "path" | "data", "intrinsics", "T_ref_from_camera"} dicts or a {name: image} mapping. Any size: images other than 1600x900 are resized like the node does (uint8 bilinear)
calibration	dict	per camera intrinsics (3x3, of the image as sent) and T_ref_from_camera (camera optical frame -> frame_id), or {"preset": "synthetic_6cam"} (the shipped rig); frame_id: "base_link" (default; then T_ref_from_lidar = the network's LIDAR_TOP -> base_link, default a virtual nuScenes LIDAR_TOP: -90 deg yaw, 1.84 m up) or "LIDAR_TOP"; scale_intrinsics (default true: K rescaled to 1600x900; false: the node's quirk)
stream	dict	{"id", "reset", "timestamp_s", "T_world_from_ego"}: the BEV history. A new id, reset: true or a timestamp gap above BEVFORMER_MAX_GAP_S (2 s) starts a fresh history (use_prev_bev = 0); T_world_from_ego (4x4, default identity) drives the ego-motion compensation (can_bus position / yaw deltas, TSA shift, prev_bev rotation). Up to BEVFORMER_MAX_STREAMS (4) ids keep their own device history (least recently used evicted)
ego	dict	optional CAN dynamics in base_link: {"accel": [3], "rotation_rate": [3], "velocity": [3]} (missing = 0; BEVFormer's training layout of can_bus[7:16])
score_threshold	float [0, 1]	the node's score_thre (default 0.2)
max_num	int [1, 9000]	top-k over the 900 x 10 scores before the threshold (default 300)
physical_boxes	bool	false (default): the node's message (size [w, l, h], yaw -rot + pi); true: size [l, w, h], yaw -rot - pi/2 (the heading of the length axis)

Input types

• Images: a path, PNG/JPEG bytes, a PIL.Image, a uint8 HxWx3 array or 3xHxW tensor, or a float array in [0, 1].
• Calibration: a dict in the /predict calibration envelope (SERVING.md section 3.1) or {"preset": name} (code/tt_bevformer/calib/); transforms as 4×4 matrices, {translation, rotation_wxyz} or Autoware {x, y, z, roll, pitch, yaw}.

Output

tt_bevformer.Output = ttaw.outputs.Detections3D in the node's frame LIDAR_TOP (frame_id), rows sorted by score: boxes [N, 7] (x, y, z gravity centre, size, yaw per physical_boxes), scores, label_ids (network class 0-9: meta["class_names"]), labels (the Autoware label the node publishes per class: bus, trailer and construction_vehicle -> TRUCK; barrier, traffic_cone -> UNKNOWN), velocities [N, 2] (vx, vy in LIDAR_TOP), timing_ms (preprocess, device, device_run, postprocess, total), meta (class_names, the box convention, the raw network angle rot and the decoder query of every box, stream = {id, fresh, use_prev_bev}). out.to_dict() is exactly the /predict body.

Lifetime and information

• model.close() releases the traces and the device tensors and closes the chip if the model opened it; idempotent. with calls it for you; an unclosed model is closed when Python exits.
• model.info: weights (repo, tag, revision, path), device (dispatch, grid), variant, warm variants, warm-up times, runtime parameter defaults.
• Calls from several threads are safe. With BEVFORMER_PIPELINE=1 (default) one caller's host half overlaps another's device stage, frames keep their arrival order; two callers on one stream measured 16.3 frames/s (1600×900) and 18.1-18.5 frames/s (PandaSet). One model per process per chip.

Speed

Measured with code/scripts/bench.py (ETH dispatch, 1 CQ, 12×10 grid, AICLK 1350 MHz, warm, batch 1; p50 of 30 iterations on a shared 8-core host; 2026-10-13, VERIFICATION_OPT_2026-10-13.md, OPT_BASELINE.md "After optimization"):

	first release	fast (default)	accurate
model(...) end to end, 1600×900 images (synthetic)	939 ms	59.0 ms	88.9 ms
model(...) end to end, PandaSet 1920×1080 (decoded before timing): one frame / the f30-35-40 stream	1,555 / 1,619 ms	71.0 / 73.6 ms	101.9 / 104.2 ms
host pre-processing (request, the node's uint8 resize for 1920×1080, staging)	303 / 919 ms	3.4 / 12-15 ms	3.5 / 15 ms
device incl. the uint8 image upload and the read-back (timing_ms.device, 1600×900)	633 ms	54.1 ms	84.1 ms
trace image (pre-processing on the device, ResNet-101-DCN + FPN + SCA value maps) / trace bev (3 encoder + 6 decoder layers + heads), back to back	245.0 / 240.5 ms	34.9 / 14.5 ms	60.9 / 18.7 ms
host post-processing	1.4 ms	1.4 ms	1.4 ms

Back to back, the two traces take 49.3 ms per frame (20.3 frames/s device-bound; first release 485.6 ms) with fast, 79.6 ms with accurate. BEVFORMER_NUM_CQS=2 is an A/B option (about +0.9 ms of device time; no gain for one request at a time). The many other BEVFORMER_* knobs are A/B switches of the optimization rounds (OPT_REPORT.md); the card's numbers use the defaults.

Limits

• Batch 1 on the chip; one model per process; six cameras in the nuScenes slot order (other rigs need a virtual LIDAR_TOP and rescaled K, see calibration).
• A rig whose cameras each see more than 6,144 BEV cells is refused with an InputError (HTTP 400): the static rebatching capacity of the spatial cross-attention (the node's dense graph accepts any rig; every rig tested fits).
• History state for BEVFORMER_MAX_STREAMS (4) stream ids; the history advances once per call (the node's cadence) and restarts on a new id, reset or a timestamp gap above BEVFORMER_MAX_GAP_S (2 s).
• Output in the node's LIDAR_TOP frame only; convert with the calibration's T_ref_from_lidar (SERVING.md 3.2).