Python API: TransFusion-L (Autoware lidar_transfusion) on Blackhole
Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are the same.
Install
Install the package on top of an environment that already has ttnn (a tt-metal python_env at 44d66500520
with patches/tt-metal-eth-dispatch.patch, or the tt-model container). From the root of the model repository (the
directory that holds pyproject.toml, README.md and code/):
pip install -e . # the Python API
pip install -e ".[server,test]" # + the HTTP server and the tests
The pip project is the repository's top-level pyproject.toml; it installs the package from code/tt_transfusion (there is no
pyproject.toml inside code/, because the container build copies code/ over the tt-metal tree).
The package carries tt_transfusion.ttaw, the shared code of the Autoware ports to Blackhole (device open, trace runner,
decoders, model base class, HTTP app), vendored at the version recorded in code/tt_transfusion/ttaw/VENDORED.json.
| You want to run | Extras |
|---|---|
| the Python API | none |
the HTTP server (tt_transfusion.server.app, see SERVING.md) |
server |
host tests (no device; device tests are skipped): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_transfusion/tests |
server,test |
device tests: python -m pytest -q -s code/tt_transfusion/tests/test_pcc_device.py code/tt_transfusion/tests/test_e2e_device.py code/tt_transfusion/tests/test_heldout_device.py |
test |
Quickstart
from tt_transfusion import TransFusion
with TransFusion.from_pretrained(device_id=0) as model:
out = model("code/tt_transfusion/samples/test_pcd.npz") # Autoware's test.pcd, Apache-2.0
print(out.to_dict())
examples/quickstart.py runs this snippet on the shipped sample and also writes quickstart_bev.png (the points and
the boxes from above, tt_transfusion.viz.render_bev).
TransFusion.from_pretrained(...)
TransFusion.from_pretrained(
model_id=None, # HF repo or a local directory with the weights files; default AutowareFoundation/lidar_transfusion
*,
revision=None, # default for the default repo: the validated commit 8f07eb93038 (tag v2.1)
variant=None, # one network: "default" (also $TRANSFUSION_VARIANT); sampling is a per-call input, not a variant
device_id=None, # chip to open; default $TT_DEVICE_ID or 0
device=None, # an already-opened ttnn device (tt_transfusion.device.open_device); close() does not close it
dispatch=None, # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $TRANSFUSION_DISPATCH or "eth"
num_command_queues=None, # default $TRANSFUSION_NUM_CQS or 1
weights_dir=None, # explicit local weights directory; no Hub access
warmup_variants="default", # trace variants to capture now; see "Warm-up"
verbose=False,
**compile_params, # none for this model: the device shapes are fixed (pillar buckets up to 65,536, 768x768 grid, K = 500)
) -> TransFusion
What it does: resolves the weights first, so a Hub problem never claims the chip (weights_dir >
$TRANSFUSION_WEIGHTS_DIR > a local model_id directory > the HF snapshot at the pinned revision, restricted to
transfusion.onnx, transfusion_ml_package.param.yaml, detection_class_remapper.param.yaml, deploy_metadata.yaml, with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ;
the other open parameters are DEVICE_DEFAULTS in tt_transfusion/device.py, overridable with TRANSFUSION_*), builds the
graph, then compiles and captures the metal trace of every warm-up variant. If ETH dispatch cannot open (tt-metal
without the patch), it warns and falls back to WORKER dispatch.
Warm-up
from_pretrained returns a warm model: it builds the graph, runs every trace variant once eagerly (this first run
compiles the kernels and uploads the weights) and captures one metal trace per variant, so no call compiles anything.
The variants are pillar-capacity buckets (TRANSFUSION_PFN_BUCKETS, default 8,192, 12,288, 16,384, 20,480, 24,576,
32,768, 40,960, 49,152, plus the full 65,536): a frame runs the trace of the smallest capacity that holds its pillars,
with a 768x768 canvas index. The 9 traces share the device weights, and every bucket gives the same outputs as the
full 65,536-pillar trace (checked bit for bit on the device on two frames by test_buckets_equal_full_capacity).
- Measured (2026-10-11, the shipped sample, ETH dispatch, 1 CQ, 12x10): with an empty JIT cache the load took 156.7 s (firmware and every kernel compiled); with that cache warm, in a new process, 17.7 s (weights 0.3 s, graph build 9.8 s including the compile of the numba host twins, eager warm-up and capture of the 9 traces 3.4 s, the rest the device open). The traces are captured during the load, so the first call costs about as much as the next ones (41.3 / 38.5 ms, first / second call, on a shared host).
warmup_variants="none"(server:TRANSFUSION_WARMUP=none) skips the capture at load; callmodel.warmup()before the first frame (it is idempotent).- The JIT cache is tt-metal's (
~/.cache/tt-metal-cache, orTT_METAL_CACHE); inside the container it is~/.cache/tt-model/transfusion-p150/cacheof the host. A process with WORKER dispatch (11x10) compiles its own kernels.
Call: model(points, **kwargs)
| Argument | Type | Description |
|---|---|---|
points (first positional) |
path, bytes + fmt=, (N, C) array / tensor, structured array, PointCloud, JSON envelope |
one LiDAR cloud in base_link with x, y, z, intensity (see "Input types"); a column named time_lag marks Autoware's already densified 2-sweep cloud |
sweeps |
list of {"points", "time_lag_s", "T_current_from_sweep"} |
client-side densification: the past sweeps, moved into the current frame with Autoware's float32 sweep arithmetic |
stream |
{"id", "timestamp_s", "T_world_from_ego", "reset"} |
server-side densification exactly as Autoware's PointCloudDensification (densification_num_past_frames 1, per stream id, LRU of 16); a frame without its pose is refused and not cached (Autoware's TF failure) |
sampling |
"clean" (default) | "autoware_compat" | {"mode": "autoware_compat", "offset": int} |
how the points of a pillar are chosen: clean keeps the first 20 points of each pillar in input order (deterministic); autoware_compat emulates the deployed node's capacity-wide shuffle (a fixed permutation of 2,000,000 slots, seed 0, starting at offset) and its skips of frames with fewer than 5,000 or more than 60,000 pillars (no inference, no objects, meta.skipped) |
score_threshold |
float in [0, 1], default 0.1 | strict > on the decoded score; <= 0 means 0.2, as the node's config |
circle_nms_dist_threshold |
float in [0, 10] m, default 0.5 | class-agnostic circle NMS between the score-sorted proposals (<= 0 means 0.5 m, as the node) |
iou_nms_threshold |
float in [0, 1], default 0.1 | Autoware's IoU-BEV NMS: suppress when the BEV IoU is above it |
iou_nms_search_distance_2d |
float in [0, 200] m, default 10 | the IoU-BEV NMS search radius |
remap_classes |
bool, default True |
Autoware's area class remapper (CAR to TRUCK for 12.1-36 m², CAR / TRUCK / BUS to TRAILER from 36 m²) |
max_detections |
int >= 1, default None (no cap, as Autoware) |
keep the N best after NMS (not an Autoware knob) |
Every knob is host-only (PLAN.md 2.5, RT-host): no device shape changes, and the defaults reproduce Autoware's
transfusion.param.yaml and the ml_package exactly (the yaw-norm thresholds are not a knob: the node's values, one per
class, 0.3 for CAR, TRUCK, BUS, BICYCLE and 0 for PEDESTRIAN). They are validated before the device runs (a bad value raises
tt_transfusion.io.InputError, HTTP 400).
Input types
- Point clouds: a path (
.binraw float32 rows -- passfields=if notx, y, z, intensity;.npy;.npz;.pcdascii / binary / binary_compressed), raw bytes withfmt=, an (N, C) numpy array or torch tensor, a structured array, the JSON envelope of/predict, ortt_transfusion.io.PointCloud. Frame:base_link, metres. An unnamed (N, 4) array is x, y, z, intensity; name the columns (structured array,PointCloud,fields=) for any other layout, e.g. x, y, z, intensity, time_lag. intensityis a network input here (unlike CenterPoint): Autoware's uint8 value, 0-255. Float values are clipped and truncated like a uint8 cast (meta.intensity_changedcounts the values that changed); a cloud without anintensityfield is refused.- Three ways to give Autoware's 2-sweep input: a cloud with a
time_lagcolumn (already densified, used as is),sweeps=(densified on the host), orstream=(the node's own state machine). A plain cloud is one sweep. - Rows with a non-finite x / y / z are dropped, and so are rows at exactly (0, 0, 0): the missing returns of organized clouds (Autoware's voxelizer would keep such a point).
- At most 65,536 pillars (a fixed device capacity above Autoware's 60,000; the densest frame seen so far, a 2-sweep
frame of Autoware's sample rosbag, has 62,527). Above it the first 65,536 pillars in raster order are kept and
meta["pillar_overflow"]is true. 20 points per pillar (seesampling). - Images, calibration and planner inputs are refused (a LiDAR-only model).
Output
model(...) returns a tt_transfusion.Output (= ttaw.outputs.Detections3D); out.to_dict() is exactly the
POST /predict body (SERVING.md section 3.2).
| field | type | meaning |
|---|---|---|
boxes |
float32 [N, 7] |
x, y, z (box centre), length, width, height, yaw in base_link metres / radians, the convention of Autoware's lidar_transfusion node (yaw counter-clockwise from +x, no conversion); sorted by score |
scores |
float32 [N] |
Autoware's existence_probability (query_heat · σ(heatmap) of the winning class) |
label_ids |
int32 [N] |
autoware_perception_msgs/ObjectClassification values (CAR 1, TRUCK 2, BUS 3, TRAILER 4, BICYCLE 6, PEDESTRIAN 7); labels holds the names |
meta |
dict | variant; num_points, num_points_in_range, num_pillars, num_pillars_total, max_points_in_pillar, pillars_over_cap, pillar_overflow; sampling (mode, offset, the voxelizer's input size); skipped (autoware_compat: Autoware's reason to skip the frame) and autoware_would_skip (the same verdict, reported in clean mode); intensity_changed; densify (mode, points per sweep); counts (proposals, after_threshold, after_circle_nms, after_iou_nms); per detection orientation_availability, label_before_remap and proposal (the top-k slot) |
timing_ms |
dict | preprocess, device (staging + H2D + trace + D2H + unpack), postprocess, total of this call |
out.to_dicts() gives [{"label", "label_id", "score", "center", "size", "yaw"}]; len(out) the number of
detections. No velocity is returned: Autoware ignores the network's velocity rows.
Lifetime and information
model.close()releases the traces and the device tensors and closes the chip if the model opened it; idempotent.withcalls it for you; an unclosed model is closed when Python exits.model.info: weights (repo, tag, revision, path), device (dispatch,grid,cores,num_command_queues,fallback), variant, warm variants, warm-up times, runtime parameter defaults, the config, the pre-processing and the trace descriptions.- Calls from several threads are safe: the device calls are serialised by a lock. One model per process per chip.
tt_transfusion.viz.render_bev(points, out.to_dicts(), reference=...)draws a bird's-eye view (PIL, no OpenCV).
Speed
Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, the shipped precision and knob defaults (code/scripts/bench.py,
100 iterations; the independent re-measurement of VERIFICATION_OPT_2026-10-11.md, 2026-10-11, on a shared host
whose load moves the host stages):
| stage, p50 | test.pcd (shipped) | PandaSet 019 f40 (not shipped) |
|---|---|---|
model(path) end to end |
36.0 ms | 71.0 ms |
| host pre-processing (decode, densify, voxelize, decorate, index) | 10.7 ms | 36.5 ms |
| input staging (the kept pillars to bf16, pair-tiled) + H2D | 2.7 + 3.5 ms | 6.2 + 6.9 ms |
| device trace, one forward (pillar bucket) | 17.23 ms (16,384) | 18.11 ms (40,960) |
| D2H + unpack | 0.63 + 0.28 ms | 0.63 + 0.29 ms |
| host post-processing (decode, NMS, remap) | 0.73 ms | 0.94 ms |
- Back-to-back replays take 17.16 ms per frame on test.pcd (58.3 frames/s of device throughput) and 18.04 ms on the PandaSet frame (55.4 frames/s). A synchronous call gives 27.8 (test.pcd) and 14.1 (PandaSet) calls/s.
- test.pcd is one sweep of 48,048 points (13,058 pillars); the PandaSet frame two sweeps of 213,007 points (38,158 pillars). The device time grows a little with the pillar bucket; the host stages grow with the cloud.
- The first release (2026-10-09) took 139.39 ms per back-to-back replay and 211.8 ms per
model()call on test.pcd.OPT_REPORT.mdhas every optimization step with its A/B measurement: among them a two-term pillar feature net and scatter as one custom kernel (a megakernel per pillar bucket), custom epilogue kernels for the two-term convs, the pillar buckets and compiled host stages. All but one step give bit-identical outputs; the two-stage top-k picks a different query on a tie and passes every frozen accuracy gate.
Tests
From the repository root, with code/ on PYTHONPATH (or the package installed):
- Host (no device, no ttnn call):
TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_transfusion/tests. Inside the tt-models workspace the suite runs on its fake ttnn (-p fake_ttnn_plugin), which also enablestest_tt_host.py(the TT graph emulated on the CPU against the fp32 reference); tests that need workspace-only oracles or data skip elsewhere, and a missing sample FAILS inside the workspace. - Device (one p150):
python -m pytest -q -s code/tt_transfusion/tests/test_pcc_device.py code/tt_transfusion/tests/test_e2e_device.py code/tt_transfusion/tests/test_heldout_device.py(per-module PCC and the strong-proposal selection against the fp32 CPU reference on replay outputs, end-to-end agreement on 21 development frames and 34 held-out frames, replay == eager, API ==/predict). The public-frame gates run only inside the tt-models workspace, where their data lives: the frames are PandaSet, nuScenes and Argoverse 2 sweeps and Autoware's sample rosbag, which this repository does not ship. - The gates are frozen in
tests/*.gates.jsonat the first green run and are never loosened.
Limits
- Batch 1 on the chip; one model per process; at most 65,536 pillars (a grid-independent capacity above Autoware's 60,000, so no clean-sampled frame seen so far is cut; overflow keeps the first pillars in raster order and is flagged) and 20 points per pillar (the first 20 in input order, instead of Autoware's GPU-atomics order).
- Outputs differ slightly from the fp32 reference: an object near one of Autoware's decision thresholds (score, yaw-norm gate, NMS, remap area) or at a heatmap near-tie can appear, disappear or come from a neighbouring proposal (README "Caveats" has the measured rates).
- One p150, ETH dispatch (12x10);
dispatch="worker"is an A/B opt-in (11x10). Not a ROS 2 node.