# Python API: TransFusion-L (Autoware lidar_transfusion) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the
API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are
the same.

## Install

Install the package on top of an environment that already has `ttnn` (a tt-metal `python_env` at `44d66500520`
with `patches/tt-metal-eth-dispatch.patch`, or the tt-model container). From the root of the model repository (the
directory that holds `pyproject.toml`, `README.md` and `code/`):

```bash
pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests
```

The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_transfusion` (there is no
`pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree).

The package carries `tt_transfusion.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner,
decoders, model base class, HTTP app), vendored at the version recorded in `code/tt_transfusion/ttaw/VENDORED.json`.

| You want to run | Extras |
|---|---|
| the Python API | none |
| the HTTP server (`tt_transfusion.server.app`, see `SERVING.md`) | `server` |
| host tests (no device; device tests are skipped): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_transfusion/tests` | `server,test` |
| device tests: `python -m pytest -q -s code/tt_transfusion/tests/test_pcc_device.py code/tt_transfusion/tests/test_e2e_device.py code/tt_transfusion/tests/test_heldout_device.py` | `test` |

## Quickstart

```python
from tt_transfusion import TransFusion

with TransFusion.from_pretrained(device_id=0) as model:
    out = model("code/tt_transfusion/samples/test_pcd.npz")       # Autoware's test.pcd, Apache-2.0
print(out.to_dict())
```

`examples/quickstart.py` runs this snippet on the shipped sample and also writes `quickstart_bev.png` (the points and
the boxes from above, `tt_transfusion.viz.render_bev`).

## `TransFusion.from_pretrained(...)`

```python
TransFusion.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default AutowareFoundation/lidar_transfusion
    *,
    revision=None,          # default for the default repo: the validated commit 8f07eb93038 (tag v2.1)
    variant=None,           # one network: "default" (also $TRANSFUSION_VARIANT); sampling is a per-call input, not a variant
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_transfusion.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $TRANSFUSION_DISPATCH or "eth"
    num_command_queues=None,  # default $TRANSFUSION_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
    **compile_params,       # none for this model: the device shapes are fixed (pillar buckets up to 65,536, 768x768 grid, K = 500)
) -> TransFusion
```

What it does: resolves the weights first, so a Hub problem never claims the chip (`weights_dir` >
`$TRANSFUSION_WEIGHTS_DIR` > a local `model_id` directory > the HF snapshot at the pinned revision, restricted to
`transfusion.onnx, transfusion_ml_package.param.yaml, detection_class_remapper.param.yaml, deploy_metadata.yaml`, with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ;
the other open parameters are `DEVICE_DEFAULTS` in `tt_transfusion/device.py`, overridable with `TRANSFUSION_*`), builds the
graph, then compiles and captures the metal trace of every warm-up variant. If ETH dispatch cannot open (tt-metal
without the patch), it warns and falls back to WORKER dispatch.

## Warm-up

`from_pretrained` returns a warm model: it builds the graph, runs every trace variant once eagerly (this first run
compiles the kernels and uploads the weights) and captures one metal trace per variant, so no call compiles anything.
The variants are pillar-capacity buckets (`TRANSFUSION_PFN_BUCKETS`, default 8,192, 12,288, 16,384, 20,480, 24,576,
32,768, 40,960, 49,152, plus the full 65,536): a frame runs the trace of the smallest capacity that holds its pillars,
with a 768x768 canvas index. The 9 traces share the device weights, and every bucket gives the same outputs as the
full 65,536-pillar trace (checked bit for bit on the device on two frames by `test_buckets_equal_full_capacity`).

- Measured (2026-10-11, the shipped sample, ETH dispatch, 1 CQ, 12x10): with an empty JIT cache the load took 156.7 s
  (firmware and every kernel compiled); with that cache warm, in a new process, 17.7 s (weights 0.3 s, graph build
  9.8 s including the compile of the numba host twins, eager warm-up and capture of the 9 traces 3.4 s, the rest the
  device open). The traces are captured during the load, so the first call costs about as much as the next ones
  (41.3 / 38.5 ms, first / second call, on a shared host).
- `warmup_variants="none"` (server: `TRANSFUSION_WARMUP=none`) skips the capture at load; call `model.warmup()` before
  the first frame (it is idempotent).
- The JIT cache is tt-metal's (`~/.cache/tt-metal-cache`, or `TT_METAL_CACHE`); inside the container it is
  `~/.cache/tt-model/transfusion-p150/cache` of the host. A process with WORKER dispatch (11x10) compiles its own
  kernels.

## Call: `model(points, **kwargs)`

| Argument | Type | Description |
|---|---|---|
| `points` (first positional) | path, `bytes` + `fmt=`, (N, C) array / tensor, structured array, `PointCloud`, JSON envelope | one LiDAR cloud in `base_link` with x, y, z, intensity (see "Input types"); a column named `time_lag` marks Autoware's already densified 2-sweep cloud |
| `sweeps` | list of `{"points", "time_lag_s", "T_current_from_sweep"}` | client-side densification: the past sweeps, moved into the current frame with Autoware's float32 sweep arithmetic |
| `stream` | `{"id", "timestamp_s", "T_world_from_ego", "reset"}` | server-side densification exactly as Autoware's `PointCloudDensification` (`densification_num_past_frames` 1, per stream id, LRU of 16); a frame without its pose is refused and not cached (Autoware's TF failure) |
| `sampling` | `"clean"` (default) \| `"autoware_compat"` \| `{"mode": "autoware_compat", "offset": int}` | how the points of a pillar are chosen: `clean` keeps the first 20 points of each pillar in input order (deterministic); `autoware_compat` emulates the deployed node's capacity-wide shuffle (a fixed permutation of 2,000,000 slots, seed 0, starting at `offset`) and its skips of frames with fewer than 5,000 or more than 60,000 pillars (no inference, no objects, `meta.skipped`) |
| `score_threshold` | float in [0, 1], default 0.1 | strict `>` on the decoded score; <= 0 means 0.2, as the node's config |
| `circle_nms_dist_threshold` | float in [0, 10] m, default 0.5 | class-agnostic circle NMS between the score-sorted proposals (<= 0 means 0.5 m, as the node) |
| `iou_nms_threshold` | float in [0, 1], default 0.1 | Autoware's IoU-BEV NMS: suppress when the BEV IoU is above it |
| `iou_nms_search_distance_2d` | float in [0, 200] m, default 10 | the IoU-BEV NMS search radius |
| `remap_classes` | bool, default `True` | Autoware's area class remapper (CAR to TRUCK for 12.1-36 m², CAR / TRUCK / BUS to TRAILER from 36 m²) |
| `max_detections` | int >= 1, default `None` (no cap, as Autoware) | keep the N best after NMS (not an Autoware knob) |

Every knob is host-only (PLAN.md 2.5, RT-host): no device shape changes, and the defaults reproduce Autoware's
`transfusion.param.yaml` and the ml_package exactly (the yaw-norm thresholds are not a knob: the node's values, one per
class, 0.3 for CAR, TRUCK, BUS, BICYCLE and 0 for PEDESTRIAN). They are validated before the device runs (a bad value raises
`tt_transfusion.io.InputError`, HTTP 400).

### Input types

- Point clouds: a path (`.bin` raw float32 rows -- pass `fields=` if not `x, y, z, intensity`; `.npy`;
  `.npz`; `.pcd` ascii / binary / binary_compressed), raw bytes with `fmt=`, an (N, C) numpy array or torch tensor,
  a structured array, the JSON envelope of `/predict`, or `tt_transfusion.io.PointCloud`. Frame: `base_link`, metres.
  An unnamed (N, 4) array is x, y, z, intensity; name the columns (structured array, `PointCloud`, `fields=`) for any
  other layout, e.g. x, y, z, intensity, time_lag.
- `intensity` is a network input here (unlike CenterPoint): Autoware's uint8 value, 0-255. Float values are clipped
  and truncated like a uint8 cast (`meta.intensity_changed` counts the values that changed); a cloud without an
  `intensity` field is refused.
- Three ways to give Autoware's 2-sweep input: a cloud with a `time_lag` column (already densified, used as is),
  `sweeps=` (densified on the host), or `stream=` (the node's own state machine). A plain cloud is one sweep.
- Rows with a non-finite x / y / z are dropped, and so are rows at exactly (0, 0, 0): the missing returns of organized
  clouds (Autoware's voxelizer would keep such a point).
- At most 65,536 pillars (a fixed device capacity above Autoware's 60,000; the densest frame seen so far, a 2-sweep
  frame of Autoware's sample rosbag, has 62,527). Above it the first 65,536 pillars in raster order are kept and
  `meta["pillar_overflow"]` is true. 20 points per pillar (see `sampling`).
- Images, calibration and planner inputs are refused (a LiDAR-only model).

## Output

`model(...)` returns a `tt_transfusion.Output` (= `ttaw.outputs.Detections3D`); `out.to_dict()` is exactly the
`POST /predict` body (SERVING.md section 3.2).

| field | type | meaning |
|---|---|---|
| `boxes` | float32 `[N, 7]` | x, y, z (box centre), length, width, height, yaw in `base_link` metres / radians, the convention of Autoware's lidar_transfusion node (yaw counter-clockwise from +x, no conversion); sorted by score |
| `scores` | float32 `[N]` | Autoware's `existence_probability` (`query_heat · σ(heatmap)` of the winning class) |
| `label_ids` | int32 `[N]` | `autoware_perception_msgs/ObjectClassification` values (CAR 1, TRUCK 2, BUS 3, TRAILER 4, BICYCLE 6, PEDESTRIAN 7); `labels` holds the names |
| `meta` | dict | `variant`; `num_points`, `num_points_in_range`, `num_pillars`, `num_pillars_total`, `max_points_in_pillar`, `pillars_over_cap`, `pillar_overflow`; `sampling` (mode, offset, the voxelizer's input size); `skipped` (`autoware_compat`: Autoware's reason to skip the frame) and `autoware_would_skip` (the same verdict, reported in `clean` mode); `intensity_changed`; `densify` (mode, points per sweep); `counts` (`proposals`, `after_threshold`, `after_circle_nms`, `after_iou_nms`); per detection `orientation_availability`, `label_before_remap` and `proposal` (the top-k slot) |
| `timing_ms` | dict | `preprocess`, `device` (staging + H2D + trace + D2H + unpack), `postprocess`, `total` of this call |

`out.to_dicts()` gives `[{"label", "label_id", "score", "center", "size", "yaw"}]`; `len(out)` the number of
detections. No velocity is returned: Autoware ignores the network's velocity rows.

## Lifetime and information

- `model.close()` releases the traces and the device tensors and closes the chip if the model opened it; idempotent.
  `with` calls it for you; an unclosed model is closed when Python exits.
- `model.info`: weights (repo, tag, revision, path), device (`dispatch`, `grid`, `cores`, `num_command_queues`,
  `fallback`), variant, warm variants, warm-up times, runtime parameter defaults, the config, the pre-processing and
  the trace descriptions.
- Calls from several threads are safe: the device calls are serialised by a lock. One model per process per chip.
- `tt_transfusion.viz.render_bev(points, out.to_dicts(), reference=...)` draws a bird's-eye view (PIL, no OpenCV).

## Speed

Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, the shipped precision and knob defaults (`code/scripts/bench.py`,
100 iterations; the independent re-measurement of `VERIFICATION_OPT_2026-10-11.md`, 2026-10-11, on a shared host
whose load moves the host stages):

| stage, p50 | test.pcd (shipped) | PandaSet 019 f40 (not shipped) |
|---|---:|---:|
| `model(path)` end to end | 36.0 ms | 71.0 ms |
| host pre-processing (decode, densify, voxelize, decorate, index) | 10.7 ms | 36.5 ms |
| input staging (the kept pillars to bf16, pair-tiled) + H2D | 2.7 + 3.5 ms | 6.2 + 6.9 ms |
| device trace, one forward (pillar bucket) | 17.23 ms (16,384) | 18.11 ms (40,960) |
| D2H + unpack | 0.63 + 0.28 ms | 0.63 + 0.29 ms |
| host post-processing (decode, NMS, remap) | 0.73 ms | 0.94 ms |

- Back-to-back replays take 17.16 ms per frame on test.pcd (58.3 frames/s of device throughput) and 18.04 ms on the
  PandaSet frame (55.4 frames/s). A synchronous call gives 27.8 (test.pcd) and 14.1 (PandaSet) calls/s.
- test.pcd is one sweep of 48,048 points (13,058 pillars); the PandaSet frame two sweeps of 213,007 points (38,158
  pillars). The device time grows a little with the pillar bucket; the host stages grow with the cloud.
- The first release (2026-10-09) took 139.39 ms per back-to-back replay and 211.8 ms per `model()` call on test.pcd.
  `OPT_REPORT.md` has every optimization step with its A/B measurement: among them a two-term pillar feature net and
  scatter as one custom kernel (a megakernel per pillar bucket), custom epilogue kernels for the two-term convs, the
  pillar buckets and compiled host stages. All but one step give bit-identical outputs; the two-stage top-k picks a
  different query on a tie and passes every frozen accuracy gate.

## Tests

From the repository root, with `code/` on `PYTHONPATH` (or the package installed):

- Host (no device, no ttnn call): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_transfusion/tests`. Inside the
  tt-models workspace the suite runs on its fake ttnn (`-p fake_ttnn_plugin`), which also enables `test_tt_host.py`
  (the TT graph emulated on the CPU against the fp32 reference); tests that need workspace-only oracles or data skip
  elsewhere, and a missing sample FAILS inside the workspace.
- Device (one p150): `python -m pytest -q -s code/tt_transfusion/tests/test_pcc_device.py
  code/tt_transfusion/tests/test_e2e_device.py code/tt_transfusion/tests/test_heldout_device.py` (per-module PCC and
  the strong-proposal selection against the fp32 CPU reference on replay outputs, end-to-end agreement on 21
  development frames and 34 held-out frames, replay == eager, API == `/predict`). The public-frame gates run only
  inside the tt-models workspace, where their data lives: the frames are PandaSet, nuScenes and Argoverse 2 sweeps and
  Autoware's sample rosbag, which this repository does not ship.
- The gates are frozen in `tests/*.gates.json` at the first green run and are never loosened.

## Limits

- Batch 1 on the chip; one model per process; at most 65,536 pillars (a grid-independent capacity above Autoware's
  60,000, so no clean-sampled frame seen so far is cut; overflow keeps the first pillars in raster order and is
  flagged) and 20 points per pillar (the first 20 in input order, instead of Autoware's GPU-atomics order).
- Outputs differ slightly from the fp32 reference: an object near one of Autoware's decision thresholds (score,
  yaw-norm gate, NMS, remap area) or at a heatmap near-tie can appear, disappear or come from a neighbouring proposal
  (README "Caveats" has the measured rates).
- One p150, ETH dispatch (12x10); `dispatch="worker"` is an A/B opt-in (11x10). Not a ROS 2 node.