# Python API: Point Transformer V3 (Autoware autoware_ptv3) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the
API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are
the same.

## Install

Install the package on top of an environment that already has `ttnn` (a tt-metal `python_env` at `44d66500520`
with `patches/tt-metal-eth-dispatch.patch`, or the tt-model container). From the root of the model repository (the
directory that holds `pyproject.toml`, `README.md` and `code/`):

```bash
pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests
```

The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_ptv3` (there is no
`pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree).

The package carries `tt_ptv3.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner,
decoders, model base class, HTTP app), vendored at the version recorded in `code/tt_ptv3/ttaw/VENDORED.json`.

| You want to run | Extras |
|---|---|
| the Python API | none |
| the HTTP server (`tt_ptv3.server.app`, see `SERVING.md`) | `server` |
| host tests (no device; device tests are skipped): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_ptv3/tests/test_bundle_host.py code/tt_ptv3/tests/test_host_units.py code/tt_ptv3/tests/test_io_host.py code/tt_ptv3/tests/test_server_host.py` (and the other `test_*_host.py` / reference files; see "Tests") | `server,test` |
| device tests: `python -m pytest -q -s code/tt_ptv3/tests/test_pcc_device.py code/tt_ptv3/tests/test_e2e_device.py code/tt_ptv3/tests/test_e2e_seg_device.py` | `test` |

## Quickstart

Run it from the model repo root: `code/tt_ptv3/samples/test_pcd.npz` (Autoware's `test.pcd`, Apache-2.0) is a path
relative to it.

```python
from tt_ptv3 import PTv3

with PTv3.from_pretrained(device_id=0) as model:                  # variant "seg_det": both heads
    out = model("code/tt_ptv3/samples/test_pcd.npz")
print(out.class_counts())                                         # input points per class
print(out.detections.to_dicts())                                  # the boxes (None with variant="seg")
body = out.to_dict()                                              # the same JSON as POST /predict
```

`examples/quickstart.py` runs this and also writes `quickstart_bev.png` (`tt_ptv3.viz.render_bev`: the points coloured
by class, the boxes, from above).

## `PTv3.from_pretrained(...)`

```python
PTv3.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default AutowareFoundation/ptv3
    *,
    revision=None,          # default for the default repo: the validated commit 1673104843a (tag v4.0)
    variant=None,           # load-time variant: "seg_det" (encoder + seg head + det head, traces A and B) or "seg"
                            # (encoder + seg head, trace A: the autoware_launch wiring); default $PTV3_VARIANT or "seg_det"
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_ptv3.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $PTV3_DISPATCH or "eth"
    num_command_queues=None,  # default $PTV3_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
    **compile_params,       # none for this model (the capacity buckets are knobs, see "Warm-up")
) -> PTv3
```

What it does: resolves the weights first, so a Hub problem never claims the chip (`weights_dir` >
`$PTV3_WEIGHTS_DIR` > a local `model_id` directory > the HF snapshot at the pinned revision, restricted to
`ptv3_encoder.onnx, ptv3_seg3d_head.onnx, ptv3_det3d_head.onnx, ml_package_ptv3_*.param.yaml, deploy_metadata.yaml`, with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ;
the other open parameters are `DEVICE_DEFAULTS` in `tt_ptv3/device.py`, overridable with `PTV3_*`), builds the
graph, then compiles and captures the metal trace of every warm-up variant. If ETH dispatch cannot open (tt-metal
without the patch), it warns and falls back to WORKER dispatch.

## Warm-up

`from_pretrained` returns a warm model: it reads the three ONNX graphs (`tt_ptv3.reference.weights`, BN and the CPE
linear folded exactly), uploads the weights, runs each trace once eagerly (this first run compiles the kernels) and
captures it as a metal trace, so no call compiles anything. The graph is captured once per **capacity bucket**
(`PTV3_BUCKETS=ladder`, `PTV3_POOL_PROFILES=p2,p3`: 15 stage-0 capacities from 16k to 224k voxels, each with two
deep-stage profiles, plus the node's 256k maximum `n256k`; 31 buckets, one trace A `encseg_<bucket>` and, for
`seg_det`, one trace B `det_<bucket>` each, sharing the weights). Each call runs in the cheapest bucket whose five
stage capacities hold the frame's stage counts, chosen on the host (`tt.model.select_bucket`; `meta.frame.stage_counts`
in the output); the outputs do not depend on the bucket (bit-identical to `n256k` on the 22 gate frames).

- Measured (2026-10-15, ETH dispatch, 1 CQ, 12x10, warm JIT cache, `VERIFICATION_OPT_2026-10-15.md`): 57-64 s for
  `seg_det` with the whole ladder (graph build and weight upload 12.5 s, eager warm-up + capture of the 62 traces
  40-47 s, the rest the device open). A process with an empty JIT cache compiles every kernel of every bucket first
  (tens of minutes for the full ladder); `PTV3_BUCKETS=32k,80k` (a list of stage-0 capacities, + n256k) trims the ladder
  and the load. The traces are captured during the load, so the first call costs as much as the next.
- The load prints one stderr line per warm-up variant (`[ptv3-p150/seg_det] warm-up 33/62 encseg_n96k +12.5 s`).
  On a cold cache the lines come minutes apart for the large buckets (compiling); a load whose last line is many
  minutes old with no new kernels in the JIT cache is stalled.
- `warmup_variants="none"` (server: `PTV3_WARMUP=none`) skips the capture at load; call `model.warmup()` before the
  first frame (it is idempotent).
- The JIT cache is tt-metal's (`~/.cache/tt-metal-cache`, or `TT_METAL_CACHE`); inside the container it is
  `~/.cache/tt-model/ptv3-p150/cache` of the host.

## Call: `model(points, **kwargs)`

| Argument | Type | Description |
|---|---|---|
| `points` (first positional) | path, `bytes` + `fmt=`, (N, C) array / tensor, structured array, `PointCloud`, JSON envelope | one concatenated LiDAR cloud in `base_link` with x, y, z, intensity (see "Input types") |
| `run_detection` | bool, default `True` | run trace B (the det head) and return boxes; `False` skips it (same segmentation). The node runs a head only when its output topic has a subscriber. No effect with `variant="seg"` (no det head) |
| `score_threshold` | float in [0, 1], default 0.1 | the node's per-distance-bin score threshold (all four bins are 0.1 in its config) |
| `reconstruction` | `"full"` (default) \| `"partial"` \| `"none"` | which points get an output row (the node's `source_reconstruction` at @5eb9286): `full` every input point in input order (out-of-crop points INVALID); `partial` the points inside the crop, in input order; `none` one row per voxel (its first point) |
| `neighbour_mode` | `"alias-first"` (default) \| `"exact"` | how the host builds the sub-manifold neighbour tables: spconv's linear-key aliasing of out-of-range top-z voxels, as the TensorRT engine builds it (the lowest-index voxel wins an aliased key), or exact keys. Only the host tables change; both are validated against their own CPU goldens under the same gates |
| `min_voxels` | int in [0, 256000], default 16000 | the node's minimum: below it the node publishes nothing, and so does the model (`meta["rejected"]`, empty output, no device run). `0` disables it (every single-sweep nuScenes key-frame has 10-16k voxels) |

Every knob is host-only (PLAN.md 2.7, RT-host): no device shape changes. Defaults reproduce node @5eb9286's
`config/ptv3.param.yaml` and the weights' ml_package files. A bad value raises `tt_ptv3.io.InputError` (HTTP 400)
before the device runs. `sweeps`, `stream`, `images`, `calibration` and `inputs` (the shared call signature of the
Autoware ports) are refused: the v4.0 node reads one concatenated cloud, with no densification and no state.

### Input types

- Point clouds: a path (`.bin` raw float32 rows -- pass `fields=` if not `x, y, z, intensity`; `.npy`;
  `.npz`; `.pcd` ascii / binary / binary_compressed), raw bytes with `fmt=`, an (N, C) numpy array or torch tensor,
  a structured array, the JSON envelope of `/predict`, or `tt_ptv3.io.PointCloud`. Frame: `base_link`, metres.
  An unnamed (N, C) array is x, y, z, intensity, ...: the first four columns are used (Autoware's concatenated
  `XYZIRC...` layouts start with them).
- `intensity` is Autoware's uint8 value, 0-255: the feature is `intensity / 255` (the node's uint8-intensity path).
- The node's crop is x, y in [-122.88, 122.88) m and z in [-3, 5) m (base_link); points outside it get label 255.
  Voxels are 0.12 m; the first point of a voxel (in input order) is its feature, so point order matters; at most
  256,000 voxels (the rest is clipped, as the node does).

## Output

`model(...)` returns a `tt_ptv3.PTv3Output` (a `ttaw.outputs.Segmentation3D`); `out.to_dict()` is exactly the
`POST /predict` body (SERVING.md section 3.2).

| field | type | meaning |
|---|---|---|
| `label_ids` | uint8 `[N]` | the PTv3 class per output point (`class_names`: car, truck, bus, bicycle, pedestrian, traffic_cone, barrier, debris, drivable_flat, non_drivable_flat, vegetation, building, vertical_thin, static_clutter, noise); 255 = outside the crop. With `reconstruction="full"`, N = input points, in input order |
| `scores` | float32 `[N]` | the probability of that class (softmax over 15); 0 for invalid points |
| `classification` | uint8 `[N]` | Autoware's `PointCloudClassification` value of the node's `class_mapping` (CAR 0, TRUCK 1, BUS 2, BICYCLE 4, PEDESTRIAN 5, HAZARD 7, FLAT_SURFACE 8, STRUCTURE 9, VEGETATION 10, NOISE 11; INVALID 255) |
| `entropy` | float32 `[N]` | the normalised entropy of the 15 probabilities, `-sum p ln p / ln 15`; NaN for invalid points |
| `filter_keep` | bool `[N]` | membership of the node's `~/output/pointcloud/filtered` (the obstacle cloud): False for drivable_flat, non_drivable_flat and noise; points outside the crop are kept |
| `detections` | `Detections3D` or None | `boxes` float32 `[M, 7]` (x, y, z = box centre, length, width, height, yaw in base_link), `scores`, `label_ids` / `labels` (CAR, TRUCK, BUS, BICYCLE, PEDESTRIAN, TRAFFIC_CONE, BARRIER), `velocity` `[M, 2]`, `meta["autoware_labels"]` (the `ObjectClassification` name the node publishes: CAR, TRUCK, BUS, BICYCLE, PEDESTRIAN; TRAFFIC_CONE and BARRIER become HAZARD; also `autoware_label` per box in the JSON), after the score threshold, the yaw-norm gate, IoU-BEV NMS and the area class remapper; None with `run_detection=False` or `variant="seg"` |
| `meta` | dict | `frame` (points, points in range, voxels, stage counts, neighbour mode, `rejected` / `clipped` / `overflow` flags), `neighbour_mode`, `reconstruction`; `rejected` (the reason) when the node would publish nothing |
| `timing_ms` | dict | `preprocess` (decode, voxelization, serialization, neighbour maps), `device` (tables, H2D, trace A (+ B), D2H, unpack), `postprocess`, `total` of this call |

`out.class_counts()` gives the points per class (key `"255"` for invalid points); `out.detections.to_dicts()` the
box list `[{"label", "label_id", "score", "center", "size", "yaw", "velocity"}]`.

## Lifetime and information

- `model.close()` releases the traces and the device tensors and closes the chip if the model opened it; idempotent.
  `with` calls it for you; an unclosed model is closed when Python exits.
- `model.info`: weights (repo, tag, revision, path), device (dispatch, grid), variant, warm variants, warm-up times,
  runtime parameter defaults.
- Calls from several threads are safe: the device calls are serialised. One model per process per chip.

## Speed

Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, the shipped precision and knobs, variant `seg_det`
(`code/scripts/bench.py`, 20 iterations, mean of the p50s of 2 processes; `VERIFICATION_OPT_2026-10-15.md`, an
independent re-measurement of the optimized code, 2026-10-15, on a shared host whose load moves the host stages; the
first release's numbers are in `OPT_BASELINE.md`):

| stage, p50 | test.pcd (shipped; bucket `n32k_p3`) | PandaSet 019 f40 (not shipped; bucket `n80k`) |
|---|---:|---:|
| `model(path)` end to end, `seg_det` | 72.8 ms | 164.4 ms |
| `model(path, run_detection=False)` (no trace B) | 62.8 ms | 141.9 ms |
| host pre-processing (decode, voxelize, serialize, pooling metadata, numba neighbour maps) | 25.8 ms | 59.4 ms |
| device tables + host tensors + H2D | 5.6 + 4.5 + 3.5 ms | 11.3 + 6.9 + 6.8 ms |
| trace A (encoder + seg head), one replay | 20.96 ms | 51.20 ms |
| trace B (det head), one replay | 9.18 ms | 9.53 ms |
| D2H A + B | 0.42 + 0.21 ms | 0.72 + 0.20 ms |
| host post-processing (reconstruction, LUT, filter; decode, NMS, remap) | 2.9 ms | 14.8 ms |

- Back-to-back replays take 30.0 ms per frame on test.pcd (A 20.9 + B 9.2; 33.3 frames/s of device throughput) and
  60.6 ms on the PandaSet frame (16.5 frames/s); the first release took 885 ms for any cloud. A synchronous call gives
  13.7 (test.pcd) and 6.1 (PandaSet) calls/s.
- **Trace A grows with the cloud**: it runs at the selected bucket's capacities, so a denser frame (the T4 frames
  have 96k-132k voxels) runs in a larger bucket and takes longer; the host pre-processing grows with the point count
  too. `OPT_REPORT.md` has every optimization step, its A/B and the remaining backlog.

## Limits

- Batch 1 on the chip; one model per process; the largest capacity bucket is the node's maximum (256,000 / 179,200 /
  107,520 / 53,760 / 27,136 voxels per stage): voxels beyond 256k are clipped as the node does, a frame whose pooled
  stages exceed a capacity is
  refused (`InputError`), never truncated.
- Frames below 16,000 voxels give the node's empty output unless `min_voxels` is lowered.
- Outputs differ slightly from the fp32 reference (README "Demo & Performances": 99.65-99.89 % of point labels agree;
  boxes near the score threshold or an NMS / top-k near-tie can appear or disappear).
- One p150, ETH dispatch (12x10); `dispatch="worker"` is an A/B opt-in (11x10). Not a ROS 2 node.

## Tests

From the repository root, with `code/` on `PYTHONPATH` (or the package installed). Name the test files; never run
pytest on the whole directory on a shared device host.

- Host (no device, no ttnn call): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_ptv3/tests/test_bundle_host.py
  code/tt_ptv3/tests/test_host_units.py code/tt_ptv3/tests/test_io_host.py code/tt_ptv3/tests/test_server_host.py`.
  Inside the tt-models workspace the suite also runs `test_tt_host.py`, `test_tt_fake_host.py` (the whole graph on a
  fake ttnn), `test_reference_cpu.py` and `test_host_vs_research.py` against workspace oracles (ONNX
  ReferenceEvaluator, ONNX Runtime, the research reference); tests that need workspace-only data skip elsewhere, and
  a missing sample FAILS inside the workspace.
- Device (one p150): `python -m pytest -q -s code/tt_ptv3/tests/test_pcc_device.py
  code/tt_ptv3/tests/test_e2e_device.py code/tt_ptv3/tests/test_e2e_seg_device.py` (per-module PCC against the fp32
  CPU reference fed identical host tables, on replay outputs; end-to-end agreement on 21 frames in both neighbour
  modes and both variants; the samples against their stored references; replay == eager; API == `/predict`). The
  frame gates run only inside the tt-models workspace, where their data lives (T4 sample-dataset, PandaSet and
  nuScenes frames, which this repository does not ship); outside it only the shipped `test.pcd` checks run.
- The gates are frozen in `tests/*.gates.json` at the first green run and are never loosened.