code/PYTHON.md

Python API: Point Transformer V3 (Autoware autoware_ptv3) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are the same.

Install

Install the package on top of an environment that already has ttnn (a tt-metal python_env at 44d66500520 with patches/tt-metal-eth-dispatch.patch, or the tt-model container). From the root of the model repository (the directory that holds pyproject.toml, README.md and code/):

pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests

The pip project is the repository's top-level pyproject.toml; it installs the package from code/tt_ptv3 (there is no pyproject.toml inside code/, because the container build copies code/ over the tt-metal tree).

The package carries tt_ptv3.ttaw, the shared code of the Autoware ports to Blackhole (device open, trace runner, decoders, model base class, HTTP app), vendored at the version recorded in code/tt_ptv3/ttaw/VENDORED.json.

You want to run	Extras
the Python API	none
the HTTP server (tt_ptv3.server.app, see SERVING.md)	server
host tests (no device; device tests are skipped): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_ptv3/tests/test_bundle_host.py code/tt_ptv3/tests/test_host_units.py code/tt_ptv3/tests/test_io_host.py code/tt_ptv3/tests/test_server_host.py (and the other test_*_host.py / reference files; see "Tests")	server,test
device tests: python -m pytest -q -s code/tt_ptv3/tests/test_pcc_device.py code/tt_ptv3/tests/test_e2e_device.py code/tt_ptv3/tests/test_e2e_seg_device.py	test

Quickstart

Run it from the model repo root: code/tt_ptv3/samples/test_pcd.npz (Autoware's test.pcd, Apache-2.0) is a path relative to it.

from tt_ptv3 import PTv3

with PTv3.from_pretrained(device_id=0) as model:                  # variant "seg_det": both heads
    out = model("code/tt_ptv3/samples/test_pcd.npz")
print(out.class_counts())                                         # input points per class
print(out.detections.to_dicts())                                  # the boxes (None with variant="seg")
body = out.to_dict()                                              # the same JSON as POST /predict

examples/quickstart.py runs this and also writes quickstart_bev.png (tt_ptv3.viz.render_bev: the points coloured by class, the boxes, from above).

PTv3.from_pretrained(...)

PTv3.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default AutowareFoundation/ptv3
    *,
    revision=None,          # default for the default repo: the validated commit 1673104843a (tag v4.0)
    variant=None,           # load-time variant: "seg_det" (encoder + seg head + det head, traces A and B) or "seg"
                            # (encoder + seg head, trace A: the autoware_launch wiring); default $PTV3_VARIANT or "seg_det"
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_ptv3.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $PTV3_DISPATCH or "eth"
    num_command_queues=None,  # default $PTV3_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
    **compile_params,       # none for this model (the capacity buckets are knobs, see "Warm-up")
) -> PTv3

What it does: resolves the weights first, so a Hub problem never claims the chip (weights_dir > $PTV3_WEIGHTS_DIR > a local model_id directory > the HF snapshot at the pinned revision, restricted to ptv3_encoder.onnx, ptv3_seg3d_head.onnx, ptv3_det3d_head.onnx, ml_package_ptv3_*.param.yaml, deploy_metadata.yaml, with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ; the other open parameters are DEVICE_DEFAULTS in tt_ptv3/device.py, overridable with PTV3_*), builds the graph, then compiles and captures the metal trace of every warm-up variant. If ETH dispatch cannot open (tt-metal without the patch), it warns and falls back to WORKER dispatch.

Warm-up

from_pretrained returns a warm model: it reads the three ONNX graphs (tt_ptv3.reference.weights, BN and the CPE linear folded exactly), uploads the weights, runs each trace once eagerly (this first run compiles the kernels) and captures it as a metal trace, so no call compiles anything. The graph is captured once per capacity bucket (PTV3_BUCKETS=ladder, PTV3_POOL_PROFILES=p2,p3: 15 stage-0 capacities from 16k to 224k voxels, each with two deep-stage profiles, plus the node's 256k maximum n256k; 31 buckets, one trace A encseg_<bucket> and, for seg_det, one trace B det_<bucket> each, sharing the weights). Each call runs in the cheapest bucket whose five stage capacities hold the frame's stage counts, chosen on the host (tt.model.select_bucket; meta.frame.stage_counts in the output); the outputs do not depend on the bucket (bit-identical to n256k on the 22 gate frames).

• Measured (2026-10-15, ETH dispatch, 1 CQ, 12x10, warm JIT cache, VERIFICATION_OPT_2026-10-15.md): 57-64 s for seg_det with the whole ladder (graph build and weight upload 12.5 s, eager warm-up + capture of the 62 traces 40-47 s, the rest the device open). A process with an empty JIT cache compiles every kernel of every bucket first (tens of minutes for the full ladder); PTV3_BUCKETS=32k,80k (a list of stage-0 capacities, + n256k) trims the ladder and the load. The traces are captured during the load, so the first call costs as much as the next.
• The load prints one stderr line per warm-up variant ([ptv3-p150/seg_det] warm-up 33/62 encseg_n96k +12.5 s). On a cold cache the lines come minutes apart for the large buckets (compiling); a load whose last line is many minutes old with no new kernels in the JIT cache is stalled.
• warmup_variants="none" (server: PTV3_WARMUP=none) skips the capture at load; call model.warmup() before the first frame (it is idempotent).
• The JIT cache is tt-metal's (~/.cache/tt-metal-cache, or TT_METAL_CACHE); inside the container it is ~/.cache/tt-model/ptv3-p150/cache of the host.

Call: model(points, **kwargs)

Argument	Type	Description
points (first positional)	path, bytes + fmt=, (N, C) array / tensor, structured array, PointCloud, JSON envelope	one concatenated LiDAR cloud in base_link with x, y, z, intensity (see "Input types")
run_detection	bool, default True	run trace B (the det head) and return boxes; False skips it (same segmentation). The node runs a head only when its output topic has a subscriber. No effect with variant="seg" (no det head)
score_threshold	float in [0, 1], default 0.1	the node's per-distance-bin score threshold (all four bins are 0.1 in its config)
reconstruction	"full" (default) | "partial" | "none"	which points get an output row (the node's source_reconstruction at @5eb9286): full every input point in input order (out-of-crop points INVALID); partial the points inside the crop, in input order; none one row per voxel (its first point)
neighbour_mode	"alias-first" (default) | "exact"	how the host builds the sub-manifold neighbour tables: spconv's linear-key aliasing of out-of-range top-z voxels, as the TensorRT engine builds it (the lowest-index voxel wins an aliased key), or exact keys. Only the host tables change; both are validated against their own CPU goldens under the same gates
min_voxels	int in [0, 256000], default 16000	the node's minimum: below it the node publishes nothing, and so does the model (meta["rejected"], empty output, no device run). 0 disables it (every single-sweep nuScenes key-frame has 10-16k voxels)

Every knob is host-only (PLAN.md 2.7, RT-host): no device shape changes. Defaults reproduce node @5eb9286's config/ptv3.param.yaml and the weights' ml_package files. A bad value raises tt_ptv3.io.InputError (HTTP 400) before the device runs. sweeps, stream, images, calibration and inputs (the shared call signature of the Autoware ports) are refused: the v4.0 node reads one concatenated cloud, with no densification and no state.

Input types

• Point clouds: a path (.bin raw float32 rows -- pass fields= if not x, y, z, intensity; .npy; .npz; .pcd ascii / binary / binary_compressed), raw bytes with fmt=, an (N, C) numpy array or torch tensor, a structured array, the JSON envelope of /predict, or tt_ptv3.io.PointCloud. Frame: base_link, metres. An unnamed (N, C) array is x, y, z, intensity, ...: the first four columns are used (Autoware's concatenated XYZIRC... layouts start with them).
• intensity is Autoware's uint8 value, 0-255: the feature is intensity / 255 (the node's uint8-intensity path).
• The node's crop is x, y in [-122.88, 122.88) m and z in [-3, 5) m (base_link); points outside it get label 255. Voxels are 0.12 m; the first point of a voxel (in input order) is its feature, so point order matters; at most 256,000 voxels (the rest is clipped, as the node does).

Output

model(...) returns a tt_ptv3.PTv3Output (a ttaw.outputs.Segmentation3D); out.to_dict() is exactly the POST /predict body (SERVING.md section 3.2).

field	type	meaning
label_ids	uint8 [N]	the PTv3 class per output point (class_names: car, truck, bus, bicycle, pedestrian, traffic_cone, barrier, debris, drivable_flat, non_drivable_flat, vegetation, building, vertical_thin, static_clutter, noise); 255 = outside the crop. With reconstruction="full", N = input points, in input order
scores	float32 [N]	the probability of that class (softmax over 15); 0 for invalid points
classification	uint8 [N]	Autoware's PointCloudClassification value of the node's class_mapping (CAR 0, TRUCK 1, BUS 2, BICYCLE 4, PEDESTRIAN 5, HAZARD 7, FLAT_SURFACE 8, STRUCTURE 9, VEGETATION 10, NOISE 11; INVALID 255)
entropy	float32 [N]	the normalised entropy of the 15 probabilities, -sum p ln p / ln 15; NaN for invalid points
filter_keep	bool [N]	membership of the node's ~/output/pointcloud/filtered (the obstacle cloud): False for drivable_flat, non_drivable_flat and noise; points outside the crop are kept
detections	Detections3D or None	boxes float32 [M, 7] (x, y, z = box centre, length, width, height, yaw in base_link), scores, label_ids / labels (CAR, TRUCK, BUS, BICYCLE, PEDESTRIAN, TRAFFIC_CONE, BARRIER), velocity [M, 2], meta["autoware_labels"] (the ObjectClassification name the node publishes: CAR, TRUCK, BUS, BICYCLE, PEDESTRIAN; TRAFFIC_CONE and BARRIER become HAZARD; also autoware_label per box in the JSON), after the score threshold, the yaw-norm gate, IoU-BEV NMS and the area class remapper; None with run_detection=False or variant="seg"
meta	dict	frame (points, points in range, voxels, stage counts, neighbour mode, rejected / clipped / overflow flags), neighbour_mode, reconstruction; rejected (the reason) when the node would publish nothing
timing_ms	dict	preprocess (decode, voxelization, serialization, neighbour maps), device (tables, H2D, trace A (+ B), D2H, unpack), postprocess, total of this call

out.class_counts() gives the points per class (key "255" for invalid points); out.detections.to_dicts() the box list [{"label", "label_id", "score", "center", "size", "yaw", "velocity"}].

Lifetime and information

• model.close() releases the traces and the device tensors and closes the chip if the model opened it; idempotent. with calls it for you; an unclosed model is closed when Python exits.
• model.info: weights (repo, tag, revision, path), device (dispatch, grid), variant, warm variants, warm-up times, runtime parameter defaults.
• Calls from several threads are safe: the device calls are serialised. One model per process per chip.

Speed

Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, the shipped precision and knobs, variant seg_det (code/scripts/bench.py, 20 iterations, mean of the p50s of 2 processes; VERIFICATION_OPT_2026-10-15.md, an independent re-measurement of the optimized code, 2026-10-15, on a shared host whose load moves the host stages; the first release's numbers are in OPT_BASELINE.md):

stage, p50	test.pcd (shipped; bucket n32k_p3)	PandaSet 019 f40 (not shipped; bucket n80k)
model(path) end to end, seg_det	72.8 ms	164.4 ms
model(path, run_detection=False) (no trace B)	62.8 ms	141.9 ms
host pre-processing (decode, voxelize, serialize, pooling metadata, numba neighbour maps)	25.8 ms	59.4 ms
device tables + host tensors + H2D	5.6 + 4.5 + 3.5 ms	11.3 + 6.9 + 6.8 ms
trace A (encoder + seg head), one replay	20.96 ms	51.20 ms
trace B (det head), one replay	9.18 ms	9.53 ms
D2H A + B	0.42 + 0.21 ms	0.72 + 0.20 ms
host post-processing (reconstruction, LUT, filter; decode, NMS, remap)	2.9 ms	14.8 ms

• Back-to-back replays take 30.0 ms per frame on test.pcd (A 20.9 + B 9.2; 33.3 frames/s of device throughput) and 60.6 ms on the PandaSet frame (16.5 frames/s); the first release took 885 ms for any cloud. A synchronous call gives 13.7 (test.pcd) and 6.1 (PandaSet) calls/s.
• Trace A grows with the cloud: it runs at the selected bucket's capacities, so a denser frame (the T4 frames have 96k-132k voxels) runs in a larger bucket and takes longer; the host pre-processing grows with the point count too. OPT_REPORT.md has every optimization step, its A/B and the remaining backlog.

Limits

• Batch 1 on the chip; one model per process; the largest capacity bucket is the node's maximum (256,000 / 179,200 / 107,520 / 53,760 / 27,136 voxels per stage): voxels beyond 256k are clipped as the node does, a frame whose pooled stages exceed a capacity is refused (InputError), never truncated.
• Frames below 16,000 voxels give the node's empty output unless min_voxels is lowered.
• Outputs differ slightly from the fp32 reference (README "Demo & Performances": 99.65-99.89 % of point labels agree; boxes near the score threshold or an NMS / top-k near-tie can appear or disappear).
• One p150, ETH dispatch (12x10); dispatch="worker" is an A/B opt-in (11x10). Not a ROS 2 node.

Tests

From the repository root, with code/ on PYTHONPATH (or the package installed). Name the test files; never run pytest on the whole directory on a shared device host.

• Host (no device, no ttnn call): TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_ptv3/tests/test_bundle_host.py code/tt_ptv3/tests/test_host_units.py code/tt_ptv3/tests/test_io_host.py code/tt_ptv3/tests/test_server_host.py. Inside the tt-models workspace the suite also runs test_tt_host.py, test_tt_fake_host.py (the whole graph on a fake ttnn), test_reference_cpu.py and test_host_vs_research.py against workspace oracles (ONNX ReferenceEvaluator, ONNX Runtime, the research reference); tests that need workspace-only data skip elsewhere, and a missing sample FAILS inside the workspace.
• Device (one p150): python -m pytest -q -s code/tt_ptv3/tests/test_pcc_device.py code/tt_ptv3/tests/test_e2e_device.py code/tt_ptv3/tests/test_e2e_seg_device.py (per-module PCC against the fp32 CPU reference fed identical host tables, on replay outputs; end-to-end agreement on 21 frames in both neighbour modes and both variants; the samples against their stored references; replay == eager; API == /predict). The frame gates run only inside the tt-models workspace, where their data lives (T4 sample-dataset, PandaSet and nuScenes frames, which this repository does not ship); outside it only the shipped test.pcd checks run.
• The gates are frozen in tests/*.gates.json at the first green run and are never loosened.