code/PYTHON.md

# Python API: FRNet (Autoware lidar_frnet) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the
API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are
the same.

## Install

Install the package on top of an environment that already has `ttnn` (a tt-metal `python_env` at `44d66500520`
with `patches/tt-metal-eth-dispatch.patch` and `patches/tt-metal-reshape-rm-sys1419.patch`, or the tt-model container). From the root of the model repository (the
directory that holds `pyproject.toml`, `README.md` and `code/`):

```bash
pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests
```

The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_frnet` (there is no
`pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree).

The package carries `tt_frnet.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner,
decoders, model base class, HTTP app, the K1 `segment_reduce` kernel), vendored at the version recorded in
`code/tt_frnet/ttaw/VENDORED.json` (0.23.5).

| You want to run | Extras |
|---|---|
| the Python API | none |
| the HTTP server (`tt_frnet.server.app`, see `SERVING.md`) | `server` |
| host tests (no device; device tests are skipped): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_frnet/tests` | `server,test` |
| device tests: `python -m pytest -q -s code/tt_frnet/tests/test_pcc_device.py code/tt_frnet/tests/test_e2e_device.py` | `test` |

## Quickstart

Run from the repository root (the sample paths are relative to it):

```python
from tt_frnet import FRNet

with FRNet.from_pretrained(device_id=0) as model:                  # variant="qt128" for the Hesai QT128 profile
    out = model("code/tt_frnet/samples/synthetic_ot128.npz",       # one sweep in the sensor frame
                calibration={"preset": "synthetic_ot128"})         # its sensor -> base_link transform (ego crop box)

print(out.class_counts())                                          # points per class (raw + interpolated)
labels, scores, xyz = out.label_ids, out.scores, out.points[:, :3]
```

`examples/quickstart.py` runs this snippet and writes `quickstart.json` (the `/predict` body) and `quickstart_bev.png`
(a bird's-eye view of the p150 classes next to the stored CPU reference of the shipped sample).

## `FRNet.from_pretrained(...)`

```python
FRNet.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default AutowareFoundation/lidar_frnet
    *,
    revision=None,          # default for the default repo: the validated commit a53a1c11b0b (tag v2.0)
    variant=None,           # load-time variant: "ot128" (Hesai OT128, Autoware's default) | "qt128"; default $FRNET_VARIANT or "ot128"
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_frnet.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $FRNET_DISPATCH or "eth"
    num_command_queues=None,  # default $FRNET_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
) -> FRNet
```

What it does: resolves the weights first, so a Hub problem never claims the chip (`weights_dir` >
`$FRNET_WEIGHTS_DIR` > a local `model_id` directory > the HF snapshot at the pinned revision, restricted to
`frnet_*.onnx, ml_package_frnet_*.param.yaml, deploy_metadata.yaml`, with an offline fallback to the cache), checks
the sha256 of the variant's ONNX against the pinned one, reads the variant's `ml_package_frnet_<variant>.param.yaml`
(FOV, frustum and interpolation grids, the 27 class names, the palette, the point / cell profiles), opens the chip
(ETH dispatch, 12×10, 1 CQ, `l1_small_size` 64 KiB, `trace_region_size` 256 MiB: `DEVICE_DEFAULTS` in
`tt_frnet/device.py`, overridable with `FRNET_*`), builds the graph, then compiles and captures the metal trace. If
ETH dispatch cannot open (tt-metal without the patch), it warns and falls back to WORKER dispatch.

`variant` picks one of the two networks of the weights repo. Each is tied to its sensor (Autoware's `sensor_model`):

| variant | sensor | FOV up / down | frustum grid (H × W) | interpolation grid | points (min / max) | unique cells (min / max) |
|---|---|---|---|---|---|---|
| `ot128` (default) | Hesai OT128 | +15° / −25° | 128 × 1024 | 128 × 4096 | 5,000 / 160,000 | 3,000 / 60,000 |
| `qt128` | Hesai QT128 | +52.6° / −52.6° | 128 × 256 | 128 × 1024 | 5,000 / 120,000 | 3,000 / 60,000 |

Precision knobs (read once at load from the environment, pinned in the package's `serve.env`; the defaults are the
validated configuration and every number of the model card uses them): `FRNET_POINT_TERMS=2` (each point-wise linear
as two-term bf16 operands, about 16 mantissa bits, fp32 accumulation), `FRNET_MAP_DTYPE=float32` (fp32 image-branch
maps, bf16 conv operands), `FRNET_CONV_TERMS=1`. Other values are A/B switches for the optimization phase; they fail
the frozen accuracy gates (`OPT_BASELINE.md`, "The precision cost of the policy").

## Warm-up

`from_pretrained` returns a warm model: it builds the graph of the variant, runs each of its trace variants once
eagerly on an empty frame (this first run compiles the kernels and uploads the weights) and captures them as metal
traces, so no call compiles anything. There are three trace variants per profile, all captured at load: the full
capacity (`N_cap` points = 160,000 / 120,000, `M_cap` cells = 60,000) and two capacity buckets (ot128 40,000 / 80,000
points, qt128 29,984 / 60,000); a frame runs in the smallest trace that holds its points and cells, and its valid rows
are bit-identical in every trace. The numba twins of the host pre- and post-processing are compiled at load too.

- Measured (2026-10-14, the shipped sample, optimized build): with an empty JIT cache the load took 178.6 s for ot128
  and 178.0 s for qt128 (firmware and every kernel compiled, about 0.7 GB of cache); with that cache warm, in a new
  process, 14.6 s and 13.4-14.0 s (graph build 6.4 s, warm-up + capture of the three traces 2.4-2.8 s, the rest mostly
  the numba compile). The first call costs about as much as the next ones (100 / 99 ms for ot128, 88 / 78 ms for qt128,
  first / second call, on a shared host).
- `warmup_variants="none"` (server: `FRNET_WARMUP=none`) skips the capture at load; call `model.warmup()` before the
  first frame (it is idempotent).
- The JIT cache is tt-metal's (`~/.cache/tt-metal-cache`, or `TT_METAL_CACHE`); inside the container it is
  `~/.cache/tt-model/frnet-p150/cache` of the host.

## Call: `model(points, calibration=..., **kwargs)`

| Argument | Type | Description |
|---|---|---|
| `points` (first positional) | path, `bytes` + `fmt=`, (N, C) array / tensor, structured array, `PointCloud`, JSON envelope | ONE LiDAR sweep in the **sensor frame** (x forward, y left, z up, metres), fields `x, y, z, intensity` with the raw intensity the node casts (0-255 for uint8 drivers); see "Input types" |
| `calibration` | `{"T_base_link_from_sensor": <transform>}`, `{"preset": name}` (a file of `code/tt_frnet/calib/`), optional `"ego_crop_box"` | the static sensor -> `base_link` transform the node looks up once for its ego crop box. `ego_crop_box`: `[min_x, min_y, min_z, max_x, max_y, max_z]` in `base_link` (default Autoware's `(-1.53, -1.15, -0.1)` .. `(6.1, 1.15, 3.1)`), or `None` (or all zeros) to disable the box, the node's own switch; a disabled box needs no transform |
| `class_probability_threshold` | float in [0, 1], default 0.05 | the filtered cloud keeps a point unless its probability of a filter class is >= this value (Autoware's `filter.class_probability_threshold`) |
| `filter_classes` | comma-separated class names, `""` = none, default `"drivable_surface"` | Autoware's `filter.classes` (at most 16); an unknown name is refused |

Both knobs only change `filter_keep`; the class decision (`label_ids`, `scores`) never depends on them. They are
host-only (no device shape changes), the defaults reproduce Autoware's `frnet.param.yaml`, and a bad value raises
`tt_frnet.io.InputError` (HTTP 400) before the device runs. `model.info["runtime_params"]` (and the server's `/info`)
lists them with their defaults.

### Input types

- Point clouds: a path (`.bin` raw float32 rows -- pass `fields=` if not `x, y, z, intensity`; `.npy`; `.npz`;
  `.pcd` ascii / binary / binary_compressed), raw bytes with `fmt=`, an (N, C) numpy array or torch tensor, a
  structured array, the JSON envelope of `/predict`, or `tt_frnet.io.PointCloud`. The frame is the **sensor** frame
  (Autoware's `~/input/pointcloud` before any transform), one sweep, not motion-compensated.
- Rows with a non-finite x / y / z are dropped by the decoder; `input_index` refers to the decoded rows.
- Calibration transforms: a 4×4 (or 3×4) row-major matrix, `{"translation": [x, y, z], "rotation_wxyz": [w, x, y, z]}`,
  or Autoware's `{x, y, z, roll, pitch, yaw}`. With the crop box on and no transform the call is refused (the node would
  skip every frame because its TF lookup fails); the error names both fixes.
- `sweeps`, `images`, `stream` and `inputs` are refused: FRNet takes one sweep and its calibration only.
- A frame outside the variant's profile (input points, points after interpolation, or unique cells out of the table
  above) is not an error: as the node, which logs and publishes nothing, the call returns an empty output whose
  `meta["skipped"]` holds the node's message, and nothing runs on the device.

## Output

`model(...)` returns a `tt_frnet.FRNetSegmentation` (a `ttaw.outputs.Segmentation3D`); `out.to_dict()` is exactly the
`POST /predict` body (SERVING.md section 3.2). The point order is the node's: the input points that survive the ego
crop box, in input order, then the points Autoware's interpolation appends (D4: published as Autoware does).

| field | type | meaning |
|---|---|---|
| `label_ids` | uint8 `[N]` | Autoware's `class_id` (first maximum of the 27 probabilities, as `fill_cloud_kernel`); names in `class_names` |
| `scores` | float32 `[N]` | Autoware's `probability`: the softmax probability of that class |
| `points` | float32 `[N, 4]` | x, y, z, intensity of every output point in the sensor frame (the segmentation cloud's coordinates; the interpolated points carry interpolated values) |
| `filter_keep` | bool `[N]` | membership of Autoware's `~/output/pointcloud/filtered` cloud (False = removed by the class filter) |
| `num_points_raw` | int | the first `num_points_raw` rows are input points, the rest are interpolated |
| `input_index` | int32 `[num_points_raw]` | the decoded input row of every raw output point |
| `class_names` | 27 str | `drivable_surface`, `other_flat_surface`, `sidewalk`, `manmade`, `vegetation`, `car`, `bus`, `emergency_vehicle`, `train`, `truck`, `tractor_unit`, `semi_trailer`, `construction_vehicle`, `forklift`, `kart`, `motorcycle`, `bicycle`, `pedestrian`, `personal_mobility`, `animal`, `pushable_pullable`, `traffic_cone`, `debris`, `stroller`, `other_stuff`, `noise+ghost_point`, `unknown` (Autoware's names; ids 22 / 23 were trained as stroller / debris, see Limits) |
| `meta` | dict | `variant`; `num_points_input`; `ego_crop` (the box and transform used); the network-input counts (`num_points_raw`, `num_points_interpolated`, `num_points`, `num_unique_coors`, `cropped_by_ego_box`, `crop_applied`, `max_points_per_cell`); `plan` (`n`, `m`, `n_cap`, `m_cap`); `filter` (classes, threshold, points kept); `skipped` (the node's message) for an out-of-profile frame |
| `timing_ms` | dict | `preprocess`, `device` (staging + H2D + trace + D2H + un-permute), `postprocess`, `total` of this call |

`out.class_counts()` gives the points per class (raw + interpolated); `out.to_dicts()` one row per class
(`label`, `label_id`, `points`); `len(out)` the number of output points. The node's visualization cloud is
`model.config.palette` indexed by `label_ids` (`tt_frnet.host.postprocess.visualization_rgb`).

## Lifetime and information

- `model.close()` releases the trace and the device tensors and closes the chip if the model opened it; idempotent.
  `with` calls it for you; an unclosed model is closed when Python exits.
- `model.info`: weights (repo, tag, revision, path), device (`dispatch`, `grid`, `cores`, `num_command_queues`,
  `fallback`), variant, warm variants, warm-up times, runtime parameter defaults, the variant config (`frnet`), the
  trace and graph descriptions.
- Calls from several threads are safe: the device calls are serialised by a lock. One model per process per chip.
- `tt_frnet.viz.render_bev(points, label_ids, palette=...)` and `render_compare(...)` draw bird's-eye views (PIL, no
  OpenCV).
- `model.device_probs(plan)` returns the device's `pred_probs` `(N, 27)` of a prepared frame (tests and tools; call it
  under `model._lock`).

## Speed

Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, every `FRNET_*` knob at its default (= the serve pins)
(`code/scripts/bench.py`, 50 iterations, 2026-10-14, the optimized build; a shared host whose load moves the host
stages; the baseline release's numbers are in `OPT_BASELINE.md`):

| stage, p50 | ot128, shipped sample | ot128, PandaSet 019 f40 | qt128, shipped sample | qt128, PandaSet 019 f40 |
|---|---:|---:|---:|---:|
| `model(path)` end to end | 94.9 ms | 143.3 ms | 76.7 ms | 93.0 ms |
| host pre-processing (decode, crop, projection, fp16 interpolation, unique cells, device plan; numba) | 23.7 ms | 43.3 ms | 20.7 ms | 34.1 ms |
| H2D (9 inputs) | 3.01 ms | 5.44 ms | 4.07 ms | 4.06 ms |
| device trace, one forward | 61.51 ms | 81.44 ms | 43.61 ms | 45.08 ms |
| D2H (the raw TILE rows of `pred_probs`) | 1.21 ms | 2.51 ms | 1.94 ms | 1.84 ms |
| un-permute + host post-processing (argmax, filter; numba) | 3.6 ms | 7.9 ms | 4.2 ms | 6.0 ms |

- Back-to-back replays take 61.46 / 81.37 ms (ot128, shipped sample / PandaSet) and 43.55 / 45.02 ms (qt128):
  16.3 / 12.3 and 23.0 / 22.2 frames/s of device throughput (baseline release: 211.64 / 134.87 ms on PandaSet).
- Capacity buckets: each profile captures three traces, the full capacity (`N_cap` 160,000 / 120,000 points) and two
  smaller ones (ot128 `n40000`, `n80000`; qt128 `n29984`, `n60000`); a frame runs in the smallest trace that holds its
  points and cells. The ot128 sample (N 52,613) runs in `n80000`; PandaSet (N 145,961) and the qt128 sample
  (N 69,553) need the full trace. Valid rows are bit-identical in every trace.
- The shipped samples are synthetic scans: ot128 53,411 points (N = M = 52,613 after the crop, no interpolated point),
  qt128 45,783 points (N = 69,553 after interpolation, M = 18,926). PandaSet 019 frame 40 is a 107,146-point Pandar64
  sweep (N = 145,961 / M = 40,691 on ot128); it is not shipped.
- `OPT_REPORT.md` lists every optimization step with its A/B measurement and accuracy check.

## Tests

From the repository root, with `code/` on `PYTHONPATH` (or the package installed):

- Host (no device, no ttnn call): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_frnet/tests`. Inside the
  tt-models workspace the suite runs on its fake ttnn (`-p fake_ttnn_plugin`), which also enables `test_tt_host.py`;
  tests that need workspace-only oracles or data skip elsewhere, and FAIL inside the workspace when that data is
  missing. `test_reference_cpu.py` needs onnxruntime.
- Device (one p150): `python -m pytest -q -s code/tt_frnet/tests/test_pcc_device.py
  code/tt_frnet/tests/test_e2e_device.py` (per-module PCC against the fp32 CPU reference on replay outputs, the K1
  segment-max sites included; end-to-end agreement with ONNX Runtime on 17 OT128 / 5 QT128 frames, the shipped samples
  against their stored references, replay == eager, API == `/predict`). The public-frame gates run only inside the
  tt-models workspace, where their data lives: PandaSet, nuScenes and Autoware's sample rosbag, of which only PandaSet
  may be redistributed.
- The gates are frozen in `tests/*.gates.json` at the first green run and are never loosened.

## Limits

- Batch 1 on the chip; one model per process; one frame (one sweep) per call; one variant per model.
- The device input has a fixed size per profile (`N_cap` 160,000 / 120,000 points, `M_cap` 60,000 cells): a frame
  outside Autoware's profile ranges is skipped exactly as the node skips it.
- Label names 22 / 23: Autoware's `class_names` read `debris` / `stroller`, but the weights were trained with
  stroller = 22 and debris = 23 (AWML config). The port keeps Autoware's names for parity.
- Numerics: the device's per-point probabilities differ slightly from the fp32 reference, so points near a class
  boundary can take the other class (at least 98.49 % of the points agree on every public frame, 99.26 % on every gated frame; see the model card).
- Not ported: Autoware's `PointCloud2` layouts and the filtered cloud's `output_format` conversion; the API takes
  arrays or files and returns arrays.