| # Python API: SceneSeg (Autoware VisionPilot) on Blackhole |
|
|
| Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the |
| API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are |
| the same. |
|
|
| ## Install |
|
|
| Install the package on top of an environment that already has `ttnn` (a tt-metal `python_env` at `44d66500520` |
| with `patches/tt-metal-eth-dispatch.patch`, or the tt-model container). From the root of the model repository (the |
| directory that holds `pyproject.toml`, `README.md` and `code/`): |
|
|
| ```bash |
| pip install -e . # the Python API |
| pip install -e ".[server,test]" # + the HTTP server and the tests |
| ``` |
|
|
| The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_sceneseg` (there is no |
| `pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree). |
|
|
| The package carries `tt_sceneseg.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner, |
| decoders, model base class, HTTP app), vendored at the version recorded in `code/tt_sceneseg/ttaw/VENDORED.json`. |
|
|
| | You want to run | Extras | |
| |---|---| |
| | the Python API | none | |
| | the HTTP server (`tt_sceneseg.server.app`, see `SERVING.md`) | `server` | |
| | host tests (no device; device tests are skipped): see "Tests" | `server,test` | |
| | device tests: `python -m pytest -q -s code/tt_sceneseg/tests/test_pcc_device.py code/tt_sceneseg/tests/test_e2e_device.py` | `test` | |
|
|
| ## Quickstart |
|
|
| Run from the repository root (`code/tt_sceneseg/samples/highway_normal_1.png` is a path relative to it): |
|
|
| ```python |
| from tt_sceneseg import SceneSeg |
| |
| with SceneSeg.from_pretrained(device_id=0) as model: # weights -> your HF cache, trace captured |
| out = model("code/tt_sceneseg/samples/highway_normal_1.png") |
| |
| mask = out.mask # uint8 (H, W) at the input size: 255 = foreground (the VisionPilot topic) |
| print(out.class_counts()) # pixels per class of out.class_map (background / foreground / road) |
| print(out.to_dict()["meta"]) # the same JSON as POST /predict |
| ``` |
|
|
| `examples/quickstart.py` runs this snippet and writes the `/predict` JSON, the mask and a tinted picture. |
|
|
| ## `SceneSeg.from_pretrained(...)` |
| |
| ```python |
| SceneSeg.from_pretrained( |
| model_id=None, # HF repo or a local directory with the weights files; default changh95/sceneseg-p150 (weights/) |
| *, |
| revision=None, # default for the default repo: the validated weights commit a6829b246a0 |
| variant=None, # load-time variant (bgr, rgb); default $SCENESEG_VARIANT or "bgr" |
| device_id=None, # chip to open; default $TT_DEVICE_ID or 0 |
| device=None, # an already-opened ttnn device (tt_sceneseg.device.open_device); close() does not close it |
| dispatch=None, # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $SCENESEG_DISPATCH or "eth" |
| num_command_queues=None, # default $SCENESEG_NUM_CQS or 1 |
| weights_dir=None, # explicit local weights directory; no Hub access |
| warmup_variants="default", # trace variants to capture now; see "Warm-up" |
| verbose=False, |
| **compile_params, # none for this model: the network input is fixed at 640x320 |
| ) -> SceneSeg |
| ``` |
| |
| What it does: resolves the weights first, so a Hub problem never claims the chip (`weights_dir` > |
| `$SCENESEG_WEIGHTS_DIR` > a local `model_id` directory > the HF snapshot at the pinned revision, restricted to |
| `weights/SceneSeg_FP32.onnx, weights/provenance.json, weights/LICENSE, weights/NOTICE`, with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ; |
| the other open parameters are `DEVICE_DEFAULTS` in `tt_sceneseg/device.py`, overridable with `SCENESEG_*`), builds the |
| graph of the selected channel order, then compiles and captures the metal trace. If ETH dispatch cannot open |
| (tt-metal without the patch), it warns and falls back to WORKER dispatch. |
|
|
| `variant` is the **input channel order**, a load-time choice folded into the first conv's weights on the device: |
|
|
| | variant | what it is | |
| |---|---| |
| | `bgr` (default) | exactly what VisionPilot's `scene_seg_model` node computes: the deployed C++ pre-processing feeds B, G, R planes (with the ImageNet constants reordered to match) to the network | |
| | `rgb` | the order the network was trained with (`scene_seg_trainer.py` @ `ca58cb50`): clearly better masks, most of all on two-wheelers (README "Channel order") | |
|
|
| Both run the same device graph and are validated against the fp32 CPU reference of their own order. |
|
|
| ## Warm-up |
|
|
| `from_pretrained` returns a warm model: it builds the graph, runs it once eagerly (this first run compiles the |
| kernels and prepares the conv weights on the device) and captures the whole forward as ONE metal trace, so no call |
| compiles anything. There are no shape buckets: every image, whatever its size, is squashed on the host to the same |
| 640x320 uint8 canvas, so one trace serves every input. |
|
|
| - Measured on the shared host (2026-10-13, `logs/sceneseg/republish/final/`): with the JIT cache warm, a load takes |
| 18.5-18.7 s (the served app: graph build 11.4 s, which prepares the S1 and K11 weights on the host, and warm-up + |
| capture 2.8 s). With an empty JIT cache the first load took 275 s (measured before R8.1, which saves about 21 s of |
| graph build). The first call took 28-30 ms, the later ones 22.5-24 ms on the shipped sample. |
| - `warmup_variants="none"` (server: `SCENESEG_WARMUP=none`) skips the capture at load; call `model.warmup()` before |
| the first frame (it is idempotent). |
| - The JIT cache is tt-metal's (`~/.cache/tt-metal-cache`, or `TT_METAL_CACHE`); inside the container it is |
| `~/.cache/tt-model/sceneseg-p150/cache` of the host. |
|
|
| ## Call: `model(image, **params)` |
| |
| | Argument | Type | Description | |
| |---|---|---| |
| | `image` (first positional) or `images=[image]` | path, `bytes`, `PIL.Image`, uint8 array / tensor, float array, `CameraImage` | one camera image of any size (see "Input types") | |
| | `class_map` | bool, default `True` | also encode the 3-class map in `to_dict()` (the counts and `out.class_map` are always there). The node itself has no runtime parameter: its decision is a plain argmax | |
| |
| ### Input types |
| |
| - The image (first argument, or `images=[x]` with exactly one entry): a path to a PNG / JPEG file, encoded PNG / JPEG |
| `bytes`, a `PIL.Image`, a uint8 array or torch tensor in **RGB** order (`HxWx3`, `3xHxW`; `HxW` grey is replicated, |
| a 4th channel is dropped), a float array in [0, 1], or a `tt_sceneseg.io.CameraImage`. Any size from 2x2 up. |
| - **Channel order of the input.** Arrays are read as RGB and converted to BGR8, which is what the VisionPilot node |
| receives (`bgr8` image topic). An OpenCV or ROS `bgr8` array must be flipped first: `model(bgr[:, :, ::-1])`. |
| Encoded files and PIL images carry their own channel order. (This is independent of `variant`, which selects |
| the plane order the *network* is fed.) |
| - The node squashes the image to 640x320 with OpenCV's `cv::resize` (INTER_LINEAR, aspect ratio not kept); the port |
| reproduces it bit for bit on the host. SceneSeg's Python tooling also had a bicubic resize; it is not offered, |
| because the deployed node never used it. |
| - There is no calibration, no camera model and no temporal state: one image per call, as one node instance |
| subscribes to one image topic. Inline intrinsics or transforms in a `/predict` image entry are accepted by the |
| shared schema and ignored. |
| |
| ## Output |
| |
| `model(...)` returns a `tt_sceneseg.Output` (= `SceneSegMask`, a `ttaw.outputs.Mask2D`); `out.to_dict()` is exactly the |
| `POST /predict` body (SERVING.md section 3.2). |
| |
| | field | type | meaning | |
| |---|---|---| |
| | `mask` | uint8 `[H, W]` | what the node publishes on `/autoseg/scene_seg/mask`: mono8 at the input's size, 255 where the per-pixel argmax is class 1 (foreground), else 0; resized back from 640x320 with INTER_NEAREST | |
| | `class_map` | uint8 `[H, W]` | the 3-class argmax on the same grid: 0 background (incl. sky), 1 foreground objects, 2 drivable road (a by-product the node does not publish) | |
| | `class_names` | tuple | `("background", "foreground", "road")` | |
| | `meta` | dict | `channel_order`, `source_hw`, `network_hw` `[320, 640]`, `foreground_class` 1, `mask_value` 255, `foreground_fraction` | |
| | `timing_ms` | dict | `preprocess`, `device` (H2D + trace + D2H), `postprocess`, `total` of this call | |
|
|
| `out.class_counts()` gives the pixels per class of `class_map`; `out.to_dicts()` the same as |
| `[{"label", "label_id", "pixels"}]`. Ties between classes resolve to the lowest index, as the node's strict `>` scan does. |
|
|
| ## Lifetime and information |
|
|
| - `model.close()` releases the trace and the device tensors and closes the chip if the model opened it; idempotent. |
| `with` calls it for you; an unclosed model is closed when Python exits. |
| - `model.info`: model name, weights (repo, revision, path, license), device (`dispatch`, `grid`, `cores`, |
| `num_command_queues`, `fallback`), variant and `channel_order`, warm variants, warm-up times, runtime parameter |
| defaults, plus `graph` (the device graph description), `trace` and `knobs` (the build-time `SCENESEG_*` knobs). |
| - Calls from several threads are safe: the device calls are serialised by a lock. One model per process per chip. |
|
|
| ## Speed |
|
|
| Warm calls, batch 1, ETH dispatch, 1 CQ, 12x10, `bgr`, the published numerics (`code/scripts/bench.py`, 100 |
| iterations, 2026-10-13 on a shared host; `OPT_BASELINE.md` "After optimization"): |
|
|
| | stage | shipped sample 727x414 PNG | PandaSet 1920x1080 JPEG | |
| |---|---:|---:| |
| | `model(path)` end to end, p50 (p99) | 22.54 (23.38) ms | 30.00 (32.01) ms | |
| | host decode + squash to 640x320 (PIL decode; exact `cv::resize` as a numba twin on 4 threads) | 8.53 ms | 13.75 ms | |
| | H2D of the uint8 640x320 canvas (320 rows of 1,920 B) | 0.21 ms | 0.34 ms | |
| | device trace, one forward | 12.63 ms | 12.64 ms | |
| | D2H (the uint8 class map) | 0.23 ms | 0.30 ms | |
| | host post-processing (INTER_NEAREST to the source size, numba twin) | 0.62 ms | 3.06 ms | |
| |
| - Back-to-back replays take 12.58 ms per frame (79.5 frames/s of device throughput); a synchronous call gives |
| 44.4 frames/s on the sample. The `rgb` profile replays in the same time (e2e p50 22.66 ms on the sample). |
| - The first release measured 90.85 ms (sample) and 116.12 ms (PandaSet) end to end and 61.06 ms per replay; |
| `OPT_REPORT.md` lists every step in between. |
| - A raw RGB array skips the decode. The device path does not depend on the image size; the host stages grow with it. |
|
|
| ## Tests |
|
|
| From the repository root, with `code/` on `PYTHONPATH` (or the package installed): |
|
|
| - Host (no device, no ttnn call): `TT_VISIBLE_DEVICES=none python -m pytest -q |
| code/tt_sceneseg/tests/test_bundle_host.py code/tt_sceneseg/tests/test_io_host.py |
| code/tt_sceneseg/tests/test_server_host.py code/tt_sceneseg/tests/test_host_preprocess.py |
| code/tt_sceneseg/tests/test_host_postprocess.py code/tt_sceneseg/tests/test_reference_host.py`. Inside the |
| tt-models workspace the suite runs on its fake ttnn (`-p fake_ttnn_plugin`), which also enables |
| `test_tt_graph_host.py` and `test_tt_layers_host.py`; `test_reference_cpu.py` (the CPU reference against ONNX |
| Runtime) needs `onnxruntime`. |
| - Device (one p150): `python -m pytest -q -s code/tt_sceneseg/tests/test_pcc_device.py |
| code/tt_sceneseg/tests/test_e2e_device.py` (per-module PCC against the fp32 CPU reference, end-to-end agreement on |
| the shipped tutorial and comma10k samples, replay == eager, API == `/predict`, the stored-reference smoke gates; |
| `SCENESEG_VARIANT=rgb` for the other order). The public-frame gates (23 frames with logits, 415 held-out frames) |
| run only inside the tt-models workspace, where their data live: PandaSet, nuScenes and KITTI frames, of which |
| nuScenes and KITTI may not be redistributed. There, a missing input fails the test instead of skipping it. |
| - The gates are frozen in `tests/*.gates.json` at the first green run and are never loosened. |
|
|
| ## Limits |
|
|
| - Batch 1 on the chip; one model per process; one image per call. |
| - The network input is fixed at 640x320 (the SceneContext reshape hard-codes the 10x20 stride-32 grid, batch 1); any |
| source image size is accepted, squashed on the host without keeping the aspect ratio, and the mask is resized back |
| with nearest neighbour. |
| - Only `SceneSeg_FP32.onnx`, the network the node deploys. The INT8 QAT ONNX (never deployed; it disagrees with FP32) |
| and SceneSegLite (a different 19-class network) are not supported. |
|
|
