Blame for code/PYTHON.md

# Python API: METEOR (TIER IV, AutowareFoundation/meteor) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the
API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are
the same.

## Install

Install the package on top of an environment that already has `ttnn`: a tt-metal `python_env` at `44d66500520`
with `patches/tt-metal-eth-dispatch.patch` and `patches/tt-metal-reshape-rm-sys1419.patch` applied, or the tt-model
container. From the root of the model repository (the directory that holds `pyproject.toml`, `README.md` and
`code/`):

```bash
pip install -e .                        # the Python API (numpy<2, pillow, pyyaml, onnx, huggingface_hub, safetensors, opencv-python-headless)
pip install -e ".[server,test]"         # + the HTTP server and the tests
```

The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_meteor` (there
is no `pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree). `ttnn`
and `torch` are not declared, so pip never replaces tt-metal's own build; tt-metal's `python_env` already has
OpenCV 4.8.1, which satisfies the `opencv-python-headless>=4.8,<4.12` requirement.

The package carries `tt_meteor.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner,
decoders, model base class, HTTP app), vendored at the version recorded in `code/tt_meteor/ttaw/VENDORED.json`
(0.24.1).

| You want to run | Extras |
|---|---|
| the Python API | none |
| the HTTP server (`tt_meteor.server.app`, see `SERVING.md`) | `server` |
| host tests (no device; the device tests are skipped): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_meteor/tests` | `server,test` |
| device tests: `python -m pytest -q -s code/tt_meteor/tests/test_pcc_device.py code/tt_meteor/tests/test_e2e_device.py code/tt_meteor/tests/test_variants_device.py` (most of them need the fp32 goldens of the development workspace and skip without them) | `test` |

## Quickstart

```python
from tt_meteor import METEOR, load_sample

with METEOR.from_pretrained(device_id=0) as model:
    out = model(**load_sample("code/tt_meteor/samples/synthetic_8cam.json"))   # 8 cameras + calibration + ego speed + stream
print(out.to_dict()["plan"]["mode"], [d["label"] for d in out.to_dicts()])
```

`examples/quickstart.py` runs the same snippet and writes the `/predict` JSON and a bird's-eye view
(`quickstart_bev.png`). The shipped sample is a synthetic test frame generated by this repository (see
`code/tt_meteor/samples/README.md`); feed your own rig's eight images and calibration for real use.

## `METEOR.from_pretrained(...)`

```python
METEOR.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default AutowareFoundation/meteor
    *,
    revision=None,          # default for the default repo: the validated commit 01a5f6d71df (tag v1.0)
    variant=None,           # load-time variant: "default" (the only one); default $METEOR_VARIANT
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_meteor.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $METEOR_DISPATCH or "eth"
    num_command_queues=None,  # default $METEOR_NUM_CQS or 1 (D14: a second queue gains nothing for synchronous calls)
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
    **compile_params,       # load-time knobs (below), e.g. input_norm="imagenet"
) -> METEOR
```

What it does: resolves the weights first, so a Hub problem never claims the chip (`weights_dir` >
`$METEOR_WEIGHTS_DIR` > a local `model_id` directory > the HF snapshot at the pinned revision, restricted to
`meteor_v157c3Z.onnx, meteor_v157.param.yaml, LICENSE, SHA256SUMS`, with an offline fallback to the cache), opens the
chip (ETH dispatch, 12×10, 1 CQ; the other open parameters are `DEVICE_DEFAULTS` in `tt_meteor/device.py`:
`l1_small_size` 32 KiB, `trace_region_size` 256 MiB, overridable with `METEOR_*`), reads the parameters of the ONNX
file by consuming node, builds the graph, then compiles and captures the metal trace. If ETH dispatch cannot open
(tt-metal without the patch), it warns and falls back to WORKER dispatch.

Load-time knobs (`compile_params` in lower case, or the environment variable `METEOR_<NAME>`; read once at load):

| knob | default | meaning |
|---|---|---|
| `input_norm` | `onnx` | `onnx` = the released graph's `/255` (ONNX Runtime parity); `imagenet` = the trained ImageNet mean / std (an upstream export defect); with `imagenet` an absent camera is zeroed after normalising, as in training |
| `depth_mean_bins` | `log` | `depth_mean` bin centres: log-spaced as exported (parity) or `linear` 1 + 1.25 b m (the trained bins) |
| `max_streams` | 16 | host temporal states kept (seg fusion, yaw tracks, mode hysteresis); the least recently used is dropped |
| `profile` | `accurate` | the precision profile (the served default is `fast`, see "Precision profiles"): `accurate` = the image branch, the 3D detection path and the box refiner in fp32 activations with every conv as three bf16 terms; `fast` = reduced precision, 2.7x faster on the device. A knob set explicitly wins over its profile's value |
| `image_precision` | `terms3` (`fast`: `bf16`) | image branch: `terms3` = fp32 activations, every conv as three bf16 terms; `bf16` = single-term bf16 activations with the port's fp32 weights at HiFi4 (bf16 weights at HiFi2 in the 2D detection, traffic-light and 2D segmentation heads) |

## Precision profiles

`METEOR.from_pretrained(..., profile="fast")` (or `METEOR_PROFILE=fast`; `tt serve` uses `fast` unless
`--profile accurate` is given) chooses the precision of the whole graph at load time:

| profile | numerics | device time per frame | judged by |
|---|---|---:|---|
| `fast` (served default) | image branch, det stem and box refiner as single-term bf16 activations (fp32 weights at HiFi4; bf16 weights at HiFi2 in the 2D heads), BEV trunk and heads bf16 activations with bf16 weights at HiFi2 where measured safe, bilinear resizes as one block-sparse kernel | 102.8 ms | task-level agreement with the fp32 CPU reference: 3D boxes recall / precision 0.971 / 0.989, worst planned-path deviation 0.18 m, same plan mode and traffic light on every frame, lane agreement >= 98.6 % (45 public and in-domain frames) |
| `accurate` (Python default) | fp32 activations and three-term convs in the image branch, the 3D detection path and the box refiner; fp32 weights, HiFi4 | 278.6 ms | the per-output agreement gates (PCC, argmax agreement, strict box matching) |

Both keep fp32 accumulation. `fast` moves the dense maps by about 1 % of pixels (lane, 2D segmentation and depth
argmax agreement >= 98.6 / 99.0 / 98.3 % against 99.3 / 99.6 / 99.3 % with `accurate`), so near-threshold boxes and
class borders flip more often; use `accurate` when per-output parity with the fp32 network matters.

## Warm-up

The load runs one eager frame of the whole graph (it compiles every kernel and fills the program cache) and then
captures it as ONE metal trace, `frame`, so no call compiles anything. The first call with a new
calibration also builds and uploads its lift tables (about 0.2 s once; see "Rig changes").
With a warm JIT cache the load takes about 40-46 s (`fast` 41.2 s: weights 0.3 s, graph build 1.3 s, eager frame and
capture 35.6 s; `accurate` 46.1 s). A cold kernel cache compiles for minutes (a new dispatch configuration on the build host:
ETH-2CQ 142 s, WORKER 11×10 768 s). `warmup_variants="none"` skips the capture (the first call then captures).

## Call: `model(...)`

| Argument | Type | Description |
|---|---|---|
| `images` | list or mapping | the cameras `CAM_FRONT_WIDE`, `CAM_FRONT_LEFT`, `CAM_FRONT_RIGHT`, `CAM_BACK_WIDE`, `CAM_BACK_LEFT`, `CAM_BACK_RIGHT`, `CAM_FRONT_NARROW`, `CAM_BACK_NARROW`, any order: `CameraImage`s, dicts `{"camera", "image", "intrinsics", "T_ref_from_camera"}` or `{name: image}`. Raw, unrectified frames of at least 768×432; larger frames are resized with OpenCV INTER_AREA semantics (bit-exact numpy port) and K is scaled per axis. The six wide / corner cameras are required; a missing or all-zero narrow camera is **absent**: a zero image plus its donor's K and pose (`CAM_FRONT_WIDE` / `CAM_BACK_WIDE`), the trained 7-camera configuration |
| `calibration` | dict | `{"cameras": {name: {"intrinsics", "T_ref_from_camera"}}}` (K of the image as sent; camera optical frame -> base_link: x forward, y left, z up, origin on the road) or `{"preset": name}` (`code/tt_meteor/calib/`); inline calibration of a camera wins |
| `ego_speed` | float | m/s, the graph's `v0` (required) |
| `stream` | dict | optional: `{"id", "reset", "timestamp_s", "T_world_from_ego"}` (or `"pose": [x, y, yaw]`): METEOR's host temporal post-processing per stream id (BEV seg fusion needs the pose; yaw smoothing; plan-mode hysteresis). A new id or `reset=True` starts fresh |
| runtime params | keyword | host post-processing only (`METEOR.RUNTIME_PARAMS`, METEOR's C++ renderer defaults): 3D `det3d_threshold` 0.15, `det3d_topk` 64, `vehicle_threshold` 0.35, `vru_threshold` 0.15, `bev_nms_iou` 0.3, `bev_nms_containment` 0.6, `stationary_logit_threshold` 0.0; 2D `det2d_threshold` 0.30, `det2d_topk` 48, `det2d_hide` "7" (road paint); `unk2d` True, `unk2d_threshold` (= det2d), `ground_z` 0.0; plan `mode_hysteresis` 0.35, `straight_margin` 1.0; `seg_fuse` True, `thin_road_edge` True, `yaw_smoothing` True; `heads` False (adds the dense heads to `to_dict("npz")`) |

`tt_meteor.load_sample(path)` turns a sample manifest (`samples/<name>.json`: image paths relative to the file, a
preset, the ego speed and a stream) into these keyword arguments.

### Input types

- Images: a path, PNG / JPEG bytes, a `PIL.Image`, a uint8 H×W×3 RGB array, or a float array in [0, 1].
- Transforms: 4×4 (or 3×4) matrices, `{"translation", "rotation_wxyz"}` (or `rotation_xyzw`), or Autoware
  `{"x", "y", "z", "roll", "pitch", "yaw"}`. Intrinsics: 3×3 K, 3×4 P or `{"fx", "fy", "cx", "cy"}`.

## Output

`model(...)` returns a `MeteorOutput` (`tt_meteor.Output`); `out.to_dict()` is the `/predict` JSON:

| key | content |
|---|---|
| `detections` (`out.to_dicts()`) | 3D BEV boxes sorted by score: `label` (VEHICLE / VRU), `label_id`, `score`, `center` [x, y] and `size` [length, width] in metres (base_link; METEOR predicts no z, height or velocity), `yaw` (rad, CCW from +x), `stationary`, `future` (6 × [x, y] at 0.5 .. 3 s, the agent's best mode), `future_mode` |
| `trajectory`, `columns` | the selected ego path, 6 × [x, y] at 0.5 s steps |
| `plan` | `mode`, `mode_probs` (softmax of the hysteresis-adjusted logits), `mode_logits`, `paths` (3 × 6 × [x, y]), `steer` (rad), `accel` (m/s²), `brake_prob`, `dt` |
| `detections_2d` | per camera: `label` (10 classes), `label_id`, `score`, `box_xyxy` (pixels at 768×432) |
| `unknown_obstacles` | [x, y] of 2D "obstacle" boxes placed on the ground plane (METEOR's `unk2d`) |
| `traffic_light` | `state` (none / green / yellow / red), `state_id`, `probs` |
| `lane`, `lane_classes` | the BEV lane map, uint8 [800, 500] as PNG (0.2 m cells, row 0 = +80 m ahead, column 0 = +50 m left; classes bg, road, sidewalk, crosswalk, laneline, stopline, road_edge, marking, parking), after the optional seg fusion and road-edge thinning |
| `stationary_head_healthy`, `meta` | METEOR's health check of the stationary head; `present` cameras, `v0`, `stream_id` |
| `heads` (`to_dict("npz")` with `heads=True`) | the dense outputs: `seg2d`, `depth` (bins), `depth_mean`, `occupancy` (class per voxel), `risk` (sigmoid), `stationary` |
| `timing_ms` | `preprocess`, `device` (tables + upload + replay + read + conversion), `postprocess`, `total` |

`out.path` is the selected path as an array; `out.lane` the lane map; `out.boxes3d` / `out.boxes2d` the box objects.

## Lifetime and information

- `model.close()` releases the trace and the device tensors and closes the chip if the model opened it; idempotent.
  `with` calls it for you; an unclosed model is closed when Python exits.
- `model.info`: weights (repo, tag, revision, path), device (dispatch, grid), variant, warm variants, warm-up times,
  runtime parameter defaults, the load-time knobs and the trace description.
- Calls from several threads are safe: the device calls are serialised. One model per process per chip.
- **Rig changes.** The lift megakernel's tables (15.8 MB: per BEV cell the cameras that see it, their pixel
  positions and fp32 bilinear and depth-bin weights) are built on the host per calibration (the last 4 rigs are
  cached) and written to the chip only when the calibration changes. A cached rig costs a 5 ms write (call 143 ms
  instead of 136 ms with `fast`, 324 ms instead of 317 ms with `accurate`); a rig seen for the first time also pays the host build once (about 0.2 s). A fixed rig pays it
  once. A rig that puts more than 4 cameras over one 0.4 m lift cell raises `InputError` (HTTP 400); load the model
  with `METEOR_LIFT=stock` for it (any rig, 44 ms slower per frame).

## Speed

Warm, batch 1, ETH dispatch, 1 CQ, 12×10, AICLK 1350 MHz, p50 of 30 calls on the PandaSet 019 sample on a shared
8-core host (load average 4.5-5.0), 2026-10-19 (the device rows do not depend on the input):

| stage | `fast` ms | `accurate` ms |
|---|---:|---:|
| device trace, one replay (the whole network) | 102.80 (back to back 102.77: 9.73 frames/s) | 278.69 (back to back 278.64: 3.59 frames/s) |
| upload of the eight cameras (8 MB uint8, whole image rows) | 1.1 | 1.1 |
| readback + conversion to the 19 outputs (views of the packed segments) | ≈ 18 + 2.4 | ≈ 18 + 2.5 |
| host pre-processing | 3.0 | 3.1 |
| host post-processing (METEOR's C++ decode rules, temporal state) | 9.5-15.6 | 15.3-20.8 |
| `model(...)` end to end | 130.5-136.0 (7.4-7.7 calls/s) | 312.1-317.3 (3.2 calls/s) |

The end-to-end rows move with the load of the shared host.

## Limits

- Batch 1 on the chip; one model per process; eight 768×432 camera slots in METEOR's order, a 400×250 lift grid at
  0.4 m and an 800×500 BEV at 0.2 m (fixed shapes of the released graph); at most 3 cameras per lift cell on the
  rigs validated (the default lift takes up to 4).
- The released graph is camera-only and single-frame (its temporal memory was baked out upstream); the temporal
  behaviour of METEOR's runtime is host post-processing.
- Numerics: bf16 / fp32 on the chip; outputs differ slightly from the fp32 reference with `accurate` and more with
  `fast` ("Precision profiles"; the README's performance tables).