Blame for code/PYTHON.md

# Python API: BEVFormer-small (Autoware tensorrt_bevformer) on Blackhole

Use this API from Python code (a pipeline, a notebook, a ROS 2 node wrapper). You do not need the HTTP server: the
API and the server share the decoders, the device graph and the post-processing, so the outputs and the speed are
the same.

## Install

Install the package on top of an environment that already has `ttnn` (a tt-metal `python_env` at `44d66500520`
with `patches/tt-metal-eth-dispatch.patch`, or the tt-model container). From the root of the model repository (the
directory that holds `pyproject.toml`, `README.md` and `code/`):

```bash
pip install -e .                        # the Python API
pip install -e ".[server,test]"         # + the HTTP server and the tests
```

The pip project is the repository's top-level `pyproject.toml`; it installs the package from `code/tt_bevformer` (there is no
`pyproject.toml` inside `code/`, because the container build copies `code/` over the tt-metal tree).

The package carries `tt_bevformer.ttaw`, the shared code of the Autoware ports to Blackhole (device open, trace runner,
decoders, model base class, HTTP app), vendored at the version recorded in `code/tt_bevformer/ttaw/VENDORED.json`.

| You want to run | Extras |
|---|---|
| the Python API | none |
| the HTTP server (`tt_bevformer.server.app`, see `SERVING.md`) | `server` |
| host tests (no device; device tests are skipped): `TT_VISIBLE_DEVICES=none python -m pytest -q code/tt_bevformer/tests` | `server,test` |
| device tests: `python -m pytest -q -s code/tt_bevformer/tests/test_pcc_device.py code/tt_bevformer/tests/test_e2e_device.py` | `test` |

## Quickstart

```python
from tt_bevformer import BEVFormer, load_sample

with BEVFormer.from_pretrained(device_id=0) as model:
    out = model(**load_sample("code/tt_bevformer/samples/synthetic_6cam.json"))   # from the repo root
print(out.to_dict())
```

`load_sample` reads a sample manifest (six image files, a calibration or a preset, the stream with the ego pose) into
the call's keyword arguments. The shipped sample is a synthetic test pattern (`code/tt_bevformer/samples/README.md`);
`examples/quickstart.py` runs it and writes the result and a bird's-eye-view PNG.

## `BEVFormer.from_pretrained(...)`

```python
BEVFormer.from_pretrained(
    model_id=None,          # HF repo or a local directory with the weights files; default changh95/bevformer-small-weights
    *,
    revision=None,          # default for the default repo: the validated commit 0e6d9d9c7b8 (branch main; no tags)
    variant=None,           # load-time variant (default); default $BEVFORMER_VARIANT or "default"
    device_id=None,         # chip to open; default $TT_DEVICE_ID or 0
    device=None,            # an already-opened ttnn device (tt_bevformer.device.open_device); close() does not close it
    dispatch=None,          # "eth" (p150 target, 12x10 grid) | "worker" (A/B only, 11x10) | "auto"; default $BEVFORMER_DISPATCH or "eth"
    num_command_queues=None,  # default $BEVFORMER_NUM_CQS or 1
    weights_dir=None,       # explicit local weights directory; no Hub access
    warmup_variants="default",  # trace variants to capture now; see "Warm-up"
    verbose=False,
    **compile_params,       # shape-defining options (documented per model below)
) -> BEVFormer
```

What it does: resolves the weights first, so a Hub problem never claims the chip (`weights_dir` >
`$BEVFORMER_WEIGHTS_DIR` > a local `model_id` directory > the HF snapshot at the pinned revision, restricted to
`bevformer_small_epoch_24.safetensors, provenance.json`, with an offline fallback to the cache), opens the chip (ETH dispatch, 12×10, 1 CQ;
the other open parameters are `DEVICE_DEFAULTS` in `tt_bevformer/device.py`, overridable with `BEVFORMER_*`), builds the
graph, then compiles and captures the metal trace of every warm-up variant. If ETH dispatch cannot open (tt-metal
without the patch), it warns and falls back to WORKER dispatch.

## Warm-up

`from_pretrained` captures the two trace variants of the frame (`image`: ResNet-101-DCN + FPN + the spatial
cross-attention value maps; `bev`: BEV queries, history, 3 encoder layers, 6 decoder layers, heads), then runs one
synthetic frame through the whole host + device path, so the first 1600×900 call compiles nothing. A camera size other
than 1600×900 (e.g. 1920×1080) compiles its host resize kernel in a background thread on first sight (about 6 s);
until it is ready those calls use the slower generic resize kernel. A new calibration pays its host geometry once
(cached per `lidar2img`). There is one variant (`default`); the steady-state frame (`use_prev_bev == 1`) has its own
`bev` trace. `warmup_variants="none"` skips the captures (the first call then fails: capture explicitly with
`model.warmup()`). The first boot on a machine also pays the ttnn kernel JIT (minutes, cached in
`~/.cache/tt-metal-cache` afterwards); a warm load takes about 10 s (10.4 s measured on 2026-10-13: weights 0.3 s,
graph 1.3 s, warm-up 4.5 s, the rest the device open). The warm-up frame uses the shipped `synthetic_6cam`
calibration preset.

## Precision: `fast` (default) and `accurate`

The load-time `BEVFORMER_*` knobs (read once by `from_pretrained`, from the environment or from keyword arguments
with the knob's lower-case name, e.g. `precision="HiFi4+fp32+l1acc:w=fp32"`) select the precision path. With no
variable set you get the server's default profile `fast` (PLAN D20 reduced precision: HiFi2, bf16 weights, bfp8
layer-3 weights, a bfp8 residual stream in the backbone, flash-SDPA decoder self-attention; judged by task-level
gates on nuScenes mini_val). For the full-precision `accurate` path (fp32 weights, HiFi4, the first release's policy;
the frozen per-stage PCC gates pass) set the 13 variables of the `accurate` serve profile in `tt-model.yaml`
(`serve_profiles`) before loading; SERVING.md section 3.4 lists them. `model.info` reports the knob values in use.

## Call: `model(...)`

| Argument | Type | Description |
|---|---|---|
| `images` | the six cameras | `CAM_FRONT`, `CAM_FRONT_RIGHT`, `CAM_FRONT_LEFT`, `CAM_BACK`, `CAM_BACK_LEFT`, `CAM_BACK_RIGHT` (any order; the network order is baked into its camera embeddings): a list of `tt_bevformer.io.CameraImage` / `{"camera", "image" \| "path" \| "data", "intrinsics", "T_ref_from_camera"}` dicts or a `{name: image}` mapping. Any size: images other than 1600x900 are resized like the node does (uint8 bilinear) |
| `calibration` | dict | per camera `intrinsics` (3x3, of the image as sent) and `T_ref_from_camera` (camera optical frame -> `frame_id`), or `{"preset": "synthetic_6cam"}` (the shipped rig); `frame_id`: `"base_link"` (default; then `T_ref_from_lidar` = the network's LIDAR_TOP -> base_link, default a virtual nuScenes LIDAR_TOP: -90 deg yaw, 1.84 m up) or `"LIDAR_TOP"`; `scale_intrinsics` (default true: K rescaled to 1600x900; false: the node's quirk) |
| `stream` | dict | `{"id", "reset", "timestamp_s", "T_world_from_ego"}`: the BEV history. A new id, `reset: true` or a timestamp gap above `BEVFORMER_MAX_GAP_S` (2 s) starts a fresh history (`use_prev_bev = 0`); `T_world_from_ego` (4x4, default identity) drives the ego-motion compensation (can_bus position / yaw deltas, TSA shift, prev_bev rotation). Up to `BEVFORMER_MAX_STREAMS` (4) ids keep their own device history (least recently used evicted) |
| `ego` | dict | optional CAN dynamics in base_link: `{"accel": [3], "rotation_rate": [3], "velocity": [3]}` (missing = 0; BEVFormer's training layout of `can_bus[7:16]`) |
| `score_threshold` | float [0, 1] | the node's `score_thre` (default 0.2) |
| `max_num` | int [1, 9000] | top-k over the 900 x 10 scores before the threshold (default 300) |
| `physical_boxes` | bool | false (default): the node's message (size `[w, l, h]`, yaw `-rot + pi`); true: size `[l, w, h]`, yaw `-rot - pi/2` (the heading of the length axis) |

### Input types

- Images: a path, PNG/JPEG bytes, a `PIL.Image`, a uint8 HxWx3 array or 3xHxW tensor, or a float array in [0, 1].
- Calibration: a dict in the `/predict` calibration envelope (SERVING.md section 3.1) or `{"preset": name}`
  (`code/tt_bevformer/calib/`); transforms as 4×4 matrices,
  `{translation, rotation_wxyz}` or Autoware `{x, y, z, roll, pitch, yaw}`.

## Output

`tt_bevformer.Output` = `ttaw.outputs.Detections3D` in the node's frame `LIDAR_TOP` (`frame_id`), rows sorted by
score: `boxes` `[N, 7]` (x, y, z gravity centre, size, yaw per `physical_boxes`), `scores`, `label_ids` (network class
0-9: `meta["class_names"]`), `labels` (the Autoware label the node publishes per class: bus, trailer and
construction_vehicle -> TRUCK; barrier, traffic_cone -> UNKNOWN), `velocities` `[N, 2]` (vx, vy in LIDAR_TOP),
`timing_ms` (`preprocess`, `device`, `device_run`, `postprocess`, `total`), `meta` (`class_names`, the box convention,
the raw network angle `rot` and the decoder `query` of every box, `stream` = `{id, fresh, use_prev_bev}`).
`out.to_dict()` is exactly the `/predict` body.

## Lifetime and information

- `model.close()` releases the traces and the device tensors and closes the chip if the model opened it; idempotent.
  `with` calls it for you; an unclosed model is closed when Python exits.
- `model.info`: weights (repo, tag, revision, path), device (dispatch, grid), variant, warm variants, warm-up times,
  runtime parameter defaults.
- Calls from several threads are safe. With `BEVFORMER_PIPELINE=1` (default) one caller's host half overlaps another's
  device stage, frames keep their arrival order; two callers on one stream measured 16.3 frames/s (1600×900) and
  18.1-18.5 frames/s (PandaSet). One model per process per chip.

## Speed

Measured with `code/scripts/bench.py` (ETH dispatch, 1 CQ, 12×10 grid, AICLK 1350 MHz, warm, batch 1; p50 of 30
iterations on a shared 8-core host; 2026-10-13, `VERIFICATION_OPT_2026-10-13.md`, OPT_BASELINE.md "After
optimization"):

| | first release | **`fast` (default)** | `accurate` |
|---|---:|---:|---:|
| `model(...)` end to end, 1600×900 images (synthetic) | 939 ms | **59.0 ms** | 88.9 ms |
| `model(...)` end to end, PandaSet 1920×1080 (decoded before timing): one frame / the f30-35-40 stream | 1,555 / 1,619 ms | **71.0 / 73.6 ms** | 101.9 / 104.2 ms |
| host pre-processing (request, the node's uint8 resize for 1920×1080, staging) | 303 / 919 ms | 3.4 / 12-15 ms | 3.5 / 15 ms |
| device incl. the uint8 image upload and the read-back (`timing_ms.device`, 1600×900) | 633 ms | 54.1 ms | 84.1 ms |
| trace `image` (pre-processing on the device, ResNet-101-DCN + FPN + SCA value maps) / trace `bev` (3 encoder + 6 decoder layers + heads), back to back | 245.0 / 240.5 ms | 34.9 / 14.5 ms | 60.9 / 18.7 ms |
| host post-processing | 1.4 ms | 1.4 ms | 1.4 ms |

Back to back, the two traces take **49.3 ms per frame** (20.3 frames/s device-bound; first release 485.6 ms) with
`fast`, 79.6 ms with `accurate`. `BEVFORMER_NUM_CQS=2` is an A/B option (about +0.9 ms of device time; no gain for
one request at a time). The many other `BEVFORMER_*` knobs are A/B switches of the optimization rounds
(OPT_REPORT.md); the card's numbers use the defaults.

## Limits

- Batch 1 on the chip; one model per process; six cameras in the nuScenes slot order (other rigs need a virtual
  LIDAR_TOP and rescaled K, see `calibration`).
- A rig whose cameras each see more than 6,144 BEV cells is refused with an `InputError` (HTTP 400): the static
  rebatching capacity of the spatial cross-attention (the node's dense graph accepts any rig; every rig tested fits).
- History state for `BEVFORMER_MAX_STREAMS` (4) stream ids; the history advances once per call (the node's cadence)
  and restarts on a new id, `reset` or a timestamp gap above `BEVFORMER_MAX_GAP_S` (2 s).
- Output in the node's LIDAR_TOP frame only; convert with the calibration's `T_ref_from_lidar` (SERVING.md 3.2).