README.md

---
tags:
- blackhole
- p150
- tt-dit-server
- tt-model-cache
- tt-model-catalog
- tt-model-container
license: apache-2.0
pipeline_tag: robotics
base_model:
- AutowareFoundation/diffusion_planner
---

# diffusion-planner-p150 on Tenstorrent p150

> Derived from [AutowareFoundation/diffusion_planner](https://huggingface.co/AutowareFoundation/diffusion_planner) (tag `v5.0`), the Diffusion Planner network that TIER IV trained for Autoware's [`autoware_diffusion_planner`](https://github.com/autowarefoundation/autoware_universe/tree/9ceaccf026c31ffc5319bc9eeb4bd7bede0af3fd/planning/autoware_diffusion_planner) trajectory planner, with unmodified weights. Diffusion Planner: Zheng et al., [arXiv:2501.15564](https://arxiv.org/abs/2501.15564). See the original model card for training details.

Diffusion Planner v5.0, a 14.55M-parameter diffusion-transformer planner: six MLP-Mixer entity encoders and a 6-block transformer fusion encoder over 564 scene tokens, a 3-block DiT decoder with adaLN (self-attention over the ego and 320 agents, cross-attention to the scene) evaluated 11 times by a 10-step DPM-Solver++(2M) loop, and a turn-indicator head; the whole plan runs on the device. The 15 planner tensors of the Autoware node go in (ego and neighbour histories, lanes with speed limits, route lanes, polygons, line strings, goal pose, ego shape, turn-indicator history, in the ego frame). Out come an 8 s ego trajectory (80 poses at 10 Hz with velocity and acceleration), an 8 s predicted path for every valid neighbour and a turn-indicator command, with the pre- and post-processing of the `autoware_diffusion_planner` node (input normalization, agent ordering, velocity smoothing, force stop, turn-indicator decision).

Runs on Tenstorrent **p150** (mesh `P150`).

Packaged and published with [tt-model-manager](https://github.com/tenstorrent/tt-model-manager) 0.1.0 (manifest schema 5.1).

## At a glance

| | |
| --- | --- |
| Hardware | p150 |
| License | apache-2.0 |

## Intended use

Ego trajectory planning and neighbour motion prediction for Autoware-style driving stacks: research, prototyping and offline evaluation of recorded or simulated scenes, with the outputs of one `autoware_diffusion_planner` node. The caller builds the planner tensors from its own ROS messages and Lanelet2 map.

## Prerequisites

- The [Tenstorrent CLI](https://github.com/tenstorrent/tt-cli/blob/main/docs/prerequisites.md), `tt`: `uv tool install tenstorrent` (or use tt-model alone, see the Quickstart)
- Docker
- Tenstorrent hardware: **p150**

## Quickstart

```bash
tt serve changh95/diffusion-planner-p150
```

`tt serve` downloads the Docker image and the [`AutowareFoundation/diffusion_planner`](https://huggingface.co/AutowareFoundation/diffusion_planner) at `423efde67f5414734da43a7ad856c17ceb8b51aa` weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs `Application startup complete`.

Without tt-cli: `tt-model serve changh95/diffusion-planner-p150`.

Send the sample scene shipped in this repo (a scripted scene on Autoware's Kashiwanoha map, 88 neighbours) to `/predict`, on the port `tt serve` reported:

```bash
curl -sLO https://huggingface.co/changh95/diffusion-planner-p150/resolve/main/code/tt_diffusion_planner/samples/kashiwanoha_dense.npz
printf '{"inputs": {"format": "npz", "data": "%s"}}' "$(base64 -w0 kashiwanoha_dense.npz)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
```

`code/tt_diffusion_planner/server/client.py` in this repo (standard library only) builds the same request from any
`.npz` of the 15 tensors, with per-request `params`.

## Capabilities

This package exposes the model's own HTTP server on port 20000, **not** an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

`POST /predict` takes one planning instant (the 15 planner tensors) as JSON and returns the ego plan, the
neighbour predictions and the turn-indicator command. `GET /health` and `GET /info` report the server and device
state (`/info` lists the 15 tensors with their shapes); `GET /v1/models` is a stub, not a chat API.

| request field | meaning |
| --- | --- |
| `inputs` | `{"format": "npz", "data": "<base64 .npz>"}` or `{"format": "json", "arrays": {name: nested list}}` with exactly the 15 raw tensors of the node's `create_input_data()`, float32, ego (`base_link`) frame, before normalization: `sampled_trajectories` [1,321,81,4], `ego_agent_past` [1,31,4], `ego_current_state` [1,10], `neighbor_agents_past` [1,320,31,11], `static_objects` [1,5,10], `lanes` [1,140,20,33], `lanes_speed_limit` [1,140,1], `route_lanes` [1,25,20,33], `route_lanes_speed_limit` [1,25,1], `polygons` [1,10,40,3], `line_strings` [1,60,20,4], `goal_pose` [1,4], `ego_shape` [1,3], `turn_indicators` [1,31], `delay` [1,1] |
| `params` (optional) | `velocity_smoothing_window` (default 8), `stopping_threshold` (0.3 m/s), `turn_indicator_keep_offset` (-1.25), `return_denoising_steps` (false) |
| `output_format` (optional) | `json` (default) or `npz` (adds the poses as a lossless base64 npz) |

Response for the sample scene (the trajectory cut to 3 of 80 rows, the predicted-agents array elided):

```json
{"model": "diffusion-planner-p150", "frame_id": "base_link", "num_poses": 80,
 "columns": ["x", "y", "yaw", "cos", "sin", "velocity", "acceleration"],
 "trajectory": [[0.3644, 0.01, -0.0169, 0.9964, -0.0169, 3.8033, -0.0392], [0.7626, -0.0021, -0.0412, 0.9968, -0.0412, 3.7994, -0.5496], [1.1517, -0.0281, -0.0674, 0.9942, -0.0672, 3.7444, -0.4683], "... 77 more rows"],
 "turn_indicator": {"command": 1, "command_name": "DISABLE", "keep_selected": true, "held": false, "logits": [-15.96, -5.12, -5.235, -0.764, 5.502], "probabilities": [1.7e-09, 8.4e-05, 7.5e-05, 0.0066, 0.9932]},
 "predicted_agents": {"format": "npz", "key": "predicted_agents", "dtype": "float32", "shape": [88, 80, 5], "data": "<base64 npz>"},
 "meta": {"predicted_agent_columns": ["x", "y", "yaw", "cos", "sin"], "force_stop": false,
          "valid_counts": {"ego": 1, "neighbor": 88, "static": 0, "lane": 123, "route": 17, "polygon": 0, "line_string": 60, "goal": 1, "ego_shape": 1, "turn": 1}},
 "timing_ms": {"preprocess": 2.4, "device": 22.8, "postprocess": 1.2, "total": 36.9, "decode": 8.4, "model_call": 26.7}}
```

- `trajectory`: 80 poses at 0.1-8.0 s in `base_link` (the ego frame of the inputs), metres, radians, m/s and
  m/s², post-processed like the node's `~/output/trajectory`: velocity from consecutive poses, a forward moving
  average over `velocity_smoothing_window` poses, force stop (poses frozen once the smoothed speed falls below
  `stopping_threshold`), acceleration by finite difference. `yaw` is what `tf2::getYaw` reads from the node's
  quaternion; `cos` / `sin` are the raw network outputs.
- `turn_indicator.command`: 0 NO_COMMAND, 1 DISABLE, 2 ENABLE_LEFT, 3 ENABLE_RIGHT.
- `predicted_agents`: one 80-point path (x, y, yaw, cos, sin) in `base_link` per non-empty neighbour row, in input
  order (`meta.predicted_agent_rows`).
- A Python API with the same outputs (`tt_diffusion_planner.DiffusionPlanner`) is described in
  [`code/PYTHON.md`](code/PYTHON.md), including what the caller keeps between plans; it needs a tt-metal build
  with the two patches in this repo's [`patches/`](patches/) directory.

### Example output

![Diffusion Planner on a Tenstorrent p150: lanes, neighbours with their predicted paths and the planned ego path in bird's-eye view on nuScenes scene-0061, Singapore](media/demo_tt.gif)

Plans computed by this package for 33 consecutive planning instants of nuScenes scene-0061 (Singapore: following a
van, then a left turn; 2 Hz, shown at 7 frames/s), drawn from above with the lanes, the neighbours' predicted paths
and the ego plan. Also as [MP4](media/demo_tt.mp4). **Non-commercial use only:** rendered from the nuScenes
dataset, © Motional AD Inc., CC BY-NC-SA 4.0 and the [nuScenes Terms of Use](https://www.nuscenes.org/terms-of-use);
Motional does not endorse this work. Attributions for every file in `media/`: [`media/ATTRIBUTION.md`](media/ATTRIBUTION.md).

## Expected performance

Agreement with the fp32 CPU reference of the same network (same weights, same pre- and post-processing, same
initial solver state); gates: ego plan max ≤ 1.0 m and mean ≤ 0.3 m over the 8 s, turn command equal, neighbour
paths' median per-agent max ≤ 1.5 m:

| eval | this package | fp32 CPU reference | scenes |
| --- | --- | --- | --- |
| Ego plan difference (max / mean over 8 s), sample scene `kashiwanoha_dense` | 2.0 cm / 0.8 cm, turn command equal | – | 1 |
| Ego plan difference, sample scene `straight_road` | 2.3 cm / 0.9 cm, turn command equal | – | 1 |
| Ego plan difference, gate set: worst max / worst mean (median of the means) | 0.346 m / 0.154 m (1.3 cm) | – | 99 |
| Turn command equal, gate set | 99 of 99 | – | 99 |
| Neighbour paths, median per-agent max difference, worst scene of the gate set | 0.094 m | – | 99 |
| Held-out scenes within every gate: worst ego max / worst mean | 50 of 50: 0.507 m / 0.174 m | – | 50 |
| Module output PCC, sample scenes: encoder categories / scene encoding / teacher-forced decoder steps (minimum) | 0.999948 / 0.999976 / ≥ 0.9999995 | – | 2 |
| Open-loop distance to the logged nuScenes drive at 8 s, ADE / FDE | 4.75 m / 11.87 m | 4.74 m / 11.87 m | 92 |
| Turn command equal to the logged blinker, nuScenes | 88 of 92 | 88 of 92 | 92 |

Gate set: the 2 shipped sample scenes, 5 scripted research scenes and 92 planning instants converted from nuScenes
v1.0-mini (Singapore and Boston). Held-out scenes, never used to choose the device numerics: the 33 consecutive
instants of nuScenes scene-0061, 4 full-capacity and noisy-initial-state scenes and 13 scenes at every agent-bucket
boundary. The open-loop rows compare with one human driver, not a planning benchmark; the paper's nuPlan
closed-loop benchmark was not run, and the weights repo publishes no dataset-level metric to compare with.

| speed (batch 1, warm) | input | median | throughput |
| --- | --- | --- | --- |
| Device time per plan, back-to-back trace replays, up to 31 neighbours | `straight_road` sample | 17.44 ms | 57.3 plans/s |
| Device time per plan, 32-63 neighbours | nuScenes scene-0103 instant | 19.25 ms | 51.9 plans/s |
| Device time per plan, 64-95 neighbours | `kashiwanoha_dense` sample (88 neighbours) | 20.00 ms | 50.0 plans/s |
| Device time per plan, 96-127 / 128-191 / 192-320 neighbours | synthetic: `kashiwanoha_dense` with neighbours replicated to 96, 127 / 128, 191 / 192, 320 | 21.13 / 22.33 / 26.66 ms | 47.3 / 44.8 / 37.5 plans/s |
| Python `model()` call (pre-processing, device, post-processing) | `kashiwanoha_dense` sample | 26.7 ms (p99 27.5) | 37.4 plans/s |
| Served `POST /predict`, server `timing_ms.total` | `kashiwanoha_dense` sample (152 kB request) | 37.2 ms | – |
| Served `POST /predict`, client round trip on loopback | `kashiwanoha_dense` sample | 41.6 ms | – |

Measured with this package's code on one p150 in an 8-vCPU AMD EPYC-Rome host: `code/scripts/bench.py` (100 warm
iterations per stage; device time = median of 3 rounds of 50 back-to-back trace replays) and 50 served requests of
the sample scene (after 5 warm-up requests) to this package's server app run on the host with the serve settings.
The device time depends on the number of neighbours: each plan runs the trace of the smallest agent bucket that
holds the scene, with outputs equal to the full-capacity trace. The host stages (pre-processing, tensor packing,
transfers, post-processing) take about 7 ms of the `model()` call, and request decoding 8.6 ms of the served time.

## Limitations

- Do not use it to drive or control a vehicle, in closed loop or as any safety function: it is not a certified
  Autoware component and has no functional-safety qualification.
- Do not use its outputs as a Diffusion Planner benchmark: the weights are TIER IV's, trained on synthetic and real
  driving data that is not published. On nuScenes, a domain they were not trained on, the plans are plausible but
  conservative (moving plans about 17 % shorter than the logged drive), on CPU and p150 alike.
- Not a ROS 2 node: it serves HTTP and Python only. Building the 15 planner tensors from ROS messages and the
  Lanelet2 map is the caller's. The API is stateless: the node's turn-indicator hold window, the agent and ego
  histories and the initial-solver-state prefix between plans are the caller's, and the node's guidance services
  (start, stop, centerline guidance) are not available.
- Batch 1, one plan per request; requests are served one at a time. The v5.0 capacities (320 neighbours, 140 lanes,
  25 route lanes, 10 polygons, 60 line strings) and 10 solver steps are fixed; the map entities are always
  computed at full capacity.
- One p150 only. The image runs device dispatch on the chip's Ethernet cores, so multi-chip meshes are not supported.
- Differences from the deployed node: the output stays in `base_link` (the node transforms it to `map` with the ego
  pose), the turn command is decided without the node's 1 s hold window, and the `delay` input is accepted and
  ignored (the node's multi-step mode does not read it). Outputs were compared with the fp32 CPU reference, not with
  TensorRT.
- Device numerics (fp32 residual streams, LayerNorm, attention and solver state; split bf16 + fp32 matmul
  operands; bf16 weights in the encoder MLPs) move the plan slightly from the fp32 reference, by a few cm on most
  scenes and up to 0.35 m (gate set) or 0.51 m (a synthetic scene with 192 neighbours) at the worst; the effect
  differs from scene to scene. Neighbour paths are gated only by their median, so single agents can differ more:
  on the `kashiwanoha_dense` sample the per-agent max difference is 8.6 cm median, 0.23 m at the 90th percentile
  and 3.72 m for the worst of 88 agents.

## Licensing

- Weights: [AutowareFoundation/diffusion_planner](https://huggingface.co/AutowareFoundation/diffusion_planner) at tag `v5.0` (commit `423efde67f5414734da43a7ad856c17ceb8b51aa`), Apache-2.0 per its model card. Downloaded by `tt serve` at that commit, not redistributed here. The upstream card states that TIER IV trained the models on its synthetic and real driving data; the dataset composition is not published.
- Pre- and post-processing ported from autoware_universe [`planning/autoware_diffusion_planner`](https://github.com/autowarefoundation/autoware_universe/tree/9ceaccf026c31ffc5319bc9eeb4bd7bede0af3fd/planning/autoware_diffusion_planner) (Apache-2.0).
- Port and serving code (`code/`) and the tt-metal patches in `patches/`: Apache-2.0.
- Sample scenes `code/tt_diffusion_planner/samples/`: `kashiwanoha_dense.npz` is derived from the Apache-2.0 Lanelet2 map [AutowareFoundation/map-carla-kashiwanoha](https://huggingface.co/datasets/AutowareFoundation/map-carla-kashiwanoha) (0.2.0) with a scripted ego, route and agents; `straight_road.npz` is generated by this repo's code. Both Apache-2.0.
- Demo media in `media/` are renders, not part of the package. The demo video and `media/*_NC.*` come from nuScenes: **non-commercial use only**, CC BY-NC-SA 4.0, © Motional AD Inc.; Motional does not endorse this work. The renders of the sample scenes: Apache-2.0. Full attributions: [`media/ATTRIBUTION.md`](media/ATTRIBUTION.md).

## Related packages

- The [Autoware collection](https://huggingface.co/collections/changh95/autoware-6ac890f41c8ca4ad23b6ef55): the other Autoware perception and planning models packaged for the p150.

## Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/diffusion-planner-p150/discussions — that is what reaches its author. A problem with the `tt` tooling itself: `tt report issue` (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

## Provenance

The exact sources the image was built from — `code/` in this repo is byte-identical to the model code inside the image:

| component | built from |
| --- | --- |
| tt-metal | [`44d66500520fda9f2c7060c0f6b41ec48f7ab37e`](https://github.com/tenstorrent/tt-metal/commit/44d66500520fda9f2c7060c0f6b41ec48f7ab37e) *(dirty tree — the image includes uncommitted changes)* |
| `code/` digest | `5da22b97bbf89b03` (sha256, first 16 hex digits) |
| built | 2026-10-16T22:45:25+00:00 by tt-model 0.1.0 |