ttaw API guide (v0.23.0)
ttaw is the shared code of the 13 Autoware ports to one Blackhole p150b (PLAN.md section 1). Source of truth:
common/ttaw/. Every bundle gets a vendored copy at bundles/<model>-p150/code/<pkg>/ttaw/ and imports it
relatively (from .ttaw.trace import TraceRunner). Tests: common/tests/host (CPU, fake ttnn) and
common/tests/device (p150, through bin/devrun). Changes: common/CHANGELOG.md.
Rules that keep one tree valid top-level and vendored:
- relative imports only; importing any module never imports ttnn / torch / onnx, opens no device, reads no
environment and touches no network (host tests and the image's
verify:step import everything without a device); - every optimization knob is read once at model build and has an env A/B switch (
ttaw.knobs); - kernel
.cppfiles are package data underops/kernels/, located withops.kernel_path(name).
Contents: 1. bundle wiring - 2. device - 3. TraceRunner - 4. tensors - 5. weights - 6. precision and knobs - 7. metrics, goldens, gates - 8. profiling - 9. I/O and outputs - 10. ModelBase - 11. server - 12. vendoring - 13. measured facts and pitfalls - 14. image pre-processing (C14) - 15. LiDAR host pipeline (C10-C12, C15, C16) - 16. CNN ops: conv builders (C17), up-sampling and interpolation (C18) - 17. attention (C20) - 18. LiDAR device modules: gather-form scatter (C19), SECOND + SECONDFPN (C24), CenterHead (C25) - 19. grid_sample helpers (C23) - 20. ResNet builders (C27) - 21. query heads: top-k (C21), heatmap peaks (C22), TransFusion query head (C26) - 22. pillar feature net and input staging (C24 companion) - 23. segment reductions (K1) - 24. sparse-conv rulebooks (C13) - 25. gather-GEMM sparse encoder (C28).
1. Wiring a bundle (thin wrappers over ttaw)
python common/tools/vendor.py bundles/<model>-p150 # committed ttaw/ (HEAD) -> code/<pkg>/ttaw + VENDORED.json
python common/tools/vendor.py bundles/<model>-p150 --check # drift report (exit 1 on drift; section 12)
research/BUNDLE_TEMPLATE is the canonical wiring (instantiate_bundle.py --out renders it and vendors ttaw; the
values file's INPUT_KIND picks its flavour: lidar as below, camera, multi_camera, lidar_camera, planner
with their own api.py / io.py / stub / smoke / tests, research/BUNDLE_TEMPLATE/TEMPLATE_NOTES.md "Flavours"):
device.py and io.py bind ttaw to the model, api.py subclasses ModelBase, server/app.py calls create_app,
and the stdlib server/client.py / server/smoke_test.py run ttaw/server/{client,smoke}.py by path:
# code/<pkg>/device.py: ttaw.device bound to the port's validated open parameters (the model class uses the same dict)
from .ttaw.device import DeviceConfig, close_device, describe_device # noqa: F401
DEVICE_DEFAULTS = {"num_command_queues": 1, "l1_small_size": 32768, "trace_region_size": 64 << 20}
def device_config(**overrides): ... # DEVICE_DEFAULTS < <ENV>_* / TT_DEVICE_ID < non-None overrides
def open_device(device_id=None, *, dispatch=None, **overrides): ... # device_config(...).open()
# code/<pkg>/io.py: ttaw.io re-exported, decode_points / load_points rebound to the model's layout
from .ttaw.io import * # noqa: F401,F403
DEFAULT_POINT_FIELDS = ("x", "y", "z", "intensity")
# code/<pkg>/api.py (section 10 has the full hook contract)
from .ttaw.api_base import ModelBase
class CenterPoint(ModelBase):
MODEL_NAME, ENV_PREFIX = "centerpoint-p150", "CENTERPOINT"
DEVICE_DEFAULTS = device.DEVICE_DEFAULTS
...
# code/<pkg>/server/app.py (section 11)
from pathlib import Path
from ..api import CenterPoint
from ..ttaw.server.app import ServerSpec, create_app, parse_mesh_shape # noqa: F401 (tt-model.yaml verify: line)
app = create_app(ServerSpec(model_name=CenterPoint.MODEL_NAME, env_prefix="CENTERPOINT", model_cls=CenterPoint,
task="LiDAR 3D object detection", default_weights=CenterPoint.DEFAULT_REPO,
calib_dir=Path(__file__).resolve().parents[1] / "calib"))
pyproject.toml package data: "<pkg>.ttaw" = ["API.md", "VENDORED.json"], "<pkg>.ttaw.ops" = ["kernels/*.cpp", "kernels/*.hpp", "kernels/*.h"]. The tt-model extra_code path <pkg> already ships the sub-package.
2. Device (C01, ttaw.device)
from .ttaw.device import DeviceConfig, describe_device, device_session, open_device, compute_grid
with device_session(dispatch="eth", num_command_queues=2, trace_region_size=64 << 20) as dev:
print(describe_device(dev)) # {'dispatch': 'eth', 'grid': '12x10', 'cores': 120, 'num_command_queues': 2,
# 'arch': 'blackhole', 'fallback': None, 'eth_patch': True, ...}
gx, gy = compute_grid(dev) # never hard-code 12x10: program configs read the grid
cfg = DeviceConfig.from_env("CENTERPOINT", num_command_queues=1) # <ENV>_DISPATCH / _NUM_CQS / _L1_SMALL /
dev = cfg.open() # _TRACE_REGION / _WORKER_L1_SIZE, TT_DEVICE_ID
| name | signature / meaning |
|---|---|
open_device |
(device_id=0, *, dispatch="eth", num_command_queues=1, l1_small_size=32768, trace_region_size=64<<20, worker_l1_size=None, allow_fallback=True) -> ttnn device. dispatch: eth (12x10), worker (11x10, A/B only), auto (ETH if the patch marker is in tt_metal/impl/dispatch/topology.cpp, else WORKER + RuntimeWarning). A failed ETH open warns and falls back to WORKER; allow_fallback=False re-raises (use it in container smoke / CI). Warns if ETH does not give 12x10 |
DeviceConfig |
frozen dataclass of the arguments above; .from_env(prefix, env=None, **defaults), .open() |
describe_device |
(device, device_id=None) -> dict: dispatch (used), dispatch_requested, fallback, num_command_queues, grid "12x10", grid_x, grid_y, cores, arch, open sizes, eth_patch. dispatch="unknown" for devices not opened by ttaw |
open_info(device) |
what open_device recorded for this device object ({} for a device opened elsewhere; the record is tied to the object, never to a reused id) |
close_device / device_session |
close (syncs first; idempotent: a second close of the same device is a no-op) / context manager that always closes |
resolve_dispatch(dispatch, *, home=None), eth_dispatch_patch_present(home=None), tt_metal_home() |
patch detection (file read; TT_METAL_HOME, TT_METAL_RUNTIME_ROOT, or the tree holding ttnn, found without importing it) |
compute_grid(dev) / core_grid(dev) / full_core_range_set(dev) |
(x, y) / ttnn.CoreGrid / one-rectangle ttnn.CoreRangeSet of the whole grid |
| constants | DISPATCH_MODES, PATCH_MARKER, DEFAULT_L1_SMALL_SIZE, DEFAULT_TRACE_REGION_SIZE, P150_ETH_GRID |
worker_l1_size is absolute bytes of allocatable L1 per core; tt-metal's default is computed at open time
(about 1,461,248 B on this tree, TT_PLATFORM.md section 1). Shrinking it grows the kernel-config ring buffer.
3. Traces (C02, ttaw.trace)
Every device stage runs inside traces. TraceRunner owns the persistent tensors, the warm-up, the captures, the
replays and the readback.
import ttnn
from .ttaw.trace import TraceRunner, pack_outputs
runner = TraceRunner(device, num_command_queues=2, name="centerpoint") # CQ count defaults to the open's
runner.add_input("pillars", shape=(1, 1, 40000, 64), dtype="bfloat16") # ROW_MAJOR in DRAM by default
runner.add_input("index", init=warm_idx, dtype="uint32") # give real data if the graph gathers
runner.add_param("score_thr", 0.35) # RT-dev: fp32 [1,1,1,1] TILE
runner.add_state("prev_bev", shape=(1, 1, 22500, 256), dtype="bfloat16", pingpong=True) # temporal models
def forward(ctx): # ctx[name]: input / param / state read buffer; no host I/O in here
x = model.backbone(ttnn.to_layout(ctx["pillars"], ttnn.TILE_LAYOUT))
bev = model.temporal(x, ctx["prev_bev"])
ctx.write_state("prev_bev", bev) # ttnn.copy into the state's write buffer (part of the trace)
heat = ttnn.gt(model.head(bev), ctx["score_thr"])
return pack_outputs({"heat": heat, "boxes": model.box(bev)}) # one D2H for several outputs
runner.add_variant("default", forward) # one trace per variant (buckets: "n8k", "n16k", ...)
runner.capture() # warm-up of every variant, then strict captures
out = runner("default", inputs={"pillars": p, "index": idx}, params={"score_thr": 0.4}) # {"heat": ..., "boxes": ...}
runner.release() # or `with TraceRunner(...) as runner:`
What it guarantees. Persistent tensors exist (with defined contents) before the first capture; every variant is
warmed warmup_runs times before any capture, and so are the runner's own eager programs (the stage_inputs
staging copy, the state <-> bank copies) and the callables registered with add_eager_warmup(fn); capture runs with
device.set_program_cache_misses_allowed(False)
(a miss raises ... program cache miss occurred, but cache misses are forbidden naming the op), end_trace_capture
runs in finally and a failed capture is released, and the program-cache size must not change during capture;
adding a variant after a capture releases all traces, warms the new one and recaptures everything; warm-up writes to
states are undone (reset_state) before capture; unchanged params are not re-uploaded.
No program may be compiled after the first capture. A new program's kernel binaries are a DRAM buffer allocated
when it is first enqueued, in the address space of the traces' freed intermediates, so a replay can overwrite them
(tt-metal tech_reports/AdvancedPerformanceOptimizationsForModels/TraceCorrectness.md: corruption or a hang). Any
eager op the model runs between replays (host-fallback glue, an eager ttnn.to_layout, tensors.upload_u8,
fp32_island, ...) must therefore run once before the first capture: register it with add_eager_warmup(fn). The
runner refuses its own eager work that compiles after a capture (stage_fn copies, save_state / load_state,
reset_state from a device tensor of another spec, run_eager) with RuntimeError: ... compiled a new program after capture, and every port runs its device tests once with TT_METAL_TRACE_ALLOC_TRACKING=1 (the tracker refuses a
replay while such a buffer is alive; ttaw 0.2.0's stage_inputs path tripped it:
logs/ttaw/review_old_staged_tracking_probe.log).
1CQ / 2CQ. 1CQ: uploads and replays on CQ0. 2CQ: CQ1 waits for the last replay's event, uploads, records; CQ0
waits, replays, records (the check_dispatch.py pattern; the events exist even after a partially failed capture()
left some variants runnable). stage_inputs=True (2CQ only) is the tt_cnn executor
pattern: CQ1 writes DRAM staging copies while the previous replay still runs, an eager ttnn.copy (or your
add_input(..., stage_fn=lambda staging, dst: ...), e.g. a reshard into a sharded L1 input) moves them into the
trace inputs on CQ0; that copy is compiled in the first capture() (a staging buffer mirrors every
write_input). read(variant, cq_id=1) (2CQ) waits for that trace's completion event on the host and reads on CQ1,
so CQ0 can already run the next segment.
Segments. Split a long graph into variants run back to back; hand intermediates from segment k to k+1 through an
in-place state (ctx.write_state("mid", t) in seg1, ctx["mid"] in seg2): a variant cannot see another variant's
outputs at warm-up time, a state buffer exists from the start.
Ping-pong state. pingpong=True allocates two buffers and captures two traces per variant (phase 0 reads A /
writes B, phase 1 the reverse). A variant that writes the ping-pong states must write all of them ("stepping"); the
phase flips after each run of a stepping variant; read-only variants follow the current phase. In-place states
(default) read and write one buffer, so read what you need before the write. Reset with reset_state(name, value)
(host data, or a device tensor copied on the device). write_state traces one ttnn.copy; compute the new state
straight into ctx.write_target(name) (ttnn.add(old, x, output_tensor=ctx.write_target("mem")), then
ctx.write_state("mem", new)) and no copy is traced (probe P13: one program less per state per frame).
write_state / write_output check shape, layout (ttnn.copy keeps it) and dtype changes (TILE only) before tracing.
For a ring buffer use ttnn.copy / slices into a fresh tensor and write_state; ttnn.experimental.slice_write
into a TILE state re-binds the handle and breaks traces (probe P13).
Per-stream state (PLAN.md D16). A trace bakes one set of state addresses, so several stream.ids share them:
add_state(..., banks=S_MAX) allocates S_MAX DRAM banks of the state before capture, save_state(name, i) /
load_state(name, i) copy between the state and bank i (eager ttnn.copy on CQ0, ordered after the enqueued
replays; ~80 us for a StreamPETR-size state), and StreamBanks is the policy every temporal bundle uses:
from .ttaw.trace import StreamBanks
runner.add_state("prev_bev", shape=(1, 1, 22500, 256), dtype="bfloat16", pingpong=True,
banks=S_MAX if S_MAX > 1 else 0) # S_MAX = 1 at the first publish
streams = StreamBanks(runner, ["prev_bev"], max_streams=S_MAX, on_full="reject", max_gap_s=2.0)
fresh = streams.select(stream.get("id", "default"), reset=stream.get("reset", False),
timestamp_s=stream.get("timestamp_s")) # in _prepare, under the model lock
params = {"use_prev_bev": 0.0 if fresh else 1.0} # first-frame RT-dev params
select returns True when the stream starts fresh (new id, reset=True, timestamp backwards or a gap above
max_gap_s) after resetting its states to init; a new id beyond max_streams raises InputError (HTTP 400) with
on_full="reject" or takes the least recently used stream's bank with "evict" (S_MAX=1: restart the one state).
forget(id), streams (most recent first), active, describe() (for model.info).
Outputs. A variant returns either the tensors its last ops produce (allocated during the capture, alive as long
as the trace) or persistent outputs: add_output(name, shape=...) allocates a buffer before any capture and
ctx.write_output(name, value) copies into it inside the trace and returns it. Persistent outputs keep their address
across recaptures, can be shared by several variants (shape buckets with one readback) and are never overwritten by
another variant's replay; they cost one ttnn.copy per output. Op-produced outputs of a variant captured later can
be overwritten when an earlier-captured variant replays: read them before running another variant (read and
__call__ return copies), and give segments read on CQ1 while CQ0 already replays the next one persistent outputs.
Warm-up and release free op outputs without force, so an output that shares memory with something the runner
does not own (a view of a weight) is never freed by the runner.
TraceRunner member |
signature / meaning |
|---|---|
| constructor | TraceRunner(device, *, num_command_queues=None, warmup_runs=1, stage_inputs=False, forbid_cache_misses=True, alloc_tracking=None, name="model") |
add_input |
(name, init=None, *, shape=None, dtype="bfloat16", layout=ROW_MAJOR, memory_config=DRAM, stage_fn=None) -> device tensor. init: numpy / torch / ttnn host tensor (warm-up data), else zeros |
add_param |
(name, value=0.0, *, shape=(1,1,1,1), dtype="float32", layout=TILE, memory_config=DRAM) -> device tensor; broadcasts in ttnn binary ops |
add_state |
(name, init=None, *, shape=None, dtype="float32", layout=TILE, memory_config=DRAM, pingpong=False, banks=0) -> buffer or (A, B) |
add_output |
(name, *, shape, dtype="float32", layout=TILE, memory_config=DRAM) -> persistent output buffer (zeros) |
add_variant |
(name, fn, *, warmup_runs=None); fn(ctx) returns a device tensor, a Packed, or a list / tuple / dict of them |
add_eager_warmup |
(fn): eager device work the model runs between replays; fn() runs in the first capture(), before any trace (refused after it) |
capture() |
warm-up + capture of all pending variants (idempotent) |
run(variant=None, inputs=None, params=None) |
upload + non-blocking replay; returns the device outputs |
read(variant=None, *, cq_id=0, as_torch=False) |
blocking read into preallocated host tensors -> numpy (or torch); Packed -> {name: array} |
__call__(variant=None, inputs=None, params=None, *, as_torch=False) |
run + read |
upload(inputs=None, params=None) / replay(variant=None, n=1) |
the two halves of run (benches, pipelining) |
run_eager(variant=None, inputs=None, params=None) |
same function without a trace, buffers freed before returning: replay-vs-eager bit checks |
set_params(**values) / write_input(name, value) |
queue param values for the next run / upload an input now (CQ0; with 2 CQs the event the next CQ1 upload waits for is re-recorded after it) |
reset_state(name=None, value=None) / read_state(name) / state_buffer(name) / phase |
state control (on CQ0, ordered after enqueued replays); value may be a device tensor |
save_state(name, bank) / load_state(name, bank) |
state <-> bank copies on CQ0 (add_state(banks=...); D16) |
outputs(variant=None, phase=None), trace_ids(), describe(), timings_ms, phases, captured |
introspection (describe() is JSON-able: put it in model.info) |
release() |
release traces and persistent tensors (idempotent; also __exit__) |
TraceContext (the ctx of a variant): ctx[name] (input, param, persistent output, or a state's read buffer),
ctx.state(name), ctx.write_target(name), ctx.write_state(name, value), ctx.write_output(name, value) -> buffer, ctx.variant, ctx.phase, ctx.capturing, ctx.device. Registering inputs / states / variants from
inside a variant function (during warm-up or capture) raises.
StreamBanks(runner, states, *, max_streams=1, on_full="reject", max_gap_s=None): .select(stream_id, *, reset=False, timestamp_s=None) -> fresh, .forget(id), .streams, .active, .describe() (above).
pack_outputs(tensors: dict, *, dtype="float32", align=32, row_elems=None) -> Packed: one ROW_MAJOR tensor holding
every output (typecast to dtype; float32 is exact for bf16 and integers < 2**24), read with ONE device-to-host copy;
TraceRunner.read unpacks it, Packed.layout.unpack(flat) gives {name: array} (views) from any array of
layout.total elements (PackLayout(entries, total, rows, row_elems), .shape = the device shape; PackEntry(name, offset, numel, shape, pitch=0) indexes the elements in row-major order; pitch > 0: the tensor's rows of
shape[-1] elements are stored pitch elements apart, zero-padded, and unpack / entry.view(flat) return a
strided view). Layout (since 0.15.0, YOLOX PORT_LOG Q9):
- packed total <=
SINGLE_ROW_MAX_ELEMS(131072 = 512 KiB of fp32): one[1, 1, 1, total]row, each tensor flattened and zero-padded toalignelements -- exactly the layout and programs of 0.1.0-0.14.0; - above it, or with
row_elems=R(a multiple of 32):[1, 1, rows, R], R =PACK_ROW_ELEMS(1024) by default. Each tensor starts on a row boundary and is zero-padded to whole rows (a tensor of at most 64 KiB is flattened and padded by < R elements; a larger[.., n, c]gets zero rows appended untiln * cfills whole rows of R, i.e. n rounded up to a multiple ofR / gcd(c, R); when that wastes more than 1/8 of the tensor beyond the best alternative -- few rows of an awkward width: 0.15.0-0.18.0 packed[1, 1, 2, 20001]fp32 as 80 MB -- it is flattened if its padded row fits 128 KiB, else its rows are zero-padded to a pitchp= c rounded up to a power of two dividing R, or to R, the smallest segment: onettnn.padof the row and last dims, then the reshape into rows of R; segments stay below 1.4x the tensor + one row, typical shapes keep the contiguous layout), then the segments are concatenated on the row dim. No ROW_MAJOR page above 128 KiB is reshaped, so outputs of tens of MB pack; a flat vector[1, 1, 1, N]wider than that is cut intottnn.slicechunks; any other tensor with rows wider than 128 KiB raisesValueError(give it a narrower last dim). Measured (p150b,tests/device/test_pack_outputs_device.py, two runs, 1CQ / 2CQ; pack = replay of the packing trace, read = D2H + unpack): 64 KB single row 0.15-0.22 + 0.14-0.18 ms; 1 MB (YOLOX 14400 x 13 fp32 head level + a 240 x 240 uint32 map) 0.69-0.75 + 0.30-0.54 ms; 16 MB ([1, 1, 262144, 16]fp32) 1.24-1.74 + 3.7-8.1 ms (2-4 GB/s); bit-exact through trace replays (logs/ttaw/q9_pack_outputs_device*.log). Why: a single ROW_MAJOR row is ONE page, andttnn.reshape/concatstage whole pages in L1 (2 x the destination page per kernel copy for pages that are not 16-byte aligned), so the 0.14.0 layout failed above ~0.6 MB withTT_FATAL: RM reshape dest staging does not fit in L1.
CQ_COMPUTE = 0, CQ_INPUT = 1.
Alloc tracking (debug). TT_METAL_TRACE_ALLOC_TRACKING=1 must be exported before Python imports ttnn; then
ttnn.execute_trace refuses to replay while a buffer allocated after a capture is alive. alloc_tracking_enabled()
reports it; TraceRunner(alloc_tracking=True) raises if it is off. The runner acknowledges the outputs of every trace
captured after the first (plain ttnn would flag them: tests/device/test_trace_alloc_tracking_device.py).
4. Tensors (C03, ttaw.tensors)
| name | meaning |
|---|---|
TILE, round_up(n, multiple=32), tile_padded_shape(shape), pad_to_tile(a, value=0) |
tile geometry (host) |
pad_to_capacity(a, capacity, *, axis=0, value=0, truncate=False) -> (padded, n_valid) |
fixed-capacity buffers (capacities are grid-independent constants) |
as_4d(a) |
prepend unit dims to rank 4 |
float32_to_bf16_bits, bf16_bits_to_float32, round_to_bf16 |
RNE bf16 conversion identical to torch (expected values for exact tests) |
ttnn_dtype(name), dtype_name(dtype) |
"bf16", "fp32", "bfp8", "bfp4", "uint32", "int32", "uint16", "uint8" aliases |
to_host_tensor(value, dtype, layout=None, *, shape=None) |
numpy / torch / ttnn host tensor -> ttnn host tensor; integers never pass through a float intermediate (int64 input to ttnn.from_torch would) |
to_device(value, device, dtype="bf16", layout=None (TILE), memory_config=None (DRAM)) |
eager upload |
to_numpy(t) |
ttnn (device or host) / torch -> numpy (bf16 -> float32) |
to_layout(t, layout), typecast(t, dtype), to_fp32(t), to_bf16(t) |
no-ops when nothing changes (no extra program) |
fp32_island(fn, *tensors, out_dtype="bfloat16") |
run fn on fp32 copies, cast results back |
upload_u8(array, device, *, out_dtype="bfloat16", layout=None, memory_config=None) |
uint8 upload + on-device typecast (exact); in a trace keep the uint8 tensor as the input and ttnn.typecast as the first op |
HostStaging(shape, dtype) |
persistent ROW_MAJOR host tensor; .write(array) copies into its buffer through a torch.from_dlpack alias (zero_copy=True on this tree for float32 / bfloat16 / uint32 / int32 / uint16 / uint8) and returns .tensor; pass it as a run(inputs=...) value |
5. Weights (C04, ttaw.weights)
from .ttaw.weights import OnnxWeights, WeightCache, fold_bn_conv
w = OnnxWeights(weights_path / "pts_backbone_neck_head_centerpoint.onnx") # parsed as data, never executed
for node in w.nodes("Conv", "/backbone/*"): # graph order, glob or regex
wf, bf = w.fold_conv_bn(node.name) # + its BatchNormalization consumer, fp64, rounded once
k = w.param("/dit/blocks.0/attn/MatMul", 1) # anonymous initializer, addressed by consuming node + slot
cache = WeightCache("tt_centerpoint/base", w.sha256, version=f"{__version__}.prep1") # $TT_CACHE_PATH | ~/.cache/ttaw
tw = cache.get("backbone.0.conv.w", lambda: wf, dtype="bfloat16", layout=ttnn.ROW_MAJOR_LAYOUT, device=dev)
| name | meaning |
|---|---|
OnnxWeights(path, *, load_external_data=True) |
.array(name) (initializer / Constant, through Identity), .has, .initializer_names(), .state_dict(), .node(name), .nodes(op_type=None, pattern=None, *, regex=False), .find_node(pattern, op_type=None, *, regex=False), .producer(t), .consumers(t), .consumer_of(t, op_type=None), .param(node, slot), .params(node), .conv(node) -> ConvParams, .gemm(node) -> GemmParams, .matmul_weight(node), .batchnorm(node) -> BatchNorm, .fold_conv_bn(conv, bn=None, *, dtype=np.float32), .sha256, .input_names, .output_names, .opset. External data outside the model directory is refused. Unnamed nodes are "<OpType>_<index>" |
OnnxNode, ConvParams (weight, bias, strides, pads, dilations, group, kernel_shape, output_padding, auto_pad, output_shape), GemmParams (weight as stored, bias, trans_a, trans_b, alpha, beta) |
frozen records; pads is the explicit attribute: with auto_pad SAME_* derive the padding from the input size |
BatchNorm(gamma, beta, mean, var, eps=1e-5) |
float64; .scale, .shift, .channels, .apply(x, axis=1), .from_state_dict(sd, prefix, eps) |
fold_bn(w, b, bn, *, axis=0, dtype=np.float32) |
generic fold (dtype=None keeps float64 for a single final rounding to the device dtype) |
fold_bn_conv(w, b, bn) |
Conv weight [Cout, Cin/g, k...], any groups |
fold_bn_linear(w, b, bn, *, layout="out_in") |
torch Linear / Gemm transB=1 (out_in) or MatMul / Gemm transB=0 (in_out); also BN1d |
fold_bn_conv_transpose(w, b, bn, *, groups=1) |
ConvTranspose weight [Cin, Cout/g, k...] |
load_safetensors(path), load_torch_checkpoint(path, *, key=None) (torch.load(weights_only=True)), load_state_dict(path) -> StateDict |
.safetensors, .pth/.pt/.ckpt/.bin, .npz |
StateDict(tensors) |
.sub(prefix), .strip(prefix), .bn(name, eps) |
WeightCache(namespace, source_digest="", *, version="", root=None, enabled=None) |
.get(key, make, *, dtype, layout=None, device=None, memory_config=None, shape=None) caches ttnn host tensors as .tensorbin (atomic writes; rebuilt if unreadable or of another dtype / layout / shape), .path(...), .clear(), .hits, .misses; TTAW_WEIGHT_CACHE=0 disables. The cache outlives images (/tensor-cache is the host's ~/.cache/tt-model/<name>/tensors): pass version (bundle __version__ + a prep revision) and bump it whenever the code that prepares the tensors changes |
cache_root(), file_sha256(path) |
helpers |
6. Precision and knobs (C05, ttaw.precision, ttaw.knobs)
ttnn.matmul / linear drop to LoFi when a program_config or core_grid comes without a compute config, and
ttnn.WormholeComputeKernelConfig() without math_fidelity is MathFidelity.Invalid. Always pass one:
from .ttaw.precision import PrecisionPolicy, compute_kernel_config
POLICY = PrecisionPolicy({"head.reg*": "accurate", "backbone.*": "HiFi2+fp32:w=bfp8"}, default="balanced")
policy = POLICY.with_env("CENTERPOINT") # CENTERPOINT_PRECISION="backbone.*=HiFi4+fp32;*=HiFi2+fp32"
y = ttnn.linear(x, w, compute_kernel_config=policy.compute_kernel_config("head.reg.fc1"), program_config=pc)
w_dtype = policy.resolve("backbone.block3.conv2").weights_dtype()
| name | meaning |
|---|---|
compute_kernel_config(fidelity="HiFi2", *, fp32_acc=True, approx=False, packer_l1_acc=False, dst_full_sync=False) |
explicit ttnn.WormholeComputeKernelConfig (= BlackholeComputeKernelConfig) |
Precision(fidelity="HiFi2", fp32_acc=True, approx=False, packer_l1_acc=False, dst_full_sync=False, weights="bfloat16", activations="bfloat16") |
.parse("HiFi4+fp32+approx+l1acc+fullsync:w=bfp8:a=bf16" or preset), .label, .with_(**), .compute_kernel_config(), .weights_dtype(), .activations_dtype() |
PRESETS |
accurate (HiFi4 + fp32), balanced (HiFi2 + fp32, the default), fast (LoFi, bfp8 weights; only after gates pass) |
PrecisionPolicy(rules, default="balanced") |
first matching glob wins; .resolve(module), .compute_kernel_config(module) (cached), .override(spec), .with_env(prefix, env=None), .describe() (rules + which modules resolved to what) |
FIDELITIES |
("LoFi", "HiFi2", "HiFi3", "HiFi4") |
Other silent defaults to override by hand: ttnn.layer_norm epsilon 1e-12, SDPA is_causal=True, ttnn.embedding
PADDED returning the cached pad row, HARDSWISH fused into conv2d skipped.
Knobs (one per optimization, default = measured best, pinned in tt-model.yaml serve.env):
from .ttaw.knobs import Knob, Knobs
KNOBS = Knobs("CENTERPOINT", [Knob("FUSED_HEAD", True, "merged head convs"),
Knob("ACT_BLOCK_H", 64, "conv act_block_h", choices=(32, 64, 128)),
Knob("BFP8_WEIGHTS", False, "bfp8 backbone weights", experiment=True)])
knobs = KNOBS.read() # once, in _build; env CENTERPOINT_FUSED_HEAD=0 is the A/B switch
assert KNOBS.serve_env() == yaml_serve_env_subset # host test: the image pins the defaults
Knob(name, default, doc="", choices=None, experiment=False); Knobs(prefix, knobs): .read(env=None, **overrides) -> KnobValues (attribute / item access, .source(name), .overridden(), .as_dict(), immutable;
warns when an experiment knob is set), .defaults(), .serve_env(values=None), .doc_table(), .env_name(name);
parse_bool(text).
7. Metrics, goldens, gates (C06, ttaw.metrics, ttaw.golden)
# tests/test_pcc_device.py of a bundle
from ..ttaw.golden import GateRegistry, compare_taps, load_goldens
from ..ttaw.metrics import pcc, match_detections
GATES = GateRegistry.for_test(__file__, {"backbone": 0.999, "head.heatmap": 0.99,
"dets.recall": (0.95, "min", "recall"), "plan.ade": (0.5, "max", "ade")})
def test_taps(device_taps): # numpy dict from an eager device run or trace outputs
with load_goldens(SPEC_DIR / "golden/sample0.npz") as gold:
report = compare_taps(device_taps, gold, gates=GATES, names=["backbone", "head.heatmap"])
report.save_json(LOGS / "compare_backbone.json"); print(report.table())
assert report.passed
GateRegistry stores frozen gates in <test stem>.gates.json next to the test at the first green run. A later
declaration that is looser (lower min, higher max, other direction or metric) raises GateLoosenedError;
tighter ones are re-frozen; failing checks freeze nothing. TTAW_GATES_READONLY=1 (or write=False) never writes.
Changing a frozen gate = editing the JSON by hand + disclosure in VERIFICATION_.md.
| name | meaning |
|---|---|
pcc(t, r) |
float64 Pearson; NaN on non-finite input; a constant side (detected exactly, max == min) gives 1.0 for two equal constants (np.allclose) and 0.0 otherwise, also for two different constants |
masked_pcc(t, r, mask), valid_row_pcc(t, r, n_valid=None, *, rows=None, axis=0) |
padded / masked buffers |
error_stats(t, r) |
{pcc, max_abs, mean_abs, rel_l2, n} |
argmax_agreement(t, r, *, axis=-1, mask=None), label_agreement(t, r, *, mask=None) |
class decisions (ties -> lowest index) |
mask_iou(a, b), mean_iou(t, r, num_classes, *, ignore_index=None) -> (miou, per_class), box_iou_xyxy(a, b) |
IoU |
topk_overlap(t_idx, r_idx, k=None), topk_set_overlap(t_scores, r_scores, k) |
data-dependent selections |
match_detections(t_centers, t_labels, t_scores, r_centers, r_labels, r_scores, *, max_dist=0.5, dims=2, same_label=True) -> DetectionMatch |
greedy same-label BEV-centre matching (.pairs, .recall, .precision, .matched, .center_errors, .score_errors, .to_dict()); box arrays [N, 7] work as centres; a box with a non-finite centre never matches |
ade_fde(pred, ref, *, dims=2) -> (ade, fde) |
trajectories [..., T, D] |
as_array(x) |
numpy / torch (bf16 ok) / list -> numpy |
save_goldens(path, tensors, meta=None, *, compress=False), load_goldens(path) -> Goldens |
.npz + JSON __meta__; Goldens is a lazy mapping with .meta, .sha256, .close(), context manager |
TapRegistry(enabled=True, *, include=("*",), exclude=()) |
.tap(name, value) -> value (device tensors read at once; raises inside a capture), .scope(prefix), .wants, .names(), .to_dict(), .save(path, meta), .clear(); NULL_TAPS is a disabled registry |
Gate(threshold, direction="min", metric="pcc") |
.of(spec, metric=None) (a bare number takes the direction of metric from DIRECTIONS and is refused for a metric of unknown direction; a direction contradicting a known metric, e.g. a "max" PCC gate, is refused), .passes(v) (NaN fails), .looser_than(other) |
GateRegistry(path, gates=None, *, write=None) |
.for_test(test_file, gates), .check(name, value, gate=None) -> GateResult, .require(...) (AssertionError), .gate(name), .names(), .frozen, .results, .save() |
CompareReport(title="", meta=None, gates=None) |
.add(name, test, ref, *, metric="pcc", gate=None), .add_value(name, value, *, metric, gate=None, stats=None), .passed, .table(), .to_dict(), .save_json(path); TapComparison records |
compare_taps(test, golden, *, gates=None, metric="pcc", names=None, title="", meta=None) |
one report over common (or named) taps; missing taps fail when gated |
METRICS |
name -> (function, direction) used by reports (pcc, masked_pcc, argmax_agreement, label_agreement, mask_iou, max_abs, mean_abs, rel_l2) |
DIRECTIONS |
metric name -> "min" / "max" for every gate metric that may be given as a bare number: the METRICS plus recall, precision, iou, miou, agreement, topk_overlap (min) and ade, fde, center_err*, score_err*, max_abs_err (max) |
8. Bench and profile (C07, ttaw.profiling)
from .ttaw.profiling import AiclkSampler, bench_trace_runner, signposted, summarize_ops
with AiclkSampler(interval_s=0.05) as clk: # sysfs tt_aiclk + hwmon power / temperature
bench = bench_trace_runner(model.runner, "default", {"pillars": p}, iters=100, post=model.decode)
bench.save_json(f"{ROOT}/logs/centerpoint/bench_baseline.json", config=model.device_info, aiclk=clk.summary())
print(bench.table()) # host_in / h2d / trace / d2h / post / e2e / b2b, p50 p99
with signposted("trace"): # profile_ops.py under `python -m tracy -r -p -v -o ...`
model.runner.replay("default"); ttnn.synchronize_device(dev)
print(summarize_ops(f"{ROOT}/generated/profiler/centerpoint_baseline").table()) # ops, kernel sum, op2op, span
| name | meaning |
|---|---|
StageBench(name="") |
.stage(name) context, .add(name, ms), .summary() ({stage: {n, p50, p99, mean, min, max}}), .table(), .to_dict(**extra), .save_json(path, **extra); STAGES order |
bench_trace_runner(runner, variant, inputs, *, params=None, iters=100, warmup=10, post=None, b2b_iters=None, name="") |
the standard stage breakdown of a TraceRunner variant |
time_b2b(enqueue, sync, n=100, warmup=5) -> ms |
back-to-back device time per iteration |
signpost(name, message=None) -> bool, signposted(name) |
Tracy markers (no-op without tracy) |
read_device_profiler(device) |
ttnn.ReadDeviceProfiler per trace segment (the buffer holds ~1000 ops) |
summarize_ops(source, *, start="trace", end=None, last_replay_session=None, freq_mhz=None, top_gaps=10) -> OpsSummary |
ops_perf_results*.csv / cpp_device_perf_report.csv / directory / rows; OpsSummary: ops, kernel_sum_us, fw_sum_us, op2op_sum_us, span_us, by_op, fidelity, top_gaps, .table(top), .to_dict(). CLI: `python -m .ttaw.profiling <csv |
find_ops_csv(path) |
newest ops CSV under a directory |
AiclkSampler(chip=0, interval_s=0.05, *, root="/sys/class/tenstorrent") |
.start(), .stop(), context manager, .sample_once(), .summary(), .available |
common/tools/p2_trace_bench.py --mode eth-1cq|eth-2cq|eth-2cq-staged|worker-2cq measures the runner itself.
9. I/O and outputs (C08, ttaw.io, ttaw.outputs)
ttaw.io (numpy only; every client mistake raises InputError, which the server maps to HTTP 400):
b64decode(s, *, field, max_bytes), PointCloud(points, fields, frame_id) (.select(names, fill=None); rows with a
non-finite x / y / z column -- the first three columns when unnamed -- dropped),
decode_points(spec, *, max_points=2_000_000, max_bytes=None, default_fields=DEFAULT_POINT_FIELDS),
load_points(source, *, fields=None, fmt=None, frame_id="base_link", default_fields=...) (path .bin/.npy/.npz/.pcd,
bytes + fmt, arrays, structured arrays, envelopes), decode_image(raw, *, fmt="auto", max_side=8192),
load_image(image) -> RGB uint8 HxWx3, CameraImage, decode_cameras(images, calibration=None, *, order=None, require_calibration=True, max_bytes=None) (the server's images[]), its Python-API twins (0.15.0)
load_camera(source, *, name=None, calibration=None, require_calibration=False) (a CameraImage, {"camera", "image" | "path" | "data", "intrinsics", "T_ref_from_camera", "distortion", "timestamp_s"} or any load_image source
with name; inline calibration wins over calibration["cameras"][name]) and load_cameras(images, calibration=None, *, order=None, require_calibration=True, calib_dir=None) (a list or a {name: image} mapping; {"preset": name}
resolved in calib_dir; reordered to order, every camera exactly once: model(images=..., calibration=...) and
/predict see the same cameras), decode_rois(rois, *, cameras=None, labels=None, max_rois=4096) (2-D detections
as an input, [{"camera", "label", "score", "box_xyxy"}]: names / indices checked against labels, score in [0, 1],
finite ordered boxes; PointPainting rois, BUNDLE_CONVENTIONS.md 7.2), parse_transform(obj, *, field) (4x4 / 3x4 / {matrix} /
{translation, rotation_wxyz|rotation_xyzw|rotation} / {x, y, z, roll, pitch, yaw} tf2 RPY),
parse_intrinsics(obj, *, field), resolve_calibration(calibration, calib_dir) ({"preset": name} ->
calib_dir/<name>.json merged), decode_named_arrays(spec, schema=None, *, max_bytes=None) (planner inputs),
check_named_arrays(arrays, schema) (names, shapes, numeric and finite values, integral values in range for integer
dtypes; every problem is an InputError), load_named_arrays(source, schema=None) (mapping / envelope / .npz
path or bytes: the Python-API form),
encode_array(arr, *, fmt="npz"|"npz_compressed"|"raw"|"list", key), encode_png(image, *, key), to_jsonable(obj)
(float32 / float16 scalars rounded to 6 significant digits, float64 exact),
POINT_FORMATS, DEFAULT_POINT_FIELDS = ("x", "y", "z", "intensity"), MAX_POINTS_DEFAULT.
ttaw.outputs (each has to_dicts() and to_dict(output_format="json"|"npz") = the /predict body with model,
frame_id, meta, timing_ms):
| class | fields |
|---|---|
Detections3D(boxes [N,7] x y z l w h yaw, scores, label_ids, velocities=None, labels=(), model="", frame_id="base_link", timing_ms, meta) |
rows sorted by score at construction; detections[] {label, label_id, score, center, size, yaw, velocity}; npz adds arrays |
Detections2D(boxes_xyxy, scores, label_ids, labels=(), extras={}, model, frame_id="camera", ...) |
detections[] {label, label_id, score, box_xyxy}; extras (e.g. {"semseg": Mask2D}) encoded by name |
Segmentation3D(label_ids [N], class_names=(), scores=None, ...) |
labels (npz uint8 / uint16), class_names, class_counts, scores |
Mask2D(mask (H, W), class_names=(), encoding="png", ...) |
mask (png, or npz for output_format="npz" / non-uint8), class_counts; .payload(fmt) |
Trajectory(poses [T, D], columns=("x", "y", "yaw"), turn_indicator=None, predicted_agents=None, ...) |
trajectory, columns, num_poses, turn_indicator, predicted_agents (npz) |
label_name(labels, id) maps ids to names (the id as text when out of range).
10. ModelBase (C08, ttaw.api_base)
from .ttaw.api_base import ModelBase
from .ttaw.outputs import Detections3D
class CenterPoint(ModelBase):
MODEL_NAME = "centerpoint-p150"; ENV_PREFIX = "CENTERPOINT"
DEFAULT_REPO = "AutowareFoundation/lidar_centerpoint"; DEFAULT_TAG = "v4.1"
DEFAULT_REVISION = "494c8171def40bd36cc2feb323e0a5acbfab132b"; ALLOW_PATTERNS = ["base/*", "tiny/*"]
VARIANTS = ("base", "tiny"); DEFAULT_VARIANT = "base"; INPUT_KIND = "lidar"
LABELS = ("CAR", "TRUCK", "BUS", "BICYCLE", "PEDESTRIAN")
RUNTIME_PARAMS = {"score_threshold": (float, 0.0, 1.0, 0.35), "max_detections": (int, 1, 1000, 500)}
DEVICE_DEFAULTS = {"num_command_queues": 1, "trace_region_size": 64 << 20}
def _build(self): # weights -> device tensors, TraceRunner + inputs / params / states / variants
self.runner = TraceRunner(self.device, name=self.MODEL_NAME); ...
def _warm_one(self, v): # capture (default_warmup_variants() -> [{"variant": self.variant}])
self.runner.capture()
def _prepare(self, points=None, **kw): # host pre-processing; raise io.InputError for client mistakes
...
def _forward(self, prep): # under the model lock
return self.runner("default", inputs={...})
def _postprocess(self, raw, prep, params) -> Detections3D:
return Detections3D(boxes, scores, ids, labels=self.LABELS, model=self.MODEL_NAME)
def _release(self):
self.runner.release()
def extra_info(self):
return {"trace": self.runner.describe(), "knobs": self.knobs.as_dict()}
with CenterPoint.from_pretrained() as model: # weights (pinned) -> device (ETH, 12x10) -> build -> warm
out = model("samples/test.npz", score_threshold=0.4)
print(out.to_dict()["num_detections"], out.timing_ms, model.info)
from_pretrained(model_id=None, *, revision=None, variant=None, device_id=None, device=None, dispatch=None, num_command_queues=None, weights_dir=None, warmup_variants="default", verbose=False, **compile_params): weights are
resolved first (weights_dir > <ENV>_WEIGHTS_DIR > local dir model_id > HF snapshot at the pinned revision,
offline fallback to the cache), then the device opens with DEVICE_DEFAULTS < <ENV>_* / TT_DEVICE_ID < explicit
arguments; a caller-provided device is left open by close(). Variant default: <ENV>_VARIANT or
DEFAULT_VARIANT. A failing build closes everything.
Other members: resolve_weights(...) (also module-level resolve_weights(model_id, *, revision, allow_patterns, weights_dir, env_var)), device_config(**overrides), requires_calibration(), validate_params(params) (unknown /
outside [min, max] (either bound may be None) / wrong type -> InputError; bools must be real booleans, ints must
be integral and not booleans, floats finite; numeric strings are accepted), warmup(variants="default") (idempotent),
default_warmup_variants(), __call__(points=None, *, sweeps, images, calibration, stream, inputs, **params) /
predict (one re-entrant lock around prepare + forward + postprocess; fills timing_ms preprocess / device /
postprocess / total), info, closed, close() (idempotent, also at interpreter exit), context manager.
Class attributes also: WEIGHTS_LICENSE, CAMERA_ORDER, REQUIRE_CALIBRATION (None: true for multicam),
POINT_FIELDS, EXTRA_INPUTS (model-specific input keywords handed to _prepare instead of being validated as
runtime params, e.g. PointPainting ("rois",); the server's decode_extra adds them to the call kwargs) and
INPUT_SCHEMA ({name: (shape with None for free dims, dtype)}: inputs= is decoded with
io.load_named_arrays and checked on every call, API and server alike). RUNTIME_PARAMS may not reuse an input
keyword (from_pretrained raises TypeError). info adds extra_inputs and input_schema; __call__ re-checks
closed once it holds the lock.
11. Server (C08, ttaw.server)
create_app(spec: ServerSpec, *, model_factory=None) -> FastAPI implements BUNDLE_CONVENTIONS.md section 7:
GET / (routes), GET /health and /v1/health (always 200: ok / starting / error, plus device),
GET /info (model, task, io, autoware, weights, device, input limits, calibration presets, labels, variants,
runtime / compile params, warm-up and boot ms, source), GET /v1/models (stub), POST /predict. Errors: 400
(InputError, bad params, body above <ENV>_MAX_BODY_MB by Content-Length), 422 (schema: unknown fields are
refused), 503 (starting / failed boot), 500 (inference failed: <Type>: <msg>). The lifespan reads the environment
once (config_from_env), refuses a TT_MESH_SHAPE other than 1x1, builds the model through
spec.model_cls.from_pretrained (or model_factory(cfg)), and closes it on shutdown under the lock. A body holding a
NaN / infinity (strict JSON has none) is a 500 inference failed: non-finite value at body.<path>. /info input
also lists extra_inputs and input_schema.
ServerSpec(model_name, env_prefix, model_cls=None, task="", default_weights="", owner="changh95", hardware=HARDWARE, io="", autoware={}, source={}, calib_dir=None, default_variant=None, version="0.1.0", description="", request_model=PredictRequest, decode_extra=None, info_extra=None). Model-specific request fields: subclass
PredictRequest (e.g. rois: Optional[List[...]] = None), add them to the call kwargs in
decode_extra(request, kwargs, max_bytes) and list them in the model's EXTRA_INPUTS. app.state.ttaw
(ServerState) holds model, config, ready, error, lock, model_factory (tests: assign a stub before
TestClient(app)) and predict (the route handler, for API == server checks). Also exported:
parse_mesh_shape, config_from_env(spec, env=None), default_model_factory(spec), decode_request(req, spec, max_bytes), the request models PointsSpec, SweepSpec, ImageSpec, StreamSpec, ArraysSpec,
PredictRequest.
ttaw.server.client and ttaw.server.smoke are standard-library only and import nothing from ttaw, so a bundle's
smoke_test.py (run by container_smoke.sh with the host's python3, which has no numpy) loads them by path:
HERE = Path(__file__).resolve().parent # code/<pkg>/server
sys.path.insert(0, str(HERE.parent / "ttaw" / "server"))
import client as ttaw_client
import smoke as ttaw_smoke
health = ttaw_client.wait_ready(url, wait_s=600)
info, fails = ttaw_client.check_service(url, expect_dispatch="eth", expect_grid="12x10") # PLAN 0.3 item 6
pinned = ttaw_smoke.pinned_config(f"{staged}/tt_kernel_manifest.json", profile) # serve.env + profile
fails += ttaw_smoke.check_pinned(info, pinned, "CENTERPOINT") # /info runs the pins
code, body = ttaw_client.post(url, ttaw_client.build_request(points=str(SAMPLE)))
if ref := ttaw_smoke.find_reference(SAMPLE, pinned["profile"], info.get("variant")): # stored CPU output
metrics, more = ttaw_smoke.compare_with_reference(body, ttaw_smoke.load_json(ref), gates={"min_recall": 0.95})
fails += more
fails += [f] if (f := ttaw_client.check_bad_request(url)) else []
print(ttaw_client.smoke_line("centerpoint-p150", f"n={body.get('num_detections')}", fails))
Functions: build_request(points=None, fields=None, images=(), calib=None, calib_preset=None, inputs=None, params=None, stream_id=None, reset=False, output_format="json"), post(url, payload, timeout=120),
get_json(url, timeout=10), wait_ready(base, wait_s=0, poll_s=5), check_service(base, *, expect_dispatch="eth", expect_grid="12x10") -> (info, failures), check_bad_request(base, payload=None), smoke_line(model, summary, failures), b64file, main (CLI: --points/--image/--calib/--calib-preset/--inputs/--param/--out/--url).
ttaw.server.smoke functions:
| name | meaning |
|---|---|
compare_with_reference(body, reference, gates=None) -> (metrics, failures) |
served /predict body vs the stored CPU-reference body of the same input, by the reference's keys: detections (greedy same-label matching in descending reference score; 3-D rows by BEV centre distance, 2-D rows by IoU; recall, precision, max score difference), trajectory (ADE / FDE over x, y), every encoded array (integer: fraction of equal elements; float: max abs difference, equal infinities agree, a NaN fails), model / frame_id equal. Undecodable input is a failure, never an exception |
DEFAULT_GATES |
max_center_dist 0.5 m, min_iou 0.5, min_recall 0.95, min_precision 0.95, max_score_err 0.05, min_label_agreement 0.99, max_abs_err None (report only), max_ade 0.5, max_fde 1.0; gates= overrides by name (unknown names raise) |
find_reference(sample, *keys) |
first existing <stem>.<key>.reference.json (keys: serve profile, model variant), else <stem>.reference.json, else None |
pinned_config(manifest, profile=None) |
{"profile", "env", "weights"} of a staged tt_kernel_manifest.json: serve.env with the profile's env on top (tt-model's merge), default profile when None |
check_pinned(info, pinned, prefix) |
/info agrees with the pinned <prefix>_DISPATCH, _NUM_CQS, _VARIANT and the weights revision (absent pins are not checked) |
decode_array(spec) -> Array(shape, dtype, values) |
numpy-free decoding of io.encode_array (npz, npz_compressed, raw, list; C or Fortran order, either byte order) and io.encode_png (8-bit grey / RGB(A), every PNG filter) |
describe_metrics(metrics), load_json(path), ARRAY_FORMATS |
helpers |
The reference body is the to_dict() of the fp32 CPU reference on the shipped sample, stored next to it as
<stem>.reference.json (per profile or variant when the output depends on it). Synthetic samples may give no
detections, so agreement with that reference, not a detection count, is their smoke gate (PLAN.md section 6.3).
12. Vendoring (C09, common/tools/vendor.py)
python common/tools/vendor.py <bundle> [--pkg tt_<model>] [--rev REV] [--allow-dirty] [--check] [--dry-run].
- What is vendored is a committed revision:
ttaw/as committed at--rev(defaultHEAD), read withgit archive, never the working tree ofcommon/(other agents' uncommitted modules live there: YOLOX PORT_LOG Q7). Uncommitted changes undercommon/ttaw(staged, unstaged, untracked) are listed in a WARNING and left out: commit your change (git add <your paths>, bump the version + CHANGELOG), then vendor. The revision is resolved once, so a commit landing meanwhile never mixes into the recordedsource_commit. --allow-dirtyvendors the working tree instead, for a local experiment only:source_dirtyis true when it had uncommitted changes or its files differ fromttaw/of HEAD (.gitignore'd files included:source_dirty: falsealways means "verified againstsource_commit");check_bundle.pywarns about such a copy and refuses it at--stage publish. Outside a git checkout the working tree is vendored withsource_commit: null(same treatment).- The destination must be a plain directory: one that is, or reaches through a symlink,
common/ttaw(or an ancestor of it) is refused before anything is written (mirroring HEAD there would delete other agents' untracked files and revert their uncommitted changes). - The copy mirrors that tree into
code/<pkg>/ttaw(stale files removed;__pycache__,*.pyc,*.bin,*.lognever copied) and writesVENDORED.json:version(of the vendored__init__.py),source_commit(full sha),source_ref(the--revgiven),source_tree(commit|working-tree),source_dirty,vendored_at, per-file sha256.research/packaging/scripts/instantiate_bundle.py --outvendors the same way (--vendor-rev,--vendor-allow-dirty; an existing copy is kept unless--force, and always with--only <paths>). --check(exit 1 on any line): "modified in bundle" / "missing in bundle" / "extra in bundle" (the copy vs itsVENDORED.json); "not at recorded revision " (VENDORED.jsonis notttaw/ofsource_commit; "unverified:" when that commit is not incommon/; skipped for asource_dirtycopy); "outdated:" (VENDORED.jsonvsttaw/committed at--rev, default HEAD: re-vendor and re-run the gates). Uncommitted work incommon/ttawnever makes a bundle look outdated.check_bundle.pyreports "outdated:" and "unverified:" as warnings (errors at--stage publish), every other line as an error.
Never edit the vendored copy: change common/ttaw, bump __version__ + CHANGELOG, commit, re-vendor every consumer
and re-run their gates (PLAN.md section 5.1 step 5). Functions: vendor(bundle, pkg=None, *, dry_run=False, rev=None, allow_dirty=False), check(bundle, pkg=None, *, rev=None), load_source(rev=None, *, allow_dirty=False) -> Source,
rev_files(rev), rev_hashes(rev), rev_version(rev), resolve_rev(rev), dirty_paths(), tree_hashes(root),
source_version() (working tree), git_state(), find_package(bundle, pkg); git calls never take the index lock
(GIT_OPTIONAL_LOCKS=0).
13. Measured facts and pitfalls (this p150b, tt-metal 44d6650 + ETH patch)
Device results of the ttaw suites and probe P2 are recorded in common/probes/p2-trace-runner.md. In short:
- ETH dispatch opens 12x10 with 1 and 2 CQs, WORKER 11x10,
auto-> ETH (tests/device/test_open_device.py). - In-trace
ttnn.copyinto persistent float32 / uint32 / bfloat16 tensors, ping-pong traces, RT-dev scalar params, packed readback, 2CQ uploads (direct and staged), CQ1 readback and replay == eager are bit-exact. - A ROW_MAJOR tensor is one page per row, and the RM
reshape/pad/concatprograms stage whole pages in L1:reshape_rm_program_factory.cppneeds 2 x (destination page + 80 B) per kernel copy (two copies when the pages divide each other) when the pages are not 16-byte aligned, out of ~1.43 MB free L1 per core (TT_FATAL ... RM reshape dest staging does not fit in L1: need at least 1497632 B dest + 512 B source, have 1461248 B, YOLOX's[1, 1, 1, 187200]fp32 row). Keep RM rows (last dim x element size) well below 0.5 MB;pack_outputsdoes (section 3). - A program-cache miss inside a capture raises with the op name and leaves the device usable.
- numpy uint32 arrays (including values >= 2**31) upload exactly; int64 arrays must not be passed to
ttnn.from_torchdirectly (they would go through bf16): usetensors.to_host_tensor. numpy.asarray(host_buffer)fails andnumpy.from_dlpackis read-only;torch.from_dlpackis a writable alias (HostStaging).ttnn.matmulwith an explicit HiFi4 + fp32 config: max |error| 0.03 vs fp64 on a 64x512x128 product; LoFi 2.4.- Runner cost (34-program toy graph, 4 MiB input):
replayb2b equals a rawexecute_trace(0.857 ms); streamingrun()per frame 1.07 ms (1CQ), 1.09 ms (direct 2CQ: CQ1 must wait for the previous replay before overwriting the input), 0.92 ms withstage_inputs=True(upload hidden behind the replay). Use staging when uploads are large and frames are pipelined; it buys nothing for a single synchronous request. ttnn.to_torchof a host tensor copies, soread()results stay valid after later reads.- Review of 0.2.0 (ttaw 0.3.0,
logs/ttaw/review_*):ttnn.reshapeof a ROW_MAJOR tensor returns a view on the same buffer, andttnn.deallocate(view)(defaultforce=True) frees the base tensor's memory (review_force_dealloc_probe.log); a program first enqueued after a capture is a "program_cache" buffer the allocation tracker refuses before the next replay (review_old_staged_tracking_probe.log). Ping-pong states written throughctx.write_target(output_tensor=), D16 bank switches between two streams,reset_statefrom a bank,write_inputfollowed by a CQ1 upload, and a 2CQ run after a partially failed capture are bit-exact on the p150b in 1CQ and 2CQ; withTT_METAL_TRACE_ALLOC_TRACKING=1bank switches and the 2CQ staged path allocate nothing after capture. - Where the 13 specs' other needs live: fp32 row gathers are
ttnn.gather(FLOAT32, TILE; bit-exact but ~1 us per picked index), bf16 rowsttnn.embedding(3-5 us; probe P13,probes/g1-dispatch-genericop-state.md); top-k, argmax, conv / pool, attention / grid_sample facts are inprobes/g2..g4-*.md; the op wrappers C17-C23 and kernels K1-K11 are added by the first port that needs them (PLAN.md 1.2-1.3). RT-dev values of any shape areadd_param(re-uploaded only when they change) or, when they change every frame (index tables),add_input; shape buckets are variants sharingadd_outputbuffers; trace segments around a host fallback are variants handed off through states or a read +run(inputs=...); serve profiles pin<ENV>_VARIANT/_NUM_CQS/_DISPATCH, checked byserver.smoke.check_pinned; temporal state per stream isStreamBanks(section 3). - Device tests: run only your own files.
common/tests/device/also holds the probe agent'stest_probe_p*.py, some of which spawn device subprocesses and must not run inside another pytest process.
14. Image pre-processing (C14, ttaw.image)
Bit-exact host emulations of the resize kernels the Autoware camera nodes deploy, driven by per-source-size lookup
tables computed once and cached (a camera keeps its size). numpy only. A standard resize in their place moves the
network input enough to fail the PCC gates (YOLOX: cv2.resize drops the det-output PCC to 0.94,
research/yolox/SPEC.md section 3), so each preset is tested against a scalar transliteration of the deployed code.
from .ttaw.image import yolox_letterbox, yolox_letterbox_geometry
bgr = rgb[:, :, ::-1] # Autoware feeds BGR8 (cv_bridge toCvCopy BGR8)
x_u8, geom = yolox_letterbox(bgr) # (960, 960, 3) uint8 HWC, BGR, pad 114
x_onnx = yolox_letterbox(bgr, layout="nchw")[0].astype("float32") # == the ONNX input `images` (1, 3, 960, 960)
geom.scale, geom.resized_hw, geom.mask_hw # float32 scale (decoder), (r_h, r_w), mask crop
from .ttaw.image_linear import sceneseg_preprocess, sceneseg_resize, opencv_resize_nearest
u8 = sceneseg_resize(bgr) # (320, 640, 3) uint8: cv::resize INTER_LINEAR, exact
x = sceneseg_preprocess(bgr, channel_order="bgr") # == the SceneSeg ONNX input (1, 3, 320, 640) float32
mask_src = opencv_resize_nearest(mask_320x640, bgr.shape[:2]) # the node's INTER_NEAREST back to the source size
| name | meaning |
|---|---|
yolox_letterbox(image, dst_hw=(960, 960), *, pad_value=114, layout="hwc", out=None) |
autoware_tensorrt_yolox preprocess.cu:43-129 @ 9ceaccf: inverted half-magnitude "bilinear" weights, fmaf single roundings, float -> int truncation between the passes, lroundf, top-left letterbox (114), channel order kept -> (uint8 HWC or (1, C, H, W), LetterboxGeometry). Needs a source of at least 2x2 |
yolox_letterbox_geometry(src_h, src_w, dst_h=960, dst_w=960) -> LetterboxGeometry |
scale (exact float32 value as a float; scale_f32), resized_hw = (r_h, r_w), mask_hw (Autoware's mask crop, the same pair), src_hw, dst_hw, pad_value, to_dict() |
yolox_letterbox_lut(src_h, src_w, ...) -> YoloxLetterboxLUT |
the cached tables (rows, row_w, cols, col_w, k); .apply(image, *, out=None, chunk_rows=96) (any channel count) |
LUTCache(maxsize=8) |
the thread-safe LRU the presets use (.get(key, make), .hits, .misses, .clear()) |
lroundf(x), f32(x) |
C lroundf (half away from zero) and one float32 rounding, for presets and their tests |
bevdet_nearest_crop(image, *, channels="rgb", src_hw=(900, 1600), dst_hw=(256, 704), crop_hw=(140, 0), out=None) |
the bevdet::Preprocess plugin of autoware_tensorrt_bevdet (bevdet_vendor 0.2.1, research/bevdet/SPEC.md 3.2): nearest resize by r = (float)dst_w / src_w = 0.44f and crop (140, 0) of the 1600x900 image, roundf(i / r + crop_h / r) rows (318, 320, 323, ..., 898) and columns (0, 2, 5, ..., 1598) in float32 -> (3, 256, 704) uint8 B, G, R planes. channels is the input order ("rgb" from ttaw.io.load_image, "bgr" from OpenCV). The node's cv::resize of other image sizes to 1600x900 is the caller's (OpenCV) |
bevdet_nearest_lut(src_hw, dst_hw, crop_hw) -> NearestCropLUT |
the cached gather (rows, cols, resize); .apply(image, *, channels, out), .apply_planes(planes [..., C, H, W]) |
bevdet_normalize(crop, mean=BEVDET_MEAN, std=BEVDET_STD) |
(x - mean[c]) / std[c] in float32 on B, G, R planes (BEVDET_MEAN / BEVDET_STD are the "RGB" ImageNet numbers, applied to BGR planes as trained) |
bevformer_preprocess(images, *, mean=BEVFORMER_MEAN_BGR, std=BEVFORMER_STD, scale=0.8, pad_divisor=32, out=None, workers=1) |
autoware_tensorrt_bevformer's pipeline (research/bevformer/SPEC.md 3.1): N BGR uint8 images of one size (the node's 1600x900) -> normalise first ((x - mean_c) / std_c as OpenCV's convertTo: BGR means 103.53 / 116.28 / 123.675, std 1) -> cv::resize x0.8 INTER_LINEAR on float -> zero pad bottom / right to /32 -> [N, 3, 736, 1280] float32 B, G, R planes (the ONNX input image without its leading 1). workers threads over the cameras. The node's uint8 cv::resize of other sizes to 1600x900 is the caller's (OpenCV) |
bevformer_input_geometry(src_hw=(900, 1600), scale=0.8, pad_divisor=32) |
{"src_hw", "resized_hw", "padded_hw"}: (int)(src * 0.8f) in float32 and the /32 pad: (720, 1280) and (736, 1280) deployed (the network normalises its UV by the padded size) |
bevformer_normalize(image, mean, std) |
the normalisation alone, HWC float32 |
opencv_linear_lut(src_hw, dst_hw, *, strict=True) -> LinearResizeLUT; opencv_resize_linear_f32(image, dst_hw, *, strict=True) |
OpenCV's float INTER_LINEAR (resize.cpp coefficients f = (float)((d + 0.5) * scale - 0.5), border clamps; each pass fma(x1 - x0, w, x0) in float32, horizontal first), cached tables (rows0, rows1, wy, cols0, cols1, wx); .apply(image HWC), .apply_planes(planes [..., H, W]) (strided block slices for periodic taps). strict=True refuses geometries not verified bit-exact (weights not multiples of 1/8: OpenCV computes other scales' coefficients differently) and exact 2x down-scales (OpenCV's INTER_AREA path) |
PRESETS |
name -> (function, description) of the implemented presets |
image_area.meteor_resize(image, *, out=None) |
METEOR (tier4/METEOR hf/onnx_smoke_test.py:42-44, research/meteor/SPEC.md 3): one uint8 frame (any channel order) -> 432x768 with OpenCV INTER_AREA (a copy when already 432x768). Down-scaling only (raises ValueError for any axis smaller than the target) |
image_area.opencv_resize_area_u8(image, dst_hw, *, out=None), opencv_area_lut(src_hw, dst_hw) -> AreaResizeLUT |
OpenCV cv::resize(INTER_AREA) of uint8 HxW / HxWxC images, any down-scale: resizeArea_<uchar, float> (computeResizeAreaTab float weights from double, horizontal then vertical float32 accumulation in table order, cvRound) and resizeAreaFast_ for integer factors (float32 s * (1.f / area), except the 2x2 fast_mode of 1-, 3- and 4-channel images: (s + 2) >> 2); cached tables (fast, rows, row_w, cols, col_w) |
image_area.scale_intrinsics(K, src_hw, dst_hw=(432, 768)) |
K of the resized image: row 0 x dst_w / src_w, row 1 x dst_h / src_h (METEOR extract_gt.py:128-130; anisotropic resizes keep fx != fy), float64 |
image_linear.sceneseg_preprocess(image_bgr, *, channel_order="bgr") |
VisionPilot SceneSeg (middleware_recipes/common/backends/onnx_runtime_backend.cpp:41-60 @ vision_pilot 04fa3e80; research/sceneseg/SPEC.md 3): one HxWx3 B, G, R uint8 frame -> the ONNX input float32 [1, 3, 320, 640], bit-exact = sceneseg_normalize(sceneseg_resize(image)). channel_order: "bgr" (deployed: B, G, R planes, BGR-ordered ImageNet constants) or "rgb" (training order: R, G, B planes, RGB constants; a channel permutation of the bgr input) |
image_linear.sceneseg_resize(image_bgr, *, dst_hw=(320, 640), out=None), sceneseg_normalize(resized_bgr, *, channel_order="bgr") |
the node's squash cv::resize(img, Size(640, 320)) (INTER_LINEAR, aspect not kept) -> uint8 (320, 640, 3), channel order kept (the TT device input); the normalisation as OpenCV computes it: convertTo(CV_32F, 1/255) (float32 product), cv::subtract(Scalar) in float32, cv::divide(Scalar) in double rounded to float32 (a float32 division differs by 1 ulp in ~27 % of the values), cv::split -> NCHW. SCENESEG_INPUT_HW, SCENESEG_MEAN_RGB, SCENESEG_STD_RGB, SCENESEG_CHANNEL_ORDERS |
image_linear.opencv_resize_linear_u8(image, dst_hw, *, out=None), opencv_linear_u8_lut(src_hw, dst_hw) -> LinearResizeU8LUT |
OpenCV cv::resize(INTER_LINEAR) of uint8 HxW / HxWxC images, any scale: resize()'s float coefficients (fx = (float)((dx + 0.5) * scale - 0.5), x-border fix only), cvRound(w * 2048) weights, the exact int32 horizontal pass and the uchar fixed-point vertical pass (((b0 * (h0 >> 4)) >> 16) + ((b1 * (h1 >> 4)) >> 16) + 2) >> 2; a copy for equal sizes and OpenCV's INTER_AREA fast path (image_area) for an exact 2x down-scale. Cached tables (mode, cols0, cols1, a0, a1, rows, r0, r1, b0, b1); .apply(image, *, out=None) |
image_linear.opencv_resize_nearest(image, dst_hw, *, out=None), opencv_nearest_lut(src_hw, dst_hw) -> NearestResizeLUT |
OpenCV cv::resize(INTER_NEAREST) (resizeNN: min(floor(x * (1.0 / inv_scale)), W - 1) in double) of HxW / HxWxC images of any dtype (a gather): e.g. SceneSeg's 320x640 mask back to the source size (run_model_node.cpp:117-188) |
image_triangle.streampetr_preprocess(image, *, preset="autoware_main", channels="rgb", dst_hw=(480, 640), fma=False, out=None) |
autoware_camera_streampetr (universe main @ 9ceaccf, lib/network/preprocess_kernel.cu:73-176 + camera_data_store.cpp:287-316; research/streampetr/SPEC.md 3.1): one uint8 (H, W, 3) frame (channels = its byte order) -> the float32 network input (3, 480, 640): resize = max(480 / H, 640 / W) (float32), the virtual resized image ((int)(H * resize), (int)(W * resize)), top rows cropped (bottom kept) and the width centred, a PIL-style triangle filter with adaptive support (centre = (r + 0.5) * scale - 0.5, taps ceilf(centre - support) .. floorf(centre + support), `w = max(0, 1 - |
image_triangle.streampetr_preprocess_batch(images, *, preset, channels, dst_hw, fma, workers=1, out=None) |
the cameras of one frame -> (N, 3, 480, 640) float32 (each camera its own size); workers threads over the cameras |
image_triangle.autoware_resize_geometry(src_h, src_w, dst_h=480, dst_w=640) -> TriangleGeometry, triangle_resize_lut(src_hw, dst_hw=(480, 640), *, geometry=None, fma=False) -> TriangleResizeLUT |
calculate_image_processing_params in float32 (src_hw, resized_hw, roi_hw, roi_start (y, x), resize, to_dict()); the cached per-size tap tables (rows, row_w, cols, col_w; zero-weight padding) with .apply(image u8 HxWxC) -> (C, roi_h, roi_w) float32 weighted means (before the normalisation; any channel count). geometry= serves other nodes of the kernel family (BEVFusion's camera branch: verify its kernel first) |
image_triangle.fma_f32(a, b, c) |
fmaf on float32 arrays: one rounding, exact (TwoSum-corrected float64; a float64 sum landing on a float32 tie is broken towards the exact value) |
Facts: 0 mismatches against the scalar kernel on 12 source sizes and on the research golden input of the Autoware
test image; 40-170 ms per frame on the shared host (1080p about 130 ms), a few ms per new size for the tables. The
BEVDet gather equals the research golden network input of the nuScenes key-frame sample0 bit for bit
(tests/host/test_image_bevdet_host.py). The BEVFormer pipeline equals OpenCV 4.8.1 and 4.11.0 bit for bit on
arbitrary float images for every x0.8 down-scale (the textbook two-product interpolation differs in ~10 % of the
samples), the research reference's OpenCV pipeline on random uint8 images, and the research golden image of the
nuScenes key-frame sample0 (tests/host/test_image_bevformer_host.py, which also holds a scalar exact-rational
oracle); 0.5 s per 6 x 1600x900 frame single-threaded on the shared host, 0.15 s with workers=4 (OpenCV itself:
0.16 s). The device version (K9, uint8 upload + on-device resample) is an optimization item. Presets still to add
(their ports): BEVFusion triangle resize with truncation (PLAN.md C14; ttaw.image_triangle holds the same kernel
family). The METEOR INTER_AREA preset (ttaw.image_area) equals cv2.resize of OpenCV 4.8.1 and 4.11.0 bit
for bit at every METEOR source size tested and a scalar transliteration of resize.cpp on small geometries
(tests/host/test_image_area_host.py); about 0.1-0.15 s per 1920x1080 frame, 0.4 s at 2880x1860, numpy single thread.
The SceneSeg preset (ttaw.image_linear, 0.16.0) equals cv2.resize INTER_LINEAR / INTER_NEAREST and OpenCV's
subtract / divide scalar arithmetic of OpenCV 4.8.1 and 4.11.0 bit for bit on 23 fixed and 40 random geometries
(down / up, odd sizes, 1-4 channels, the exact-2x and copy shortcuts), a scalar transliteration of resize.cpp, the
23 research device inputs of the SceneSeg public-data set (input_u8_bgr_640x320.png) and, within one ulp, the SPEC
reference's float pre-processing (tests/host/test_image_sceneseg_host.py); 25-35 ms per 640x320 squash (414x727 to
1080p) and about 8 ms for the normalisation on the shared host, numpy single thread (OpenCV: 2-3 ms).
The StreamPETR preset (ttaw.image_triangle, 0.17.0) equals a scalar transliteration of the CUDA kernel bit for bit
in both contraction models (4 small geometries: down / up-scaling, crops, identity; 300 sampled pixels + the corners of
a 1920x1080 -> 480x640 frame), a scalar transliteration of calculate_image_processing_params on 15 source sizes, the
SPEC's numpy port (research/streampetr/scripts/sp_common.py) and the sha256 of the float32 network inputs of the
research goldens of the PandaSet ship sample (tests/host/test_image_triangle_host.py). The two contraction models
differ in ~43 % of the inputs by at most ~3e-5 (normalised units; the centre's last ulp moves the tap weights), far
below the device's bf16 input rounding. About 0.25-0.4 s per 1920x1080 camera on the shared host (numpy, one
thread), ~1.1 s per five-camera frame with workers=4.
15. LiDAR host pipeline (C10 ttaw.geometry, C11 ttaw.pointcloud, C12 ttaw.voxelize, C15 ttaw.nms, C16 ttaw.decode)
Host pre- and post-processing of the Autoware LiDAR detectors, built by the CenterPoint port (research/centerpoint/ SPEC.md sections 3 and 5) for CP, PP, TF and BF. numpy only. Each reproduces the deployed C++ / CUDA, including its float32 arithmetic where that decides a cell or a threshold; the non-deterministic parts of Autoware (time-seeded point shuffle, atomic slot / pillar order, unstable sorts) become fixed, documented policies (PLAN.md D1).
from .ttaw.pointcloud import StreamDensifiers, densify_sweeps, nonzero_rows
from .ttaw.voxelize import PillarGridSpec, assign_pillars, decorate_pillar_features, canvas_gather_index
from .ttaw.decode import CenterHeadDecodeConfig, decode_centerhead, sort_by_score, to_detected_objects
from .ttaw.nms import circle_nms, iou_bev_nms, ClassRemapper
spec = PillarGridSpec((-76.8, -76.8, -4.0), (76.8, 76.8, 6.0), (0.32, 0.32, 10.0), 32, 40000, order="flipped_x")
cache = StreamDensifiers(num_past_frames=1).get("front") # Autoware's PointCloudDensification, per stream
cache.enqueue(xyz, stamp_s, T_world_from_ego) # False: no pose -> frame skipped, not cached
pts, info = cache.sweep_points() # (N, 4) x, y, z, time_lag; current sweep first
pil = assign_pillars(pts, spec, shuffle_seed=0) # first 32 per cell, ids in flipped-x order
feats = decorate_pillar_features(pil, spec, encoder_in_feature_size=9) # (40000, 32, 9) generateFeatures_kernel
index = canvas_gather_index(pil.coords, pil.num_pillars, spec, sentinel=40000) # NHWC canvas = table[index]
boxes = sort_by_score(decode_centerhead(heads, cfg)) # cfg = CenterHeadDecodeConfig.create(...)
keep = circle_nms(boxes.x, boxes.y, 0.5)
objs = to_detected_objects(boxes.take(keep), cfg.class_names) # ObjectClassification labels, yaw_ros
objs = objs.take(iou_bev_nms(objs.x, objs.y, objs.length, objs.width, objs.yaw, objs.label))
objs.label = ClassRemapper.from_params(remapper_yaml_params).apply(objs.label, objs.length, objs.width)
| module / name | meaning |
|---|---|
geometry.compose, invert_rigid, transform_points, as_transform, transform_from_xyz_rpy, rotation_from_rpy |
float64 rigid transforms (T_a_from_b); as_transform accepts every io.parse_transform spelling |
geometry.invert_affine_f32, compose_f32, transform_points_f32 |
Autoware's float32 path: Eigen::Affine3f inverse (cofactors, column-0 determinant) and product, and generateSweepPoints_kernel (((m0 x + m1 y) + m2 z) + m3, no FMA) |
geometry.mmdet_yaw_to_ros, ros_yaw_to_mmdet, quaternion_wxyz_from_yaw, yaw_from_quaternion_wxyz, yaw_from_rotation, wrap_angle |
yaw conventions (yaw_ros = -yaw_net - pi/2, float result as ros_utils.cpp:55) |
pointcloud.SweepDensifier(num_past_frames=1, cloud_capacity=2_000_000) |
one stream's cache: .enqueue(xyz, stamp_s, T_world_from_sensor=None, *, intensity=None) -> bool (pose required when num_past_frames > 0; a missing pose leaves the cache unchanged), .sweep_points(*, point_feature_size=4|5) -> (points, DensifyInfo) (float32 world2current @ past2world per sweep, time_lag = float32(t_now - t_sweep), capacity drops a sweep and every older one), .reset(), .stamps, .describe() |
pointcloud.StreamDensifiers(..., max_streams=16) |
per-stream.id caches with LRU eviction: .get(id, reset=False), .forget(id), .streams, .describe() |
pointcloud.densify_sweeps(current_xyz, [(xyz, time_lag_s, T_current_from_sweep)], ...) |
the stateless client-side variant -> (points, DensifyInfo) |
pointcloud.finite_rows, nonzero_rows, range_mask, seeded_permutation(n, seed) |
hygiene masks (nonzero_rows = ` |
voxelize.PillarGridSpec(range_min, range_max, voxel_size, max_points_per_pillar=32, max_pillars=40000, order="flipped_x"|"raster") |
grid sizes in float32 as centerpoint_config.hpp (grid_x, grid_y, num_cells) |
voxelize.assign_pillars(points, spec, *, shuffle_seed=None) -> PillarSet |
deterministic pillars: optional seeded permutation, half-open range, float32 floor((v - min) / voxel) clamped, first K per cell in input order, ids in ascending cell order capped at max_pillars. PillarSet: points (P, K, F), num_points, coords (z, y, x) = (0, iy, ix), num_pillars, num_pillars_total, overflow, stats() |
voxelize.decorate_pillar_features(pillars, spec, *, encoder_in_feature_size=9|10|11) |
generateFeatures_kernel: raw columns, offsets to the slot-ordered float32 mean, offsets to the pillar centre voxel/2 + coord*voxel + min; padded slots 0 |
voxelize.scatter_canvas(features, coords, n, spec), canvas_gather_index(coords, n, spec, *, sentinel) |
scatterFeatures_kernel (C, H, W) and its gather form for the device (C19): cell iy * grid_x + ix -> pillar row or the zero sentinel row |
decode.CenterHeadDecodeConfig.create(class_names, voxel_size_xy, range_min_xy, downsample_factor, distance_bin_upper_limits, score_thresholds, yaw_norm_thresholds, has_variance=False, has_twist=False) |
Autoware's coercions (thresholds outside [0, 1) -> 0, ascending bins, one yaw threshold per class); thresholds (bins, classes); .with_score_threshold(v) |
decode.decode_centerhead(heads, cfg) -> DecodedBoxes, sort_by_score |
generateBoxes3D_kernel in float32 (first-max label, no +0.5 offset, bins, score < thr drops, yaw-norm gate, dims (w, l, h), atan2), rows in cell order; stable descending sort (ties by cell). No max-pool, no top-K |
decode.head_values_at(heads, cells) -> {name: (C, K)}, decode_cells(values, cells, grid_w, cfg) -> DecodedCells |
the same decoder at given cells, dropping none (agreement metrics: device vs fp32 reference at the same cells). DecodedCells: the DecodedBoxes fields (bit-identical to decode_centerhead's row for every cell it keeps) plus yaw_norm, distance_bin, score_threshold, yaw_norm_threshold, passes_score, passes_yaw_norm, .keep, .take, .to_boxes(); head_values_at stores the maps at a cell set (compact goldens) |
decode.to_detected_objects(boxes, class_names, *, has_twist=False) -> DetectedObjects |
box3DToDetectedObject: ObjectClassification labels (AUTOWARE_LABELS, LABEL_IDS, semantic_label), yaw_ros, (length, width, height), orientation_availability SIGN_UNKNOWN for car-like labels, object-frame twist; .take, .boxes_xyzlwh_yaw() (the outputs.Detections3D layout), .to_records() |
nms.circle_nms(x, y, dist) |
circleNMS on score-sorted boxes: float32 dx^2 + dy^2 < dist^2 (strict), greedy, class-agnostic |
nms.iou_bev_nms(x, y, length, width, yaw, labels, *, search_distance_2d=10.0, iou_threshold=0.1, scores=None, sort=False) |
perception_utils::IouBevNms::apply: every earlier object (kept or not) suppresses, pedestrian-vs-other pairs skipped, <= search gate, break at iou > thr, keep iff max_iou <= thr; exact rotated-rectangle IoU (iou_bev, clip_convex, polygon_area, bev_box_corners) with the 1e-6 / 0.01 area guards |
nms.ClassRemapper.from_params(yaml_params) / .from_lists(allow, min, max) |
DetectionClassRemapper: .apply(labels, length, width) with bev_area = float(l * w), first allowed destination in label order |
Facts (CenterPoint port, research/centerpoint): voxel tensors bit-identical to the research reference (input order
and seed 0, including a 41,918-pillar overflow frame); decode + NMS + remapper give the same objects as two
independent research implementations on 16 head-map goldens; about 0.3 s voxelization and 30-120 ms
post-processing per 200k-point frame on the shared host (numpy). Host tests:
tests/host/test_geometry_pointcloud_host.py, test_voxelize_host.py, test_decode_nms_host.py. Still to add (their
ports): PointPainting 12-feature decoration and raster order use, TransFusion / BEVFusion decoders, OpenPCDet and
METEOR NMS variants (PLAN.md C12, C15, C16).
16. CNN ops: conv builders (C17 ttaw.ops.conv), up-sampling and interpolation (C18 ttaw.ops.upsample)
Built by the YOLOX port (PLAN.md C17 / C18; probes P5, P14). A feature map is a FeatureMap(tensor, batch, height, width, channels): the device tensor holds NHW rows of C channels (ttnn's flattened [1, 1, N*H*W, C], TILE or
ROW_MAJOR), plus the spatial size ttnn's layout no longer carries. Every builder prepares its device weights on its
first (eager) call and reuses them, so call each layer once before capturing (the TraceRunner warm-up does): a
host weight write inside a capture fails ("Writes are not supported during trace capture").
from .ttaw.ops.conv import Conv2d, KSplitConv, ConvTranspose2d, FeatureMap, concat_channels, residual_add
from .ttaw.ops.upsample import upsample_nearest, Resize2d, interp_matrix
conv = Conv2d(w_oihw, b, stride=2, activation="relu6") # padding "same" (k // 2); HiFi4 + fp32 + L1 acc
x = FeatureMap(ctx["image_bf16"], 1, 960, 960, 3) # or fmap_from_numpy(nchw, device) (under devrun)
y = conv(x) # FeatureMap [1, 1, 480*480, 32], DRAM, TILE
y = residual_add(conv2(y), y, "relu6") # RELU / RELU6 fused into ttnn.add (P5)
z = concat_channels([upsample_nearest(lateral, 2), d4]) # nearest x2: bit-exact; concat along C
head = Conv2d(w, b, output_dtype="float32") # fp32 logits for thresholds
ks = KSplitConv(w_full, b, parts=(128, 128, 128), activation="relu") # conv(concat) without the concat (P14)
up = ConvTranspose2d(w_iokk, b, stride=4, activation="relu") # k = 2: conv_transpose2d; k >= 3: linear + d2s (P5)
rs = Resize2d((16, 44), (64, 176), channels=256, mode="linear", align_corners=True) # FPN_LSS-style bilinear
| name | meaning |
|---|---|
Conv2d(weight, bias=None, *, stride=1, padding="same", dilation=1, groups=1, activation=None, precision=CONV_PRECISION, output_dtype="bfloat16", output_layout="tile", slicing="auto", output_memory_config="dram", conv_config=None, weight_terms=1, name="conv") |
ttnn.conv2d with weights prepared once per input spec. __call__(fmap) (or a raw tensor + batch=, height=, width=) -> FeatureMap. slicing="auto": a DRAM input runs ttnn's DRAM-sliced path with automatic slice counts (P14: traceable; explicit counts are often rejected) and writes DRAM interleaved; an L1 input runs the L1 path; "l1" = Conv2dL1FullSliceConfig; ("height"|"width", n) explicit (diagnostics). 1x1 stride-1 convs are matmuls on the L1 path; output_memory_config ("dram" default, "l1", None = the op's sharded output) places their output. conv_config: extra ttnn.Conv2dConfig fields (optimization knobs: act_block_h_override, shard_layout, double buffers, ...). is_matmul, out_hw(h, w), takes_l1_path(t), prepared_specs, release(), describe() |
weight_terms=2 |
the weight and the bias as two bf16 terms (hi = bf16(x), lo = bf16(x - hi): ~16 significant bits; host_terms()), run as two convs into fp32 outputs, summed in fp32 with the activation fused into the add, cast to output_dtype (TILE outputs only): ~fp32 weights and bias for 2x the MACs and 3 more programs. bf16 rounding of weights and biases, not of activations, dominates a deep CNN's error (a per-channel bias error is the same at every pixel): YOLOX's mask agreement on PandaSet side cameras is 97.5 % with bf16 weights and 99.3 % with two terms (CPU emulation; the device matches it). The device's float32 weight path (precision=":w=float32") is no substitute: 1.5x less error than bf16 vs 3.5x (fp32 output) for two terms; with a bf16 output two terms land within 1.05x of the output-rounding floor (logs/yolox/m3_weight_precision_probe.log, m3_c17_weight_terms_device_r2.log) |
CONV_PRECISION |
"HiFi4+fp32+l1acc", the default of every builder. Without packer L1 accumulation a conv whose inner dim spans several blocks keeps its partial sums in the output dtype (bf16) between blocks (conv2d_op_program_factory_common.cpp:175-178): YOLOX 3x3 256 -> 256 PCC 0.99998 / max abs 0.25 vs 0.9999993 / 0.025 with it, at the same speed (logs/ttaw/c17_c18_device_results*.json) |
normalize_activation, unary_with_param, apply_activation, FUSED_ACTIVATIONS, BINARY_FUSED_ACTIVATIONS |
fused conv activations: relu, relu6, silu, gelu (exact), sigmoid; binary fused: relu, relu6. HARDSWISH is refused (fused, it is silently skipped: P5); apply ttnn.hardswish as its own op |
KSplitConv(weight, bias, parts, *, activation=None, output_dtype="bfloat16", accumulate_dtype="bfloat16", **conv_kwargs) |
conv(concat(x_1..x_n)) as sum_i conv_i(x_i) (bias on the first part, activation fused into the last add or after it); __call__([fmaps]). accumulate_dtype="float32" keeps partials fp32. split_k_weights(w, parts) is the host split |
ConvTranspose2d(weight [Cin, Cout, k, k], bias=None, *, stride=k, activation=None, precision=CONV_PRECISION, output_dtype="bfloat16", method="auto", output_layout="tile", memory_config="dram") |
kernel == stride, no padding. auto: k = 2 -> ttnn.conv_transpose2d, k >= 3 -> ttnn.linear (bias + activation fused) + depth-to-space (RM reshape / permute / reshape); linear_weights() gives the (i, j, c)-ordered host matrix |
merge_sibling_convs(weights, biases) -> (w, b, sizes), split_channels(fmap, sizes) |
one conv for sibling convs reading the same input (exact), split by ttnn.slice on C |
concat_channels(fmaps, *, memory_config="dram"), residual_add(a, b, activation=None) |
glue on maps of one spatial size |
FeatureMap (rows, hw, with_tensor(t, channels=None), deallocate()), fmap_from_numpy(nchw, device, dtype, layout, memory_config), fmap_to_numpy(fmap), conv_padding, conv_out_hw, is_dram |
helpers (TILE uploads touch the device: run under bin/devrun) |
upsample_nearest(fmap, scale=2, *, output_layout="tile", memory_config="dram") |
integer-factor nearest (floor): TILE -> RM -> [N, H, W, C] -> ttnn.upsample -> [1, 1, N*sH*H*sW*W, C] -> TILE. Bit-exact (ONNX Resize nearest / asymmetric / floor) |
interp_matrix(n_in, n_out, *, mode="linear"|"nearest", align_corners=False, scale=None) -> [n_out, n_in] float64; resize_matrices(in_hw, out_hw, ...) |
1-D interpolation matrices equal to F.interpolate (torch's half-pixel rule clamped at 0, align_corners, the scale_factor rule); host tests compare them with torch at 1e-12 |
Resize2d(in_hw, out_hw, *, channels, batch=1, mode="linear", align_corners=False, scale=None, dtype="float32", output_dtype="bfloat16", precision="accurate", output_layout="tile", tail="auto") |
separable resize on the device for any sizes: W pass [N*H, C, W] @ A_w^T (transposes around one matmul), H pass kron(I_N, A_h) @ [N*H, W'*C]; tail (0.23.1): tile reshapes the H-pass result in TILE layout, rm via ROW_MAJOR (small DRAM pages: deadlocks Blackhole under ETH dispatch, SYS-1419), auto = tile for destination pages < 4 KB unless the tt-metal tree has the reshape patch (ttaw.device.reshape_patch_present, 0.23.2: then rm, the patched single-kernel path) (env TTAW_RESIZE2D_TAIL); fp32 operands by default (bf16 cannot hold weights like 511/1023; a device fp32 matmul is TF32-like, P12). reference(nchw) is its host float64 twin. ttnn.upsample's own bilinear is half-pixel only (integer scales, sharded bf16 input) |
Device results (common/tests/device/test_conv_upsample_device.py, every case eager x2, captured with cache misses
forbidden, replayed on a new input, and equal to an eager run after the capture bit for bit; log
logs/yolox/m1_c17_c18_device_r1.log / _r2_l1acc.log, results logs/ttaw/c17_c18_device_results*.json): all 54
distinct conv shapes of YOLOX seg16 (1x1 to 9x9, stride 1 / 2, 960^2 down to 30^2, fused RELU6, fp32 head preds)
PCC >= 0.9999 (>= 0.9999992 for kxk with CONV_PRECISION); K-split 7 x 64 -> 128 0.99999; ConvTranspose k = 2 / 4
0.99999; the 7 YOLOX nearest x2 shapes bit-exact (0.03-3.6 ms: the RM round trip dominates at 16 x 480^2); Resize2d
align_corners True / False, integer and non-integer, PCC >= 0.9999998. Host tests: tests/host/test_conv_upsample_host.py
(54) on the fake ttnn plus tests/host/fake_ttnn_cnn.py (conv2d / conv_transpose2d / upsample / matmul / argmax /
where / ... for CNN graph tests; install(fake) returns its undo).
17. Attention (C20, ttaw.ops.attention)
Built by the Diffusion Planner port (PLAN.md C20; probe P7). ttnn.transformer.scaled_dot_product_attention has
four traps; sdpa is the only way the ports call it:
is_causaldefaults to True: with Sq == Sk it silently runs causal attention, with Sq != Sk it is rejected.sdpaalways passesis_causal=False.- The default
scaleis1/sqrt(padded head dim)(wrong after a 16 -> 32 head padding).scale=is required. - The default 32/32 chunks are 2.5-7x slower and inaccurate over >= 32k keys.
sdpapasses a program config fromchunk_config(sq, sk)(the P7 table plus Sk-range rules) and refusesprogram_config=Noneonce Sk >= 1,000. - With a user mask the op masks the padded keys only at tile granularity: key tiles past
ceil(Sk/32)get-inf, but the last partial tile is read from the mask, whose tile padding is 0 after a host tilize (sdpa/device/kernels/dataflow/reader_interleaved.cpp, "Mask read"). Up to 31 padded keys then join the softmax (scoreq . k_pad, valuev_pad): rel-L2 0.18 instead of 0.025 at 321 x 321 with 89 valid keys, and 0.06 for P7's masked 564 x 564 case.sdparefuses a mask withSk % 32 != 0: run masked attention onaligned_keys(n)keys (the extra keys masked with-infin the mask; free, the tiles are padded anyway). Without a mask an unaligned Sk is exact (the op generates an element-wise padding mask).
from .ttaw.ops import attention as A
q, k, v = A.split_qkv(qkv, 8) # [1, 1, S, 3*H*D] -> 3 x [1, H, S, D] (one program)
q, k, v = A.split_q_kv(q_proj, kv_proj, 8) # Q from LN(x), K|V from x, same S
qh = A.split_heads(q_cross, 8) # Q-only (cross-attention; K / V hoisted elsewhere)
mask = A.expand_key_bias(ctx["key_row"], 576) # [1,1,1,Sk] bias row (persistent input) -> [1,1,Sq,Sk] DRAM
o = A.sdpa(q, k, v, scale=1 / 32 ** 0.5, attn_mask=mask, concat_heads=True) # [1, 1, Sq, H*D]
runner.run("plan", inputs={"key_row": A.key_bias_row(np.r_[valid, np.zeros(12, bool)])}) # 564 real + 12 masked
wq, bq = A.pad_head_columns(w_q, b_q, 8, 16) # head dim 16 -> 32 in the weights (exact, free)
wo = A.pad_head_rows(w_out, 8, 16)
| name | meaning |
|---|---|
sdpa(q, k, v, *, scale, attn_mask=None, program_config="auto", compute_kernel_config=None, fp32_acc=False, concat_heads=False, memory_config=None) |
[B, H, Sq, D] x [B, Hkv, Sk, D] (bf16 / bfp8 / bfp4 TILE interleaved, D % 32 == 0) -> [B, H, Sq, D], or [B, 1, Sq, H*D] with concat_heads=True (the op's output_concat_heads, no extra program). Mask: additive bf16 TILE in DRAM, [1|B, 1|H, Sq, Sk], Sk tile-aligned; -inf is safe |
chunk_config(sq, sk) -> (q_chunk, k_chunk), CHUNK_TABLE |
exact measured shapes (evidence string per entry), else Sk >= 16,384 q64 k1024; >= 4,096 q64 k512; >= 1,000 q64 k128; shorter q64 k64 (32 when the sequence fits one tile) |
sdpa_program_config(device, sq, sk, *, q_chunk=None, k_chunk=None, exp_approx_mode=None), sdpa_compute_config(*, fp32_acc=False, fidelity="HiFi2", approx=True) |
SDPAProgramConfig over the grid read from the device; the explicit compute config equal to the op default (P7: HiFi4 / exact exp do not help; fp32_acc halves the max error at ~1.4x the time) |
split_heads(x, num_heads), split_qkv(qkv, num_heads, *, num_kv_heads=None), split_q_kv(q, kv, num_heads, *, num_kv_heads=None), merge_heads(x) |
[B, 1, S, H*D] <-> [B, H, S, D] through ttnn.experimental.nlp_create_qkv_heads / nlp_concat_heads (one program each, bit-exact moves, the logical S kept). split_q_kv needs the same S for Q and K / V; cross-attention splits Q and K / V separately |
key_bias_row(valid) -> [1, 1, 1, Sk], key_bias(valid, sq) -> [1, 1, Sq, Sk], expand_key_bias(row, sq), aligned_keys(n) |
host masks (0 / -inf, float32; upload bf16 TILE) and the traced expansion of a persistent bias row (ttnn.repeat, DRAM; one program per plan, not per call) |
padded_head_dim(d), pad_head_dim(x), pad_head_columns(w, b, H, D), pad_head_rows(w, H, D) |
head dim padding to a multiple of 32 in the projection weights (exact: the padded output columns are 0, P7) |
attention_reference(q, k, v, *, scale, bias=None), attention_matmul(q, k, v, *, scale, attn_mask=None, compute_kernel_config=None) |
float64 numpy oracle; small-MHA fallback with two matmuls + softmax for what SDPA rejects (e.g. fp32 Q / K / V; tile-aligned Sk) |
Device results (common/tests/device/test_attention_device.py, 27 cases through TraceRunner: eager x2
bit-identical, strict capture, replay on a new input set equal to an eager run bit for bit; set A nearly uniform
attention, set B peaked; gates frozen in test_attention_device.gates.json; logs/diffusion-planner/c20_device_r2.log,
results logs/ttaw/attention_results.json): PCC vs float64 0.99966-0.99983 at the DP shapes (576 x 576 and 352 x 352
masked, 352 / 321 x 564), TF self 500 x 500 head dim 16 -> 32, SP self 900 x 1,668 and BF cross 500 x 32,400
(q64 k1024, 0.46 ms); rel-L2 0.024-0.025 everywhere; split / merge / mask expansion bit-exact; the fp32 matmul
fallback PCC 0.999998. Trace ms per call: DP fusion 576 x 576 masked 0.061-0.092, DiT self 352 x 352 masked 0.063,
DiT cross 352 x 564 0.033. A canary (test_raw_op_mask_padding_leak) keeps the trap-4 evidence (raw op, 321 keys:
rel-L2 0.183); when tt-metal masks partial tiles it fails and the refusal can be relaxed. Watcher bring-up of the
smallest shape (TT_METAL_WATCHER=2) clean: logs/diffusion-planner/c20_watcher_smoke.log. Host tests:
tests/host/test_attention_host.py (27) with tests/host/fake_ttnn_attention.py (an SDPA fake that keeps the op's
defaults, so a caller forgetting is_causal=False / scale fails; head ops, repeat, matmul, softmax).
18. LiDAR device modules: gather-form scatter (C19 ttaw.ops.gather), SECOND + SECONDFPN (C24 ttaw.models.second), CenterHead (C25 ttaw.models.centerhead)
Built by the CenterPoint port (PLAN.md C19, C24, C25; probes P4, P5, P14) on the C17 builders of section 16, for CP, PP, TF and BF. Every module is built from BN-folded host (numpy) weights, prepares its device weights on its first (eager) call, keeps every output in DRAM (interleaved TILE; L1 residency is optimization work) and traces.
from .ttaw.ops.gather import check_index, scatter_rows, gather_rows, scatter_rows_numpy
from .ttaw.models.second import ConvSpec, Second, SecondFPN, RowLinear, free_maps
from .ttaw.models.centerhead import CenterHead, merge_heads
from .ttaw.ops.conv import FeatureMap
index = check_index(prep.canvas_index, num_rows=prep.num_pillars, sentinel=40000) # host, [1, H*W] uint32
backbone = Second([[ConvSpec(w, b, stride=s) for w, b, s in block] for block in blocks]) # 3x3 + ReLU
neck = SecondFPN([ConvSpec(w0, b0), ConvSpec(w1, b1, stride=2, kind="conv_transpose"),
ConvSpec(w2, b2, stride=4, kind="conv_transpose")]) # 1x1, k2 ConvT, k4 ConvT
head = CenterHead(shared_spec, [128, 128, 128], [(name, hidden_spec, final_spec), ...]) # K-split + merged heads
def forward(ctx): # a TraceRunner variant
canvas, table = scatter_rows(pillar_rows, ctx["canvas_index"]) # [1,1,P,32] rows -> [1,1,H*W,32] TILE
blocks = backbone(FeatureMap(canvas, 1, 480, 480, 32))
return head(neck(blocks)).tensor # one fp32 [1, 1, H*W, 32] tensor (15 channels used)
maps = head.unpack(runner("default", inputs=...), 480, 480) # host: {name: (C, H, W)}
| name | meaning |
|---|---|
gather.check_index(index, *, num_rows, sentinel=None) |
host validation of a gather index -> contiguous uint32 [1, N]; every entry in [0, num_rows) or equal to sentinel, else ValueError (an out-of-range index reads outside the table on the device) |
gather.gather_rows(index, table, *, sentinel=None, layout="tile", memory_config=None) |
ttnn.embedding of a UINT32 [1, N] index into a bf16 ROW_MAJOR [1, 1, V, D] table -> [1, 1, N, D]; with sentinel it is P4's fast EmbeddingsType.PADDED form (padding_idx=sentinel), which returns the table's own row there: the row must hold zeros |
gather.with_zero_rows(rows, count=1) |
[1, 1, P, D] rows (TILE or ROW_MAJOR) -> ROW_MAJOR table [1, 1, P + count, D] with zero rows appended (one ttnn.pad) |
gather.scatter_rows(rows, index, *, sentinel=None, layout="tile", memory_config=None) -> (canvas, table) |
the gather-form scatter: canvas[n] = rows[index[n]], zeros where index[n] == sentinel (default P); static shapes, no canvas clear; P4: 0.21 ms at CP 480 x 480 |
gather.gather_rows_numpy, gather.scatter_rows_numpy(rows, index, *, sentinel) |
host oracles |
second.ConvSpec(weight, bias, stride=1, padding=None, relu=True, kind="conv"|"conv_transpose", name="") |
one BN-folded layer: conv weight [Cout, Cin, k, k] (padding None = k // 2), ConvTranspose weight [Cin, Cout, s, s] (kernel == stride) |
second.Second(blocks, *, policy=None, prefix="backbone") |
SECOND: blocks[i] = list of 3x3 ConvSpec (block stride on the first); __call__(fmap) -> [block outputs] (input kept, inner layer outputs freed), out_shapes(h, w), release(), describe(). C17 Conv2d layers: a DRAM input runs ttnn's automatic DRAM slicing (P14) |
second.SecondFPN(deblocks, *, policy=None, prefix="neck") |
one deblock per block output: "conv" 1x1 -> RowLinear; ConvTranspose k = 1 -> RowLinear, k = 2 -> ttnn.conv_transpose2d, k >= 3 -> linear + depth-to-space (C17 ConvTranspose2d, P5); every output moved to DRAM; the concat is never built |
second.RowLinear(weight, bias=None, *, activation=None|"relu", precision=DEFAULT_PRECISION, output_dtype="bfloat16") |
a 1x1 conv / per-row dense layer: ttnn.linear on [..., R, K] TILE rows with an explicit compute config and core_grid from the device (keeps the ReLU fused), output DRAM; weight [K, N] or [N, K, 1, 1]; device weights uploaded on the first call |
second.DEFAULT_PRECISION |
C17's CONV_PRECISION (HiFi4+fp32+l1acc); a ttaw.precision.PrecisionPolicy overrides per module (backbone.block<i>.conv<j>, neck.deblock<i>, head.shared, head.hidden, head.out); a precision's activations dtype (:a=fp32) is the dtype of that module's OUTPUT (bf16 by default; the K-split partial sums follow the shared conv's) |
second.free_maps(*maps) |
deallocate feature maps / tensors that still hold their buffer (force=False: views are left alone, P2) |
second.second_numpy(blocks, x_nchw), second.second_fpn_numpy(deblocks, blocks_nchw) |
fp32 torch oracles (NCHW) |
centerhead.CenterHead(shared, split, heads, *, policy=None, prefix="head", output_dtype="float32", k_split=True, accumulate_dtype="bfloat16") |
relu(conv3x3(concat(inputs))) as a C17 KSplitConv over split (P14: 3.6 vs 14.8 ms at CP 480 x 480, more accurate), then the merged heads: relu(x @ [Wh_1 | ... | Wh_n] + bh) and the block-diagonal @ Wo + bo (two RowLinears) -> ONE [1, 1, H*W, C_pad] tensor (C_pad = sum of head channels rounded up to 32). shared_forward(inputs), heads_forward(shared), __call__(inputs), unpack(rows, h, w) -> {name: (C, H, W)}, slices, release(), describe() |
centerhead.merge_heads(heads, *, pad_to=32) |
host: (Wh, bh, Wo, bo, slices) of [(name, hidden 1x1 + ReLU, final 1x1)] (exact rewrite) |
centerhead.centerhead_numpy(shared, heads, inputs_nchw) |
fp32 torch oracle of the literal graph (concat, conv, n heads) |
Device results (common/tests/device/test_lidar_models_device.py, TraceRunner: eager x2, strict capture, replay on a
new input, replay == eager bit for bit): the CP scatter at its real shape (40,000 rows into 480 x 480) bit-exact,
sentinel rows zero; CP-shaped SECOND (4 / 6 / 6 convs, 64 / 128 / 256 channels) + SECONDFPN + CenterHead with random
weights at 64 x 64 and 128 x 128 (first stride 1 and 2) PCC >= 0.999 for every block, deblock, the shared conv and
every head vs the torch fp32 oracle (logs/ttaw/c19_c24_c25_device_results.json). The CenterPoint bundle runs the same
modules with the real weights (bundles/centerpoint-p150/PORT_LOG.md). Host tests:
tests/host/test_lidar_models_host.py (13) on the fake ttnn plus tests/host/fake_ttnn_lidar.py (embedding with
P4's PADDED semantics, max, EmbeddingsType; install(fake) returns its undo).
19. grid_sample helpers (C23, ttaw.ops.deform)
Built by the BEVDet port (PLAN.md C23; probe P9) for AlignBEV (8 history slots warped by per-frame affine grids)
and FPN_LSS's align_corners bilinear up-sampling; BEVFormer (DCN / rotate) and METEOR extend it. Grids are built on
the host (numpy, float32) and uploaded as ROW_MAJOR fp32 tensors (a per-frame RT-dev value or a static input);
grid_sample is the device call with the P9 rules checked first.
from .ttaw.ops import deform
ring = runner.add_state("ring", shape=(8, 128, 128, 96), dtype="bfloat16", layout=ttnn.ROW_MAJOR_LAYOUT) # C % 32
grid = deform.affine_grid(transforms_8x6, (128, 128), valid=[h >= 0 for h in history]) # [8, 128, 128, 2] fp32
runner.add_param("grid", grid, shape=grid.shape, dtype="float32", layout=ttnn.ROW_MAJOR_LAYOUT)
aligned = deform.grid_sample(ctx["ring"], ctx["grid"]) # in a variant: bilinear, zeros, align_corners=False
up = deform.grid_sample(f2_rm, static_grid) # static_grid = deform.resize_grid((16, 16), (64, 64))
| name | meaning |
|---|---|
pixel_to_grid(px, size) |
pixel coordinate (0 = first pixel centre) -> normalised (2 px + 1) / size - 1 (float32): the align_corners=False form, exact for integer px when size is a power of two |
kernel_pixels(g, size) |
what the device kernel recovers: g * size/2 + (size - 1)/2 in float32 (grid_sample_reader_common.hpp) |
affine_pixel_coords(transforms, out_hw) |
ix = a w + b h + c, iy = d w + e h + f per batch (float32, one rounding per op): the BEVDet AlignBEV pixel affine |
affine_grid(transforms [N, 6], in_hw, out_hw=None, *, valid=None) |
fp32 grid [N, H, W, 2]; valid[n] == False -> OUTSIDE (an exact zero sample: an empty history slot) |
resize_pixel_coords(in, out, align_corners=True), resize_grid(in_hw, out_hw, *, align_corners=True, batch=1) |
static bilinear-resize grids (float64 source pixels, align_corners=False normalisation); for align_corners=True every source pixel is inside, so it equals F.interpolate up to the weight truncation |
padded_channels(c), pad_channels(x) |
the input channel rule (multiple of 32; pad channels stay exactly 0 through the op) |
emulate_grid_sample(x, grid) |
host emulation of the kernel (fp32 fractions, weights truncated to bf16, taps outside skipped): the device test's oracle |
grid_sample(x, grid, *, compute_kernel_config=None, memory_config=None, batch_output_channels=False) |
ttnn.grid_sample bilinear / zeros / align_corners=False; refuses a TILE input or grid, input C % 32 != 0, a grid batch != input batch, a bf16 grid, and a FLOAT32 input with fp32_dest_acc_en (P9 defect); output DRAM by default |
OUTSIDE |
-4.0: a normalised coordinate whose four taps miss every input |
Rules (P9): fp32 grids for every data-dependent grid (bf16 grids: PCC 0.9988 at 53 px, 0.992 at 128 px); C padded to
a multiple of 32; grids expanded per batch (no broadcast); ROW_MAJOR interleaved / height-sharded input and grid;
never a FLOAT32 input with fp32_dest_acc_en; call the op with align_corners=False on the align_corners=False
normalisation of the pixel coordinates you mean (zero padding acts in pixel space, so it is the same sampling as the
model's align_corners=True formulation, but integer coordinates stay exact: an identity warp is a bit-exact copy;
the literal ac1 normalisation leaves 2.8-7.6 % of values off by up to 1/32). The bilinear weights are truncated to
bf16 (0.33-0.6 % rrmse floor; K5 removes it); C18 Resize2d (fp32 interpolation matmuls) is the accurate
alternative for static resizes.
Device results (tests/device/test_deform_device.py, TraceRunner: eager warm-up, strict capture, replay on new
data, replay == eager bit for bit; logs/ttaw/deform_device_results.json): AlignBEV [8, 128, 128, 96] identity slots
bit-exact, OUTSIDE slots exactly 0, pad channels 0, rigid-motion slots PCC >= 0.9999 vs torch fp32 and within 2
bf16 ulps of emulate_grid_sample; the FPN_LSS resizes 16 -> 64 (640 ch) and 64 -> 128 (512 ch) PCC >= 0.99999 vs
F.interpolate(align_corners=True), corners exact. Host tests: tests/host/test_deform_host.py (13). The BEVDet
bundle runs the same calls at its real shapes (bundles/bevdet-p150/PORT_LOG.md: AlignBEV moving slots PCC 0.999995
vs the CPU reference).
20. ResNet builders (C27, ttaw.models.resnet)
Built by the BEVDet port (PLAN.md C27) for its ResNet-50 image backbone (6 cameras as batch) and CustomResNet BEV
encoder; BEVFormer and METEOR reuse it. Built on the C17 builders of section 16 from BN-folded host weights; every op
traces, each conv prepares its device weights on its first (eager) call, every activation is a DRAM-interleaved TILE
[1, 1, N*H*W, C] map and intermediates are freed as soon as they are consumed.
from .ttaw.models import resnet as rn
prov = lambda module: rn.ConvParams(w[module], b[module], stride, padding_tblr, dilation, groups) # BN folded
stem = rn.ResNetStem(prov, "img_backbone.conv1") # 7x7 s2 + ReLU + maxpool 3x3 s2
r50 = rn.ResNetStages(prov, rn.mmdet_names("img_backbone"), [3, 4, 6, 3]) # bottleneck, pytorch style
c4, c5 = r50(stem(img_fmap), keep=(2, 3), free_input=True) # FPN inputs; the rest freed
bev = rn.ResNetStages(prov, rn.custom_resnet_names("img_bev_encoder_backbone"), [2, 2, 2], block="basic",
factory=rn.conv_factory("HiFi4+fp32+l1acc:w=fp32"))
f0, f1, f2 = bev(bev_in, keep=(0, 1, 2))
| name | meaning |
|---|---|
ConvParams(weight [Cout, Cin/g, kh, kw], bias=None, stride=(1, 1), padding=(t, b, l, r), dilation=(1, 1), groups=1) |
one BN-folded conv as a provider returns it (numpy float32); a provider is module name -> ConvParams (BEVDet's reads OnnxWeights by consuming node, so no kernel size or stride is re-typed) |
conv_factory(precision=DEFAULT_PRECISION, **conv_kwargs) |
(module, params, activation) -> callable: one C17 Conv2d per module; precision a spec / Precision or a PrecisionPolicy resolved per module name. DCN hook: pass your own factory and return a deformable layer for the modules you name |
mmdet_names(prefix), custom_resnet_names(prefix) |
(stage, block) -> {conv1, conv2[, conv3], downsample, block}: layer{s+1}.{b}.conv{k} / .downsample.0, or layers.{s}.{b}.conv{k} / .downsample |
Bottleneck(provider, names, *, has_downsample, factory) |
relu(conv3(relu(conv2(relu(conv1(x))))) + idt), stride on conv2 (pytorch style) |
BasicBlock(provider, names, *, has_downsample, factory) |
relu(conv2(relu(conv1(x))) + idt); the projection may be any conv (CustomResNet: 3x3 stride 2) |
ResNetStem(provider, conv_module, *, factory=None, pool=True) |
maxpool3x3s2p1(relu(conv1(x))); the pool runs DRAM-sliced and writes TILE |
ResNetStages(provider, names, layers, *, block="bottleneck"|"basic", factory=None, downsample_first=()) |
__call__(x, keep=(), *, free_input=False) -> [kept stage outputs] (input kept unless free_input; every other block output freed once consumed); stages, convs() |
maxpool_out_hw(h, w), DEFAULT_PRECISION (= C17 CONV_PRECISION) |
helpers |
resnet_numpy(x_nchw, provider, names, layers, *, block, stem=None, keep=(), downsample_first=()) |
fp32 torch oracle of the same blocks |
Precision: the ports pick it. BEVDet runs HiFi4+fp32+l1acc:w=fp32 (fp32 weights and biases: the bf16 rounding of
the BN-folded biases, applied per channel at every pixel, cost it depth PCC 0.9998 vs 0.99993 and 3 of the 52 sample0
boxes, at no measured time cost); C17's weight_terms=2 is the other accurate option. Device results
(tests/device/test_resnet_device.py, TraceRunner protocol; logs/ttaw/resnet_device_results.json): stem + two
bottleneck stages on [6, 3, 128, 352] and a CustomResNet stage (3x3 s2 projection) on [1, 864, 128, 128] -> 160,
random weights, PCC >= 0.999 vs resnet_numpy with both precisions, replay == eager bit for bit. Host tests:
tests/host/test_resnet_host.py (6, fake ttnn + fake_ttnn_cnn + a local max-pool fake). The BEVDet bundle runs the
real ResNet-50 / CustomResNet (bundles/bevdet-p150/PORT_LOG.md: depth / feat PCC 0.99994, chained heads >= 0.9995
vs the fp32 CPU reference).
21. Query heads: top-k (C21 ttaw.ops.topk), heatmap peaks (C22 ttaw.ops.heatmap), TransFusion head (C26 ttaw.models.transfusion_head)
Built by the TransFusion port (PLAN.md C21, C22, C26; probes P6, P7, P8, P13) for TF, BF and PT: everything after the dense heatmap -- the 3x3 local maximum, the top-K proposal selection and the query decoder -- as one traceable device graph with one packed output. numpy only at import.
from .ttaw.ops.heatmap import LocalMax, sigmoid_heat
from .ttaw.ops.topk import TopKSelect, selection_metrics
from .ttaw.models.transfusion_head import QueryHeadWeights, TransFusionQueryHead, assemble_transfusion
local_max = LocalMax(192, 192, 5, rule="transfusion") # "bevfusion" / "ptv3": pooled_classes, passing cells
head = TransFusionQueryHead(QueryHeadWeights(...), prefix="decoder") # host numpy weights, BN folded
head.prepare(device) # tables + LayerNorm rows (before any capture)
def forward(ctx): # a TraceRunner variant
heat, _ = sigmoid_heat(logits_fp32) # fp32 sigmoid -> bf16 (max_pool2d / top-k are bf16)
heat_nms = local_max(heat) # [1, 1, H*W, C] TILE
out = head(lidar_feat, heat_nms, ctx.get("topk_indices")) # None: device top-k; else teacher forcing
return pack_outputs(out) # heads (K x 32 fp32), query heat (Kp x 32), indices
raw = runner("default", inputs=...)
outs = assemble_transfusion(head.unpack_heads(raw["heads"]), raw["query_heat"], raw["indices"], bev_pos,
num_classes=5, num_proposals=500, cells=36864) # cls_score0 / bbox_pred0 / dir_cls_pred0
| name | meaning |
|---|---|
heatmap.local_max_masks(h, w, classes, *, rule="transfusion"|"bevfusion"|"ptv3", pooled_classes=None, kernel=3) |
the constant masks (m_pass, m_eq) (NHWC rows [h*w, classes], 0 / 1, disjoint) of keep = m_pass + (heat == maxpool_kxk_s1_p(k//2)(heat)) * m_eq: TF interior m_eq, no pass; BF pooled classes (default 0-3) as TF, the others pass everywhere; PT m_pass = 1 - m_eq (border and unpooled classes pass) |
heatmap.LocalMax(h, w, classes, *, rule, pooled_classes=None, kernel=3, memory_config="dram") |
the device chain on a bf16 TILE [1, 1, h*w, classes] heat (max_pool2d p1, eq, multiply by m_eq or addcmul(m_pass, eq, m_eq), multiply by the heat); masks uploaded by the first call; .numpy(heat_rows) its host twin; refuses fp32 (max_pool2d is bf16 only) |
heatmap.sigmoid_heat(logits, *, dtype="bfloat16") -> (heat, sig) |
the sigmoid in the logits' dtype (fp32 logits: fp32 sigmoid), cast once; sigmoid_heat_numpy its host model |
heatmap.local_max_numpy, local_max_literal_numpy(heat_nchw, rule) |
oracles: the masked chain, and each model's literal p0 + border / unpooled-class form |
topk.TopKSelect(cells, classes, k) |
__call__(heat_nhwc) -> (indices, pos, cls) (uint32 ROW_MAJOR [1, padded_k(k)]): class_major_row (transpose + untilize + reshape to one bf16 row, index c * cells + p = the ONNX TopK order), topk_indices (topk_large_indices, descending, ties unspecified), decode |
topk.decode_class_major(indices, cells, classes, *, clamp=True) |
exact decode with fp32 comparisons (cls = sum_c idx >= c * cells, pos = idx - cls * cells; no division), clamped so a 0xFFFFFFFF sentinel cannot become an out-of-range gather index; uint32 ROW_MAJOR rows for ttnn.embedding |
topk.padded_k(k), K_MULTIPLE, MASK_VALUE (-1e30), SENTINEL |
k rounded up to a multiple of 16 (500 -> 512); mask scores with a finite value, never -inf (P8) |
topk.topk_numpy(scores, k), class_major_numpy, decode_class_major_numpy, selection_metrics(dev_idx, ref_idx, *, reference_scores=None, flat_scores=None, strong=0.1) |
ONNX-semantics host top-k (ties to the lower index), layouts, and the set-based metrics of a device selection (shared, strong_total / strong_kept / strong_overlap / strong_missed) |
topk.local_max_margins(indices, heat, height, width, *, kernel=3) (0.21.0) |
per class-major proposal, the relative margin (heat[p] - max same-class neighbour) / heat[p] in the reference heat BEFORE the local max (1.0 without neighbours): the decisive / near-tie split of a strong-proposal gate (TransFusion gates margins > 0.5 %, reports the rest with near_tie_report) |
topk.topk_with_values(scores_tile, k) |
the bf16 ttnn.topk composite (values + indices, P8) |
transfusion_head.QueryHeadWeights(height, width, num_classes, num_proposals, class_table, bev_pos, self_posembed, cross_posembed, self_attn, cross_attn, norms, ffn, heads) |
model-agnostic host record (MHAWeights, PosEmbedMLP, PredictionHead; [in, out] layouts): .validate(), .query_pos_table() / .key_pos_table() (the position MLPs on bev_pos, float64), .merged_heads() |
transfusion_head.TransFusionQueryHead(weights, *, policy=None, prefix="head", attn_fp32_acc=False) |
__call__(lidar_feat, heat_nms, indices=None, taps=None) -> {"heads", "query_heat", "indices"}; pieces init_queries, keys, decoder, predict, query_heat; unpack_heads(raw) -> {name: (c, K)}; prepare(device), release(), describe() |
transfusion_head.QueryHeadOracle(weights) |
the literal head in torch fp32 (posembed MLPs on bev_pos[pos], unpadded attention, six heads) from NCHW maps and given indices: the device tests' reference |
transfusion_head.assemble_transfusion(heads, query_heat, indices, bev_pos, *, num_classes, num_proposals, cells) |
TF's (mmdeploy-patched) outputs on the host in float32: cls_score0 = query_heat * sigmoid(heatmap), bbox_pred0 = [center + query_pos, height, dim, vel], dir_cls_pred0 = rot |
Device form of the head (every rewrite exact in real arithmetic; the oracle runs the literal forms): queries
lidar_feat[pos] + class_table[cls] (bf16 ttnn.embedding row gathers, uint32 indices from the exact decode); the
query position embedding is a function of pos only, so it is a constant QPE table self_posembed(bev_pos)
gathered by pos (query_pos itself, k + 0.5 up to 191.5, never passes through bf16: S:transfusion:241); keys
lidar_feat + KPE with the constant KPE = cross_posembed(bev_pos); head dim 16 -> 32 in the projection weights
(C20 pad_head_columns / pad_head_rows, exact), scale from the weights, C20 sdpa (non-causal, chunk table);
self-attention on the logical K queries with no mask (Q / K / V from one fused projection); LayerNorm with the
model's epsilon (ttnn.layer_norm(o, residual_input_tensor=x, epsilon=...)); merged heads (relu(x @ [Wh_i]) @ blockdiag(Wo_i), fp32 out). One decoder layer only (a second layer would re-embed the predicted centres: no constant
table). Precision HiFi4+fp32:w=fp32 (default_policy(); per module <prefix>.self_attn / .cross_attn / .ffn /
.heads / .norm).
Device results (tests/device/test_transfusion_head_device.py, 13 cases through TraceRunner: eager x2, strict
capture, replay on a new input set, replay == eager bit for bit; watcher bring-up of the small shapes clean;
logs/transfusion/m1/j2_ttaw_{small_watcher,full}.log, results logs/ttaw/c21_c22_c26_device_results.json, gates
frozen in test_transfusion_head_device.gates.json): the masked local max is bit-exact against each model's
literal form at TF 192x192x5, BF 180x180x5 and PT 256x256x7 (plateaus and bright borders: ties everywhere); the
top-512 of TF 184,320 / BF 162,000 / PT 458,752 values has exactly the host's value multiset, unique in-range
indices in descending order, and an exact decode (also exhaustively over all 184,320 TF indices); surviving -inf
scores give 0xFFFFFFFF (22 of 32 in the canary) and the decode clamps them in range; the query head with random
weights, teacher-forced proposals, vs the literal fp32 oracle: TF (36,864 keys) PCC >= 0.9970 (queries 0.999998,
decoder layers >= 0.9996, heads >= 0.9970), BF (32,400 keys, the (i, j) bev_pos) >= 0.9922, query heat exact.
The TransFusion bundle runs the same modules with the deployed weights (bundles/transfusion-p150/PORT_LOG.md).
The selection chain at TF size (sigmoid, local max, class-major row, top-512, decode, one row gather) replays in
about 1.1 ms (logs/transfusion/m1/exp1_full_*.log: 2.0-2.4 ms for two chains). Host tests:
tests/host/test_transfusion_head_host.py (29) with tests/host/fake_ttnn_query.py (topk_large_indices with
HIGHEST-index ties, bf16-only max_pool2d, layer_norm with ttnn's 1e-12 default epsilon, eq / ge /
subtract / addcmul, an embedding that also records computed indices inside a capture).
22. Pillar feature net and input staging (C24 companion, ttaw.models.pillars)
The PillarFeatureNet of the pillar detectors (CenterPoint, PointPainting, TransFusion) on the device and the host
staging of its two inputs. Promoted by the PointPainting port (ttaw 0.19.0) from the CenterPoint bundle's
tt/pfn.py + FrameStaging (device-verified there: PFN PCC 0.99997) with the TransFusion bundle's generalisation
(any hidden / output width; pillars of K < 32 points). ttaw 0.21.0 adds TransFusion's two-term PFN (terms=2) and
the TransFusion bundle runs on this module from then on; the CenterPoint bundle keeps its own copy until it re-vendors
and switches (its PORT_LOG notes it).
from .ttaw.models.pillars import FrameStaging, PfnPlan, TtPillarFeatureNet, features_shape, pack_inputs
from .ttaw.ops.gather import scatter_rows
plan = PfnPlan.from_weights(w0, b0, w1, b1) # BN-folded [in, out] matrices: (F, H), (2H, O)
pfn = TtPillarFeatureNet(plan, num_pillars=40000, precision="HiFi4+fp32:w=fp32:a=fp32") # a= : hidden h / z dtype
runner.add_input("features", shape=features_shape(40000), dtype="bfloat16") # ROW_MAJOR, 1 KB / pillar
runner.add_input("canvas_index", init=np.full((1, H * W), 40000, np.uint32), dtype="uint32")
def forward(ctx): # a TraceRunner variant
rows = pfn(ctx["features"]) # [1, 1, P, O] bf16 ROW_MAJOR
canvas, table = scatter_rows(rows, ctx["canvas_index"], sentinel=40000) # C19: NHWC [1, 1, H*W, O] TILE
...
staging = FrameStaging(40000, H * W) # persistent host tensors, written in place per frame
out = runner("default", inputs=staging.write(features, canvas_index, num_pillars))
| name | meaning |
|---|---|
PfnPlan.from_weights(w0, b0, w1, b1, *, in_padded=16) |
the device matrices of the identity-folded split PFN: W0 (16, H_pad), Wz = [I_H | W1a | 0] (H_pad, Z), Wy = [W1b ; I_O ; 0] (Z, O) (H_pad = H rounded up to 32, Z = H + O rounded up to 32); F <= 16, O % 32 == 0 (the C19 gather table). .forward_numpy(features, *, hidden_round=None): its float32 host emulation (hidden_round models the hidden dtype) |
pfn_reference_numpy(features, w0, b0, w1, b1) |
the literal encoder in float64 (concat form; max over the given slots, padded slots included): the oracle |
TtPillarFeatureNet(plan, *, num_pillars, precision="HiFi4+fp32:w=fp32", name="pfn") |
[1, 1, P, 32*16] bf16 ROW_MAJOR -> [1, 1, P, O] bf16 ROW_MAJOR rows: reshape to [1, 1, P*32, 16], tilize, linear W0 + ReLU, linear Wz, max over the 32 slots (dim -2), linear Wy + b1 + ReLU, untilize. HiFi4 required (the identity blocks); Precision.activations = the dtype of the hidden h / z (the output rows are always bf16); .release(), .describe() |
TtPillarFeatureNet(..., terms=2, bias_add=True) (0.21.0), TWO_TERM_PRECISION = "HiFi4+fp32:w=bf16" |
the two-term PFN (TransFusion's default): every linear a TwoTermLinear (weights / biases as bf16 hi + lo, products hi@W_hi + hi@W_lo + lo@W_hi with fp32 outputs), fp32 h / z / slot max split by split_terms before the next linear, bf16 output rows; bias_add=True: bias-free RowMatmul (ttnn.matmul, fp32 out) + one exact fp32 ttnn.add per bias (ttnn.linear's fused bias rounds an fp32 output to ~11 bits on this tree); needs w=bf16 |
split_terms(t), RowMatmul(weight, *, precision, name), TwoTermLinear(weight, bias=None, *, activation=None, precision=TWO_TERM_PRECISION, name, bias_add=True) |
the two-term building blocks (split_terms: fp32 -> bf16 hi, lo = bf16(t - hi); also used by TransFusion's two-term SECOND convs) |
features_shape(P) -> (1, 1, P, 512); pack_features(features (P_kept, K, F), P, *, out=None) |
the device input: one 1 KB row per pillar (32 slots x 16 features, zero-padded), rows past P_kept zero; K < 32 slots: slots K..31 repeat slot 0 (TransFusion's 20-point pillars: the slot maxima are unchanged, zero rows would add relu(b0)) |
pack_inputs(features, canvas_index, num_pillars, *, capacity) |
{"features", "canvas_index"} as numpy (tests and tools); the index range-checked (C19 check_index, sentinel = capacity) |
FrameStaging(num_pillars, num_cells) |
persistent bf16 features / uint32 canvas_index host tensors (tensors.HostStaging); .write(features, canvas_index, num_pillars=None) converts only the kept pillars (torch fp32 -> bf16 RNE, K < 32 slot rule), zeroes the rows the previous frame used beyond them, copies the checked index and returns the TraceRunner inputs; .zero_copy (False: falls back to pack_features + HostStaging.write, same values) |
Device form (exact rewrites, also in floating point): one pillar = one 32-row tile, so a slot max is a tile-column
reduction; relu(. + c) is monotonic, so the per-pillar term c = m @ W1b + b1 is added after the max (no slot
broadcast); h and the max ride through the matmuls in identity blocks (x * 1.0 is exact with HiFi4). Every pillar
row is computed, also past the frame's pillar count: the scatter index never points there, so no count reaches the
device. Precision: the bf16 rounding of the hidden h / z is the PFN's largest error term (PointPainting's CPU
emulation on its golden frames: output rel. L2 0.0045 with bf16 hidden tensors, 0.0020 with fp32 hidden tensors and
TF32 weights, 0.0017 for the bf16 output rounding alone), hence the a=fp32 option; a model with large-magnitude input
columns (absolute x / y up to 121.6 m) can rewrite its first layer around the pillar centre on the host (PointPainting
tt/pfn_input.py, a promotion candidate) and feed this module unchanged.
Device results (tests/device/test_pillars_device.py, TraceRunner: eager x2, strict capture, replay on input set B,
replay == eager bit for bit; the 512-pillar cases first under TT_METAL_WATCHER=2, clean; random weights, bf16
inputs, vs the literal float64 encoder on the same inputs; logs/pointpainting/m1/j1_pillars_*.log, results
logs/ttaw/pillars_device_results.json): CenterPoint widths (9 -> 16, 32 -> 32) PCC 0.999996 / rel. L2 0.0025;
PointPainting (16 -> 16, 32 -> 32) bf16 hidden 0.999996 / 0.0025, fp32 hidden 0.9999975 / 0.0024; TransFusion (11 ->
32, 64 -> 64, 20-point pillars) bf16 hidden 0.999995 / 0.0027, fp32 hidden 0.999997 / 0.0025; the real capacity
(40,000 pillars, 28,137 kept, fp32 hidden) 0.9999975; FrameStaging's zero-copy path equals pack_features + bf16
RNE for a large frame, a smaller one (its stale rows zeroed) and 20-point pillars. The PointPainting bundle runs it
with its real weights (PFN PCC 0.99999 on two golden frames, bundles/pointpainting-p150/PORT_LOG.md). Host tests:
tests/host/test_pillars_host.py (12; fake ttnn: the plan exact against the literal encoder for the CP / PP / TF
widths and the 20-slot rule, packing, the module through TraceRunner replay == eager with bf16 and fp32 hidden
tensors, FrameStaging's fallback path).
23. Segment reductions: K1 segment_reduce and the log-step fallback (ttaw.ops.segment)
Built by the FRNet port (PLAN.md 1.3 row K1; owned by it) for FRNet's six ScatterElements(max) sites (points
sorted by frustum cell); PTv3 pooling and BEVPool (sum, optimization phase) are of the same form once the host sorts
the rows. A segmented tensor holds rows sorted by segment id: segment s is the contiguous row range
[off[s], off[s + 1]) (CSR offsets, non-decreasing); rows past off[-1] are padding that nothing reads.
from .ttaw.ops import segment as seg
off = seg.segment_offsets(seg_id_sorted, num_segments) # host CSR offsets, uint32 [S + 1]
row = seg.check_offsets(off, num_segments=S, num_rows=N_cap, length=S + 1) # [1, K] for the device
k1 = seg.SegmentReduce(seg.SegmentReduceSpec(num_rows=N_cap, channels=128, num_segments=M_cap + 1,
input_dtype="float32", input_layout="tile",
output_dtype="bfloat16"))
runner.add_input("offsets", init=row, dtype="uint32") # RT-dev: rewritten per frame
def forward(ctx): # a TraceRunner variant
table = k1(point_rows_fp32_tile, ctx["offsets"]) # [1, 1, M_cap + 1, 128] bf16 ROW_MAJOR; empty rows -> 0
return gather_rows(ctx["pix2vox"], table, sentinel=M_cap) # C19: straight into a gather table
| name | meaning |
|---|---|
SegmentReduceSpec(num_rows, channels, num_segments, input_dtype="float32", input_layout="tile", output_dtype="float32", op="max", empty_value=0.0, workers_per_core=2) |
static shape of one call site (COMPILE). num_rows % 32 == 0, channels % 32 == 0; bf16 / fp32 in, TILE or ROW_MAJOR; bf16 (round to nearest even) / fp32 out; op "max" only on the device (sum / mean raise NotImplementedError: the data-movement RISC-Vs have no float unit, see below); .l1_bytes_per_worker(), .check_l1(budget) |
SegmentReduce(spec, *, name, grid=None) |
__call__(x, offsets, *, output=None) -> [1, 1, num_segments, C] ROW_MAJOR DRAM (allocated per call, or the persistent output of that spec): ONE ttnn.generic_op, eager or inside a capture. x [1, 1, num_rows, C] DRAM interleaved of the spec's dtype / layout; offsets uint32 ROW_MAJOR [1, K >= num_segments + 1] DRAM. program(x, offsets, out) (the ProgramDescriptor), allocate_output(device), output_spec(), describe(). grid overrides the device grid (tests: one core) |
segment_reduce(x, offsets, *, num_segments, op="max", output_dtype=None, empty_value=0.0, output=None) |
one-shot form (spec from the tensors) |
logstep_segment_max(x, shift_tables, last_rows, *, layout="tile") |
the exact stock-op fallback (PLAN.md 1.3): R rounds x = max(x, x[idx_r]) (untilize + ttnn.embedding + ttnn.maximum: 3 programs each), then a PADDED gather of last_rows from the result with a zero row appended at R (empty segments -> 0). bf16 in / out only (ttnn.embedding tables are bf16); exact for segments of up to 2^R rows; R is COMPILE (bucket it per frame) |
segment_offsets(seg_id, num_segments, *, num_rows=None), check_offsets(offsets, *, num_segments, num_rows, length=None) |
host CSR offsets from sorted ids (ids >= num_segments = trailing padding); validation (starts at 0, non-decreasing, ends <= num_rows) into the device [1, length] row |
segment_rounds(max_len), segment_shift_tables(seg_id, rounds), segment_last_rows(offsets, *, empty_row) |
the fallback's tables: ceil(log2 L); idx_r[i] = i - 2^r in the same segment, else i; each segment's last row (empty_row = the row count: the appended zero row) |
segment_reduce_numpy(x, offsets, num_segments=None, *, op="max"|"min"|"sum"|"mean", empty_value=0.0), logstep_segment_max_numpy(x, shift_tables, last_rows) |
oracles (max / min exact in the input dtype; sum / mean in float64) |
worker_segments(offsets, num_segments, num_workers, *, num_rows=None) |
the kernel's work split, host twin (tests, diagnostics) |
OPS, DEVICE_OPS, KERNEL |
("max", "min", "sum", "mean"), ("max",), "segment_reduce_dm.cpp" |
Kernel (ops/kernels/segment_reduce_dm.cpp, one source for both data-movement RISC-Vs of every core). Exact by
construction: the maximum is taken on integer keys of the float bit patterns (key = bits ^ ((bits >> 31) & 0x7fffffff): signed order == float order, -0 < +0; NaN unsupported), so bf16 / fp32 results are bit-exact, and an
fp32 input with a bf16 output equals rounding first (RNE is monotonic). Work split without per-core runtime args
(probe P3 rule; common args = the three buffer addresses): worker w (= core index x 2 + RISC-V) owns the segments
whose cost off[s] + s (rows + one output row) lies in [w T / W, (w + 1) T / W), found on the device by two binary
searches over the offsets (64-byte DRAM probes), so nothing in the host tables depends on the grid and ETH (12x10)
and WORKER (11x10) give identical outputs. Each worker streams its contiguous rows in units of 32 (one tile-row in
TILE mode; double-buffered NOC reads), reduces into an L1 accumulator and writes one output row per segment through
a ring of 8 staging rows; empty segments get empty_value (FRNet's frustum2pixel zero row comes out of the kernel).
Offsets are clamped to [0, num_rows] and to non-decreasing order, so a corrupt table cannot make it read outside
x. L1 per worker: 2 units + 1 KiB offsets window + 128 B probe + C x 4 B accumulator + 8 output rows (FRNet's
widest site, 256 fp32 channels: 75 KiB). Hang protocol (PLAN.md 4.4): no CB push / pop (CBs are plain scratch), no
multicast, no semaphores; DRAM reads keep the source's 64-byte alignment offset in L1 (Blackhole
NOC_DRAM_READ_ALIGNMENT_BYTES), every read is waited on (the barrier also invalidates the Blackhole L1 cache)
before use, staging rows are reused after noc_async_writes_flushed, and the kernel ends with both barriers.
Why max only: the data-movement RISC-Vs (rv32im + Zba / Zbb, no F extension) compare integers natively (Zbb max)
but would emulate float adds in software; a sum / mean K1 needs the compute engine (the optimization-phase
consumers, PTv3 / BEVPool, add it with their device tests).
Device results (tests/device/test_segment_reduce_device.py, the FRNet port's jobs; results
logs/ttaw/k1_segment_reduce_device_results.json): tiny bring-up under TT_METAL_WATCHER=2 clean (64 x 32, 6
segments incl. empty / 1-row / tile-row-crossing ones; one core with 1 and 2 workers; eager x2 and a 2-call trace
replayed on new data, bit-exact); FRNet's real sites through TraceRunner with the offsets rewritten per frame (10
replays x 5 rewrites), every output bit-exact vs the oracle and replay == eager, 240 workers (12x10 ETH grid, 2 per
core): OT128 encoder site 160,000 x 256 fp32 -> 60,000 fp32 rows 5.52 ms per replay, OT128 point sites 160,000 x 128
fp32 -> 60,001 bf16 rows 2.89 ms, QT128-like dense segments (6,255 segments, one of 500 rows) 2.22 ms, 2,048-row cases
0.04-0.08 ms; the log-step fallback equals K1 bit for bit (4 rounds on 4,096 rows; 7 rounds on 120,000 rows with
111-row segments) (logs/frnet/m1a/j1_bringup_and_graph.log, watcher log logs/frnet/m1a/j1_tiny_watcher.log,
logs/ttaw/k1_segment_reduce_device_results.json). Inside the FRNet graph (six sites per frame, OT128 sample): every
tap downstream PCC >= 0.9995 vs the fp32 reference (bundles/frnet-p150/PORT_LOG.md). A 2,000-launch soak
(test_soak) runs in the port's next job. Host tests:
tests/host/test_segment_host.py with tests/host/fake_ttnn_segment.py (program-descriptor records, a
generic_op that emulates the kernel from its compile-time args and checks the P3 rules and the split, maximum,
hardswish, softmax, addcmul).
24. Sparse-conv rulebooks (C13, ttaw.sparse)
Built by the BEVFusion port (PLAN.md C13; owned by it) for BEVFusion's 21-layer sparse encoder and PTv3's
sub-manifold stages: the host side of the gather-GEMM sparse conv (C28). A sparse tensor is an active voxel set
coords (N, 3) = (x, y, z) in spatial_shape (X, Y, Z); every spconv layer of the Autoware graphs
(GetIndicePairsImplicitGemm + ImplicitGemm, spconv 2.3.8) is a gather, for sub-manifold and strided layers
alike: out[o] = sum_k W[:, k, :] @ in[nmap[o, k]] (taps with -1 contribute nothing), W the KRSC filter
[Cout, k0, k1, k2, Cin] (k0 along x), tap k = (a * k1 + b) * k2 + c. numpy only.
from .ttaw import sparse as sp
l1 = sp.SparseLevel(coors_zyx[:, ::-1], (1440, 1440, 41)) # (x, y, z) rows = the feature rows
nm1 = sp.subm_neighbor_map(l1, 3) # (N1, 27) int32, spconv's 13-query rule
l2, d1 = sp.strided_neighbor_map(l1, 3, 2, 1) # next level + (N2, 27) gather map
bucket, overflow = sp.select_capacity(l2.num_active, (40_960, 147_456, 262_144))
idx = sp.im2col_index(nm1, sentinel=cap, rows=cap) # (cap / 32, 27 * 32) uint32, tile-ordered
# device (C28): ttnn.embedding(idx, table [1, 1, cap + 1, Cin_pad], layout=TILE, padding_idx=cap, PADDED)
# -> experimental.view [1, 1, cap, 27 * Cin_pad] -> matmul [27 * Cin_pad, Cout] (BN folded) + ReLU
bev = sp.dense_gather_index(l_out, sentinel=n_out) # (Z, X * Y): one canvas gather per z slice
| name | meaning |
|---|---|
SparseLevel(coords, spatial_shape) |
an active set: exact int64 keys over the bounding box of grid + voxels (sorted keys + searchsorted, no aliasing, out-of-grid voxels allowed), duplicates refused; .lookup(q) -> rows (-1 = none), .in_grid(q=None), .linear_index() ((x * Y + y) * Z + z), .num_active, .describe() |
subm_neighbor_map(level, kernel=3, dilation=1, *, rule="spconv"|"exact") |
(N, K) int32. spconv uses padding = (k // 2) * dilation for sub-manifold layers. "spconv": the 13-query + symmetric-write construction (indices.py:1532,865,838-845): a tap k < K // 2 needs the neighbour inside the grid, a tap k > K // 2 the output voxel inside it (S:ptv3:158); the centre tap is the voxel itself. "exact": the plain lookup. Equal for in-grid voxel sets (every BEVFusion level); "alias" (PTv3's linear-key aliasing, S:ptv3:388) raises NotImplementedError until PTv3 adds it |
strided_neighbor_map(level, kernel, stride, padding, dilation=1) -> (out_level, nmap) |
regular sparse conv: o active iff some active i = o * s - p + k * d (0 <= o < out_shape); outputs in ascending linear-key order; one input per (output, tap); inputs must lie inside their grid |
conv_out_shape(shape, kernel, stride, padding, dilation=1), kernel_offsets(kernel, dilation=1), iter_taps(kernel) |
grid arithmetic (in + 2p - d(k - 1) - 1) // s + 1; tap offsets in tap order |
gather_index(nmap, *, sentinel, rows=None) |
(rows, K) uint32: -1 and capacity rows -> sentinel (an explicit zero row of the table: C19's PADDED rule) |
im2col_index(nmap, *, sentinel, rows=None, block=32) / im2col_numpy(table, index, *, kernel_volume) |
the tile-ordered index (rows / 32, K * 32), index[b, k * 32 + v] = gather[b * 32 + v, k]: a TILE ttnn.embedding of it into a Cin % 32 == 0 table yields [rows / 32, K * 32, Cin] whose tiles, in memory order, are those of the [rows, K * Cin] im2col matrix (row o = its K tap rows concatenated, tap-major), so ttnn.experimental.view gives the matmul input (probe P4: 4.7 ms for 256k x 27 x 32); the oracle builds that matrix |
dense_gather_index(level, *, sentinel, split_axis=2), dense_numpy(features, level) |
to-dense as gathers (index[s, a * B + b], one canvas per slice of split_axis) and the ScatterND reference [X, Y, Z, C] |
select_capacity(count, buckets) -> (bucket, overflow) |
smallest bucket holding count (grid-independent constants, PLAN.md 0.2); the largest and True when none does |
sparse_conv_numpy(features, nmap, weight, bias=None), dense_conv3d_numpy(features, level, weight, out_coords, *, stride, padding, dilation) |
oracles: the gather form (float64 accumulation) and a brute-force dense 3-D cross-correlation (Conv3d semantics) at the active outputs |
pairs_count(nmap), neighbor_stats(nmap) |
rulebook size (FLOPs = 2 Cin Cout pairs), {rows, pairs, mean_taps, max_taps} |
Facts (BEVFusion port): on the sample-rosbag frame #144 (98,334 voxels) every level's active count and every layer's
pair count equal the research reference's (research/bevfusion/scripts/bevfusion_ref.py, an independent
index_add_ scatter form): 679,726 / 325,630 / 1,383,475 / 334,714 / 736,959 / 147,183 / 253,269 / 20,710 pairs for
subm1 / down1 / subm2 / down2 / subm3 / down3 / subm4 / conv_out; all four sub-manifold and four strided maps take
about 2.2 s in numpy on the shared host (a C++ / device builder is optimization work). Host tests:
tests/host/test_sparse_host.py (15): sub-manifold and strided maps equal a dense Conv3d at BEVFusion's layer
shapes, strided output sets equal the brute-force set, "spconv" equals a literal transcription of spconv's
construction also with out-of-grid voxels (where it differs from "exact"), the im2col tile order equals the im2col
matrix's for Cin 32 / 64 / 128, BEVFusion's level chain 1440x1440x41 -> 720x720x21 -> 360x360x11 -> 180x180x5 ->
180x180x2. C28 (the device gather-GEMM) is added with its device tests by the BEVFusion port's M1.
25. Gather-GEMM sparse encoder (C28, ttaw.models.sparse_encoder)
Built by the BEVFusion port (PLAN.md C28; owned by it; probe P4) on the C13 rulebooks of section 24: every spconv layer as ONE gather + ONE matmul on the device, any chain of sub-manifold / strided layers with basic-block residuals, traceable, no count on the device. PTv3's sub-manifold stages are the second consumer.
from .ttaw import sparse as sp
from .ttaw.models.sparse_encoder import SparseConvSpec, SparseEncoder, dense_gather, pack_table
specs = [SparseConvSpec("conv_input", w0, b0, "subm1", in_terms=2), # KRSC [Cout, k0, k1, k2, Cin], BN folded
SparseConvSpec("l1.conv1", w1, b1, "subm1"),
SparseConvSpec("l1.conv2", w2, b2, "subm1", residual="conv_input"), # relu(conv + b + identity)
SparseConvSpec("down1", w3, b3, "down1"), ...]
enc = SparseEncoder(specs, policy=policy, weight_terms=1) # fp32 weights (TF32-like, P12)
runner.add_input("vfe", pack_table(vfe, cap1, terms=2), dtype="bfloat16", layout=ttnn.ROW_MAJOR_LAYOUT)
runner.add_input("subm1", sp.im2col_index(nm1, sentinel=cap1, rows=cap1), dtype="uint32") # per index name
def forward(ctx):
y, table = enc(ctx["vfe"], {n: ctx[n] for n in enc.index_names}) # last fp32 output + its bf16 table
return dense_gather(table, ctx["dense0"]) # to-dense slice [1, 1, cells, C]
| name | meaning |
|---|---|
SparseConvSpec(name, weight, bias, index, residual=None, relu=True, in_terms=1, out_terms=1) |
one layer: weight KRSC (or [Cout, K, Cin]), index = the gather index name it reads (shared by the convs of a level), residual = an earlier layer's name; .cin_pad / .cout_pad (terms * C rounded up to 32), .matrix() (the [K * cin_pad, cout_pad] im2col weight), .bias_row(), .lo_mask() |
im2col_weight(weight3, *, cin_pad, cout_pad, in_terms, out_terms) |
host matrix: tap-major rows k * cin_pad + j; the lo channel block repeats the weights, out_terms=2 repeats the columns |
TtSparseConv(spec, *, precision="HiFi4+fp32:w=fp32", weight_terms=1) |
__call__(table, index, identity=None) -> fp32 TILE [1, 1, rows, cout_pad]: TILE + PADDED ttnn.embedding of the tile-ordered [rows / 32, K * 32] index into the bf16 ROW_MAJOR [1, 1, V + 1, cin_pad] table (row V zero: the sentinel), ttnn.experimental.view to [1, 1, rows, K * cin_pad], C24 RowMatmul (fp32 out; weight_terms=2: bf16 hi + lo weights, two matmuls), one exact fp32 bias add (+ the identity, ReLU fused into the last add); gather(table, index); to_table(y) -> the next bf16 table (zero row appended; out_terms=2: [bf16(y) | bf16(y - bf16(y))]) |
SparseEncoder(specs, *, policy=None, weight_terms=1, name="sparse") |
the chain (consecutive layers must agree on terms / widths, residual sources on the output layout); __call__(table, indices, taps=None, keep_table=True) -> (y, table); index_names, input_width, input_terms; intermediate outputs freed after their last (residual) use unless tapped |
dense_gather(table, index) |
to-dense slice: C19 PADDED gather of a [1, cells] C13 dense_gather_index row -> [1, 1, cells, C] TILE |
pack_table(x, capacity, *, terms=1, width=None, out=None), table_rows_numpy(x, *, terms, width) |
the first table on the host ((1, 1, capacity + 1, width) float32 with bf16-exact values; rows past N and the sentinel zero) |
sparse_layer_numpy(table, nmap, spec, *, identity=None), sparse_encoder_numpy(specs, features, maps, *, round_tables=True) |
oracles (float64 matmul); round_tables models the device's bf16 / two-term tables, False is the fp32 network |
Activation terms. ttnn.embedding is bf16-only, so every layer's input is rounded to bf16 once. A two-term
table (in_terms=2) carries hi and lo = bf16(x - hi) side by side in the channels (one gather, one matmul over
[hi | lo] @ [W ; W]); where 2 C <= 32 (a VFE input of 4 columns, BEVFusion's 16-channel level 1) the lo block
rides in channels the 32-wide table pads anyway. On BEVFusion's PandaSet frame the CPU emulation of the 21 layers
gives conv_out PCC 0.999995 with bf16 tables (fp32 weights), 0.999999 with two-term tables: the encoder is not the
precision bottleneck (BEVFusion PORT_LOG.md "Sparse precision").
Device results (tests/device/test_sparse_encoder_device.py, TraceRunner: eager x2, strict capture, replay on a
second frame, replay == eager bit for bit; the small cases first under TT_METAL_WATCHER=2, clean;
logs/bevfusion/m1/j1_c28_small_watcher.log, j2_c28_bf.log, results logs/ttaw/c28_sparse_encoder_device_results.json):
a 6-layer BEVFusion-shaped chain (220 voxels; two-term VFE, one- / two-term tables, fp32 and two-term weights) PCC
= 0.99999 vs the oracle with the same table rounding; the 21-layer BEVFusion encoder with random weights (1440 x 1440 x 41 -> 180 x 180 x 2; 28.8 k / 37.5 k / 25.9 k / 14.0 k / 13.8 k active voxels in capacities 32,768 / 40,960 / 32,768 / 16,384 / 16,384) every layer PCC >= 0.99998, to-dense gathers bit-exact, 27.3 ms per replay (21 gathers + matmuls + table conversions + 2 to-dense gathers; first port, not optimized). Host tests:
tests/host/test_sparse_encoder_host.py(15;tests/host/fake_ttnn_sparse.py: a batch-formembeddingand anexperimental.viewthat reinterprets TILE memory tile by tile, so the tile-ordered index order is checked).