rf-detr-p150

Roboflow RF-DETR-base port on one Tenstorrent Blackhole p150a. Weights: Roboflow/rf-detr-base · Paper: arXiv:2511.09554 · Upstream code: roboflow/rf-detr · Port: changh95/tt-RF-DETR

Runs on p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart (Python)

Prerequisite: a tt-metal / ttnn environment at tt-metal 8b98410e730 with patches/tt-metal-eth-dispatch.patch applied. ttnn is not on PyPI.

hf download changh95/rf-detr-p150 --exclude "image/*" --local-dir rf-detr-p150 && cd rf-detr-p150
pip install -e code/        # adds torch, numpy<2, pillow, safetensors, huggingface_hub
pip install -e "code/[server,test]"   # also the HTTP server and the tests (pytest)

Run the snippet from the model repo root. media/demo_source.png is a path relative to the repo root.

from rf_detr import RFDETRBase

with RFDETRBase.from_pretrained(device_id=0) as model:
    det = model("media/demo_source.png", threshold=0.5)

for label, score, box in zip(det.labels, det.confidence, det.xyxy):
    print(f"{label:8s} {score:.3f}", box.round(1))   # box = [x1, y1, x2, y2] in image pixels
det.draw("media/demo_source.png").save("detections.png")

• from_pretrained downloads the Roboflow/rf-detr-base weights to your HF cache, opens the chip and captures the metal traces.
• The first start compiles the kernels (about 1 minute).
• RFDETRBase.from_pretrained() captures the traces and warms up the common inputs (about 4 s). Your first call is then as fast as the later calls (about 3 ms for a 640x480 image). For a large host-resized size, call model.warmup(size=(W, H)) once.
• The with block closes the chip at the end. Without with, call model.close().

Input (images)	A file path, PNG / JPEG bytes, a PIL.Image, a uint8 RGB numpy array (HxWx3) or torch tensor (3xHxW), or a float RGB array in [0, 1]. Any size up to 8192 per side.
Options	threshold=0.5 (score threshold), max_detections=100 (1..300). from_pretrained(device_id=0, dispatch="eth", weights_dir=None, device=None, device_preprocess_sizes=[(640, 480)]).
Output (Detections)	xyxy float32 [N, 4] in image pixels, confidence [N], class_id [N] (COCO-91), labels (class names), logits / pred_boxes (raw, all 300 queries). Sorted by score.
Methods	det.to_dicts() gives the /predict JSON list. det.to_supervision() gives a supervision.Detections. det.draw(image) gives a PIL.Image.

• Throughput: give a list of images to model([...]). You get a list of Detections. The host decodes the next image while the device runs the current image.
• Threads can share one model. A lock serializes the calls. One model uses one chip.
• 640×480 inputs use the resize and normalize on the device. Other sizes use the host path.
• The API gives the same detections as the HTTP server /predict. The two paths share the decode, the preprocessing, the device trace and the host postprocess.
• Full reference (all options, input forms, fields, helpers such as rf_detr.open_device): code/PYTHON.md. Runnable example: examples/quickstart.py.

Serving (HTTP)

tt-model pull  changh95/rf-detr-p150 --with-weights
tt-model serve changh95/rf-detr-p150       # or with tt-cli: tt serve changh95/rf-detr-p150
printf '{"image":"%s"}' "$(base64 -w0 media/demo_source.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/rf-detr-p150

• The image does not contain the weights. --with-weights puts them in your HF cache.
• The server uses port 20000 (or the next free port). It is ready when the log shows Application startup complete.
• POST /predict: image (base64 PNG/JPEG); optional threshold (0.5), max_detections (100). Also GET /health, GET /info.

{"detections": [
   {"label": "cat",    "label_id": 17, "score": 0.960, "box": [7.5, 54.4, 317.5, 470.6]},
   {"label": "remote", "label_id": 75, "score": 0.903, "box": [40.9, 72.7, 176.6, 117.7]}],
 "num_detections": 5, "image_size": {"width": 640, "height": 480}, "input_size": [560, 560],
 "timing_ms": {"decode": 2.2, "preprocess": 0.01, "inference": 2.14, "postprocess": 0.16, "total": 4.5}}

• box is [x1, y1, x2, y2] in original image pixels. Detections are sorted by score.

Demo

Input (media/demo_source.png)	Detections on p150a (media/demo_detections.png)

Demo & Performances

Warm, batch 1, a 640×480 cats image squashed to 560×560 (media/demo_source.png for /predict and model(), code/data/cats_000000039769.jpg for bench_breakdown.py), 100 iterations per run.

Metric	Performance
Python model() call, model(numpy 640×480) (device preprocessing, host postprocess included)	2.74–3.03 ms median (up to 3.31 ms under host load; 2 sessions, fresh processes) · 2.71 ms min
Served /predict client round trip, loopback	about 6.3 ms median (run medians 6.04–6.77, 12 runs; previous build about 7.7 ms)
Served /predict timing_ms.total	4.25–4.56 ms (3 server boots; previous build 4.80 ms)
Served inference stage	2.13–2.14 ms
Model call model(pixel_values) (host im2col + H2D + trace + D2H)	2.68 ms median (median of 5 run medians, 2.67–3.02; the 3.02 ms run was on a cold host) · 2.64 ms min
Device trace, one blocking forward (DINOv2 backbone, projector, two-stage, decoder with MSDA)	1.90 ms median (1.897–1.909)
Back-to-back trace replays	1.870–1.872 ms per forward
Host im2col · H2D · D2H	0.22–0.38 ms (2026-10-03) · 0.65–0.67 ms · 0.11–0.16 ms

All numbers in this table were measured with dispatch on ETH cores, 1 command queue and a 12×10 compute grid on a Blackhole chip. This configuration is p150-equivalent and is the default of code/ (2026-10-05). An independent verifier measured the numbers again on a second chip, and they agree within host-load noise. In the same session, the previous worker-dispatch default gave the same device time (b2b 1.868–1.870 ms) and bit-identical outputs. Accuracy against the torch fp32 reference (cats test image): detection-IoU 98.814 (gate 98.5; previous release 98.670), served uint8 path 98.805, transformer fg-matched boxes / logits PCC 0.999982 / 0.999232. Details: VERIFICATION_2026-10-03.md.

The 2026-10-04 build changes only host code. The device numbers and the accuracy stay the same. The server parses the request with orjson. It also decodes the PNG while the request body arrives. Thus, timing_ms.total now counts only the work after the last body byte. The client round trip includes all steps. Other Python calls (ETH dispatch, 2026-10-05): model(path to PNG) 4.96–5.11 ms median, model(numpy 500×375) (host preprocessing) 3.47–3.84 ms median (min 3.40 ms).

RTX 5090 reference measurements (2026-09-14) are unchanged. They use the port's torch reference in PyTorch 2.11, batch 1, on the same demo image. "Served-like" adds the original PIL decode, preprocess and postprocess. Full table: GPU_COMPARISON.md.

RTX 5090 precision	GPU served-like total	vs current build (4.56 ms)	GPU incl. H2D/D2H vs our model call (2.68 ms)	GPU forward only vs our device trace (1.90 ms)
fp32 strict, eager	14.72 ms	Blackhole 3.23× faster	7.99 ms: Blackhole 2.98× faster	7.73 ms: Blackhole 4.07× faster
tf32, eager	11.58 ms	Blackhole 2.54× faster	5.26 ms: Blackhole 1.96× faster	5.00 ms: Blackhole 2.63× faster
bf16 autocast, eager	13.30 ms	Blackhole 2.92× faster	5.93 ms: Blackhole 2.21× faster	5.71 ms: Blackhole 3.00× faster
fp16 autocast, eager	–	–	5.91 ms: Blackhole 2.20× faster	5.66 ms: Blackhole 2.98× faster
tf32 + torch.compile (CUDA graphs)	–	–	2.74 ms: Blackhole 1.02× faster (within host noise)	2.39 ms: Blackhole 1.26× faster
fp16 autocast + torch.compile (CUDA graphs)	–	–	2.22 ms: GPU 1.21× faster	1.79 ms: GPU 1.06× faster

The served column uses 4.56 ms, the slowest verified timing_ms.total run. This value does not include the PNG decode that overlaps the body upload. The GPU served-like total includes its full decode. The Blackhole lead over eager PyTorch comes from one metal trace with fused custom kernels on all 120 compute cores. At batch 1 the eager GPU graph is launch-bound. With torch.compile and CUDA graphs, the GPU is 1.06× faster on the forward alone. The served lead also comes from the compiled host decode (about 2.45 ms on the buffered path), against about 6 ms of PIL decode and preprocess on the GPU host.

Caveats

• Does not scale to multiple p150a in a mesh configuration. The current build uses a 12x10 compute grid of Tensix cores. To get this grid, the dispatch functions move from 10 Tensix cores to ETH cores (patches/tt-metal-eth-dispatch.patch). Thus, this build assumes that you do not need chip-to-chip ethernet communication.
• dispatch="worker" (server: RF_DETR_DISPATCH=worker) is an opt-in for Galaxy only. On a p150 it gives an 11×10 compute grid. If ETH dispatch is not available (tt-metal without the patch), the model uses this mode and gives a warning. The numbers on this card do not apply to that mode.
• Every image is squashed to 560×560; one image per request, batch 1.
• bf16 on device: scores differ slightly from the fp32 reference (see the IoU figure above). Some optimizations change numerics (for example, the GELU polynomial, the reference-point refine and the MSDA corner weights). OPT_REPORT.md lists each change.
• 640×480 inputs run the resize and normalize on the device (uint8 path). Other sizes use the host path. The two paths differ only by rounding.
• The host helpers (code/rf_detr/tt/host/*.c) compile with cc at first use. Without a C compiler, the server uses PIL and torch. The results are the same, but the server is slower.
• The Python API keeps, per query, the best class above the threshold (the server rule). Upstream rfdetr uses a top-k over all 300×91 scores.
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404.
• p150a power was not measured, so no efficiency comparison is made.

Licensing

• Weights: Roboflow/rf-detr-base, Apache-2.0.
• Port and serving code (code/): Apache-2.0, from changh95/tt-RF-DETR.
• code/data/cats_000000039769.jpg is COCO image 39769. The accuracy tests use it. It keeps its original COCO terms.

Provenance

These are the exact sources the container image was built from. code/ has since been updated (2026-10-03 optimized build; 2026-10-04 faster HTTP layer and Python API; 2026-10-05 ETH dispatch, 1 command queue and 12×10 grid as the default; see OPT_REPORT.md) and is newer than the image. tt-model serve runs the image's code until the image is rebuilt. tt-model.yaml and SERVING.md still describe the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d
code/ digest (image)	906212f3590bf302 (sha256, first 16 hex digits; the current code/ differs)
built	2026-09-12T20:08:45+00:00 by tt-model 0.1.0