Card: earlier .pt2 re-scored in the same run (52.15 / 71.24)

dd5b63b verified 4 days ago

17.2 kB

---
license: other
license_name: embedl-models-community-licence-1.0
license_link: https://github.com/embedl/embedl-models/blob/main/LICENSE
base_model:
  - Roboflow/rf-detr-base
quantized_from:
  - Roboflow/rf-detr-base
tags:
  - object-detection
  - quantization
  - onnx
  - tensorrt
  - edge
  - embedl
gated: true
extra_gated_heading: "Access Embedl RF-DETR Base"
extra_gated_description: "To access this model, please review and accept the terms below. Your contact information is collected solely to manage access and, with your explicit consent, to notify you about updated or new optimized models from Embedl."
extra_gated_button_content: "Agree and request access"
extra_gated_prompt: "By requesting access you agree to the Embedl Models Community Licence and the upstream RF-DETR License"
extra_gated_fields:
  Company: text
  I agree to the Embedl Models Community Licence and upstream RF-DETR License: checkbox
  I consent to being contacted by Embedl about products and services (optional): checkbox
---
<!-- embedl-banner:start -->
<style>
.embedl-btn-primary { transition: background 160ms ease, box-shadow 160ms ease; }
.embedl-btn-primary:hover { background: #4FDCE4 !important; box-shadow: 0 8px 22px rgba(45,212,221,0.45) !important; }
.embedl-btn-secondary { transition: background 160ms ease; }
.embedl-btn-secondary:hover { background: rgba(45,212,221,0.15) !important; }
.embedl-headline { font-size: clamp(11px, 2.15vw, 15px) !important; }
.embedl-btn-primary, .embedl-btn-secondary {
  font-size: clamp(11px, 1.65vw, 13px) !important;
  padding: clamp(6px, 1.1vw, 9px) clamp(10px, 1.6vw, 14px) !important;
}
</style>
<div style="background:radial-gradient(600px 220px at 0% 50%,rgba(45,212,221,0.22) 0%,rgba(45,212,221,0) 60%),radial-gradient(400px 180px at 100% 100%,rgba(45,212,221,0.10) 0%,rgba(45,212,221,0) 55%),linear-gradient(135deg,#0B1626 0%,#142338 100%);border:1px solid rgba(45,212,221,0.28);border-radius:12px;padding:22px 24px;margin:0 0 24px 0;color:#F2F6FA;box-shadow:0 4px 16px rgba(11,22,38,0.18);overflow:hidden;box-sizing:border-box;max-width:100%;">
  <table style="width:100%;border-collapse:collapse;border:0;background:transparent;">
    <tr style="background:transparent;">
      <td style="vertical-align:middle;border:0;padding:0;background:transparent;">
        <div style="display:inline-block;font-size:10px;letter-spacing:0.08em;text-transform:uppercase;font-weight:700;color:#2DD4DD;background:rgba(45,212,221,0.15);border:1px solid rgba(45,212,221,0.35);padding:4px 10px;border-radius:999px;margin-bottom:10px;white-space:nowrap;">Optimized by Embedl</div>
        <div class="embedl-headline" style="font-size:15px;font-weight:700;line-height:1.35;color:#F2F6FA;margin-bottom:4px;">Need to <span style="color:#2DD4DD;white-space:nowrap;">fine-tune</span>, hit <span style="color:#2DD4DD;white-space:nowrap;">performance targets</span>, or deploy on <span style="color:#2DD4DD;white-space:nowrap;">specific hardware</span>?</div>
        <div style="font-size:13px;color:#9BA7B5;">We've got you covered.</div>
      </td>
      <td width="1%" style="vertical-align:middle;border:0;padding:0 0 0 18px;white-space:nowrap;text-align:right;background:transparent;">
        <a href="https://www.embedl.com/models" class="embedl-btn-secondary" style="display:inline-block;font-size:13px;font-weight:600;padding:9px 14px;border-radius:6px;border:1px solid #2DD4DD;color:#2DD4DD;text-decoration:none;margin-right:8px;">Learn more</a>
        <a href="https://www.embedl.com/contact" class="embedl-btn-primary" style="display:inline-block;font-size:13px;font-weight:600;padding:9px 14px;border-radius:6px;border:1px solid #2DD4DD;background:#2DD4DD;color:#0B1626;text-decoration:none;box-shadow:0 6px 18px rgba(45,212,221,0.28);">Get in touch →</a>
      </td>
    </tr>
  </table>
</div>
<!-- embedl-banner:end -->
# Embedl RF-DETR Base (Quantized for TensorRT)

<!-- embedl-run-stickers:start -->
<p class="dark:hidden not-prose" style="margin:8px 0 16px 0;">
  <a href="https://embedl.com/embedl/rf-detr-base/thor?tab=run&amp;utm_source=huggingface&amp;utm_medium=model-card&amp;utm_campaign=run-sticker&amp;utm_content=embedl%2Frf-detr-base-quantized-tensorrt" title="Run rf-detr-base-quantized-tensorrt on AGX Thor: 15.6 ms, free on Embedl's device farm, embedl.com"><img src="https://storage.googleapis.com/embedl-model-media/sha256/d3785dda0e2578ca69369f4fca2061826c6c9b1c0b7b48c28f8a9942897c9c6e.png" alt="Run on AGX Thor: 15.6 ms. Free on Embedl's device farm (embedl.com)" width="270" height="108" style="display:inline-block;margin:0 2px 4px 0;"></a>
</p>
<p class="hidden dark:block not-prose" style="margin:8px 0 16px 0;">
  <a href="https://embedl.com/embedl/rf-detr-base/thor?tab=run&amp;utm_source=huggingface&amp;utm_medium=model-card&amp;utm_campaign=run-sticker&amp;utm_content=embedl%2Frf-detr-base-quantized-tensorrt" title="Run rf-detr-base-quantized-tensorrt on AGX Thor: 15.6 ms, free on Embedl's device farm, embedl.com"><img src="https://storage.googleapis.com/embedl-model-media/sha256/9746b05f167ada95151260c44d1498a1eaaaccd6afca099dd5ae75c51c585011.png" alt="Run on AGX Thor: 15.6 ms. Free on Embedl's device farm (embedl.com)" width="270" height="124" style="display:inline-block;margin:0 2px 4px 0;"></a>
</p>
<!-- embedl-run-stickers:end -->

Deployable INT8-quantized version of [`Roboflow/rf-detr-base`](https://huggingface.co/Roboflow/rf-detr-base),
optimized with [embedl-deploy](https://github.com/embedl/embedl-deploy)
for low-latency NVIDIA TensorRT inference on edge GPUs.

## Upstream Model

<a href="https://hfviewer.com/Roboflow/rf-detr-base?utm_source=huggingface&amp;utm_medium=embedded_model_card&amp;utm_campaign=Roboflow__rf-detr-base_card" target="_blank" rel="noopener">
  <img
    src="https://hfviewer.com/api/card.svg?source=Roboflow%2Frf-detr-base&amp;v=20260501clipcard"
    alt="Open Roboflow/rf-detr-base in hfviewer"
    width="100%"
  />
</a>

## Highlights

- **Mixed-precision INT8/FP16 quantization** with hardware-aware
  optimizations from [embedl-deploy](https://github.com/embedl/embedl-deploy):
  INT8 backbone, FP16 projector and decoder.
- **Drop-in replacement** for `Roboflow/rf-detr-base` in TensorRT pipelines:
  same image input (560×560), same output semantics.
- **Measured on the Jetson with TensorRT**: within 0.87 mAP points of the
  FP16 TensorRT baseline on all 5,000 COCO val2017 images, and 1.11× to
  1.36× faster (see the tables below).
- **Saved at ONNX opset 19**, so TensorRT 10.x (JetPack 6 and 7) builds it
  as intended; see [TensorRT 10 compatibility](#tensorrt-10-compatibility).
- **Faster than `trtexec --best`** on supported NVIDIA hardware (earlier
  revision, Orin Nano Super; see Performance below).
- Includes both **ONNX** (for TensorRT) and **PT2**
  (`torch.export`-loadable) artifacts plus runnable inference scripts.

## Quick Start

```bash
pip install huggingface_hub onnxruntime-gpu pillow numpy
python -c "from huggingface_hub import snapshot_download; snapshot_download('embedl/rf-detr-base-quantized-tensorrt', local_dir='.')"
python infer_trt.py --image path/to/image.jpg   # TensorRT
# or
python infer_pt2.py --image path/to/image.jpg   # pure PyTorch via torch.export
```

## Files

| File | Purpose |
|---|---|
| `embedl_rf-detr-base_int8.onnx` | INT8/FP16 ONNX with Q/DQ nodes, ready for TensorRT (opset 19). |
| `embedl_rf-detr-base_int8.pt2` | The same model for PyTorch (`torch.export` ExportedProgram): INT8 backbone, projector and decoder unquantized. |
| `infer_trt.py` | Build a TRT engine from the ONNX and run sample inference. |
| `infer_pt2.py` | Load the `.pt2` with `torch.export.load` and run sample inference. |

## Accuracy

COCO val2017, all 5,000 images, pycocotools bbox mAP. Each row is a
TensorRT 10.16.2.10 engine built and scored on the Jetson itself
(JetPack 7.2): the image resized to a square 560×560, ImageNet
normalization, top-300 over query × class. The baseline is upstream's
own ONNX export (`rfdetr` 1.8.3) built with `--fp16`.

| Device | TensorRT FP16 mAP@50-95 | **Embedl INT8 mAP@50-95** | Difference | TensorRT FP16 mAP@50 | **Embedl INT8 mAP@50** |
|---|---|---|---|---|---|
| Jetson AGX Orin | 53.29 | **52.42** | −0.87 | 72.14 | **71.60** |
| Jetson Orin NX | 53.30 | **52.43** | −0.87 | 72.21 | **71.55** |
| Jetson Orin Nano Super | 53.28 | **52.43** | −0.85 | 72.19 | **71.56** |
| Jetson AGX Thor | 53.29 | **52.43** | −0.87 | 72.20 | **71.44** |

In PyTorch (the `.pt2` model on the CPU: INT8 backbone, projector and
decoder unquantized), COCO val2017, all 5,000 images, the same
preprocessing and top-300 post-processing:

| Model | mAP@50-95 | mAP@50 |
|---|---|---|
| `Roboflow/rf-detr-base` FP32 (ours) | 53.30% | 72.17% |
| **Embedl RF-DETR Base INT8** | **52.35%** | **71.46%** |

In the same run, the earlier `.pt2` (INT8 throughout, revision
[`5492ae0e`](https://huggingface.co/embedl/rf-detr-base-quantized-tensorrt/tree/5492ae0e296d03ee6686fb02c096669c16cdeab6))
scores 52.15% / 71.24%.

## Performance

GPU compute time per image, median: `trtexec`, CUDA Graph, spin wait,
no data transfers (`--noDataTransfers`), 2 s warm-up and at least 1,000
timed runs, after a build with `--builderOptimizationLevel=3`
(`--fp16`, or `--fp16 --int8` for Embedl INT8). TensorRT 10.16.2.10 on
JetPack 7.2, batch 1, GPU clock locked at its maximum. Both builds were
timed in the same job on the same board.

| Device | TensorRT FP16 | **Embedl INT8** | Speedup vs FP16 |
|---|---|---|---|
| Jetson AGX Orin | 5.99 ms | **5.22 ms** | **1.15×** |
| Jetson Orin NX | 12.26 ms | **10.61 ms** | **1.16×** |
| Jetson Orin Nano Super | 16.39 ms | **12.02 ms** | **1.36×** |
| Jetson AGX Thor | 2.23 ms | **2.01 ms** | **1.11×** |

### Earlier measurement: revision 5492ae0e on the Jetson Orin Nano Super

Latency measured with TensorRT + `trtexec`, GPU compute time only
(`--noDataTransfers`), CUDA Graph + Spin Wait enabled, clocks locked
(max-performance `nvpmodel` mode + `jetson_clocks` on Jetson).

<a href="https://huggingface.co/spaces/embedl/Edge-Inference-Benchmarks" target="_blank" rel="noopener"><img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/rf-detr-base__orin-nano-super-palo-alto__latency.svg" alt="RF-DETR Base latency on NVIDIA Jetson Orin Nano Super"></a>

<a href="https://huggingface.co/spaces/embedl/Edge-Inference-Benchmarks" target="_blank" rel="noopener"><img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/rf-detr-base__orin-nano-super-palo-alto__memory.svg" alt="RF-DETR Base peak memory on NVIDIA Jetson Orin Nano Super"></a>

### NVIDIA Jetson Orin Nano Super

| Configuration | Mean Latency | Speedup vs FP16 |
|---|---|---|
| TensorRT FP16 | 14.83 ms | 1.00x |
| TensorRT --best (unconstrained) | 14.87 ms | 1.00x |
| **Embedl Deploy INT8** | **11.77 ms** | **1.26x** |

## Example Detections

Rendered from the earlier revision (5492ae0e).

Both panels in every figure below are decoded from the **actual
TensorRT engines running on the Jetson Orin Nano Super**: inputs fed
with `trtexec --loadInputs`, raw `dets`/`labels` outputs exported and
drawn offline. Small per-detection score differences between the two
precisions are expected for a two-stage DETR (proposal selection
reorders under precision changes); the detection sets match.

<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/detections_street.png" alt="RF-DETR Base FP16 vs Embedl INT8 detections - street scene">

<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/detections_livingroom.png" alt="RF-DETR Base FP16 vs Embedl INT8 detections - living room">

<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/detections_cats.png" alt="RF-DETR Base FP16 vs Embedl INT8 detections - COCO cats">

Confidence-threshold sweep from 0.80 down to 0.30, both engines in
sync; detections appear at nearly identical thresholds:

<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/threshold_sweep_street.gif" alt="Synced confidence-threshold sweep, TensorRT FP16 vs Embedl INT8">

## TensorRT 10 compatibility

Revision of November 2026. The earlier ONNX (revision
[`5492ae0e`](https://huggingface.co/embedl/rf-detr-base-quantized-tensorrt/tree/5492ae0e296d03ee6686fb02c096669c16cdeab6))
was exported at ONNX opset 20, where the `GridSample` operator spells its
bilinear mode `"linear"`. TensorRT's ONNX parser before 11.2 knows only the
older spelling `"bilinear"` and builds any other mode as nearest-neighbour
without a warning
([onnx-tensorrt 10.16](https://github.com/onnx/onnx-tensorrt/blob/6bde6596eca84abf9afe9b4d21c18c32800fd3ad/onnxOpImporters.cpp#L5951-L5964),
[11.2](https://github.com/onnx/onnx-tensorrt/blob/f7246cce0c88351aa8249a8d26d6113af4d5f7b6/onnxOpImporters.cpp#L6041-L6053)).
RF-DETR's decoder samples its image features with `GridSample`, so with
TensorRT 10.16 (JetPack 7.2) that file lost about 4.5 mAP points (48.8 on
the Jetson AGX Orin); TensorRT 10.3 (JetPack 6) has the same parser code.

This revision is the same model saved at opset 19, where `GridSample` is
spelled `"bilinear"` and `Gelu` is written as `x · 0.5 · (1 + erf(x / √2))`.
The opset change alone leaves the arithmetic as it was: built without its
Q/DQ nodes in FP16, the graph scores 53.29 mAP on the Jetson AGX Orin, the
same as upstream's export. Two further changes bring the INT8 build within
1 mAP point of FP16:

- the projector and the decoder (including the detection heads) run in
  FP16 rather than INT8;
- the backbone's activation scales were re-calibrated (minimum squared
  error on 128 COCO train2017 images).

The `.pt2` file carries the same model for PyTorch: the same weights and
re-calibrated backbone scales, with the projector and decoder unquantized.

## Creating Your Own Optimized Models

This artifact was produced with
[embedl-deploy](https://github.com/embedl/embedl-deploy),
Embedl's open-source PyTorch-to-TensorRT deployment library. You can
apply the same workflow to your own models - see
[the documentation](https://github.com/embedl/embedl-deploy#readme)
for installation and usage.

## License

| Component | License |
|---|---|
| Optimized model artifacts (this repo) | [Embedl Models Community Licence v1.0](https://github.com/embedl/embedl-models/blob/main/LICENSE) (no redistribution as a hosted service) |
| Upstream architecture and weights | [RF-DETR License](https://github.com/roboflow/rf-detr/blob/main/LICENSE) |

## Contact

We offer engineering support for on-prem/edge deployments and partner
co-marketing opportunities. Reach out at
[contact@embedl.com](mailto:contact@embedl.com), or open an issue on
[GitHub](https://github.com/embedl/embedl-deploy).

<!-- embedl-discord-banner:start -->
<style>
.embedl-discord-btn { transition: background 160ms ease, box-shadow 160ms ease; }
.embedl-discord-btn:hover { background: #6C77F5 !important; box-shadow: 0 8px 22px rgba(88,101,242,0.55) !important; }
</style>
<div style="background:radial-gradient(600px 220px at 0% 50%,rgba(88,101,242,0.22) 0%,rgba(88,101,242,0) 60%),radial-gradient(400px 180px at 100% 100%,rgba(88,101,242,0.10) 0%,rgba(88,101,242,0) 55%),linear-gradient(135deg,#0B1626 0%,#142338 100%);border:1px solid rgba(88,101,242,0.35);border-radius:12px;padding:22px 24px;margin:24px 0 0 0;color:#F2F6FA;box-shadow:0 4px 16px rgba(11,22,38,0.18);overflow:hidden;box-sizing:border-box;max-width:100%;">
  <table style="width:100%;border-collapse:collapse;border:0;background:transparent;">
    <tr style="background:transparent;">
      <td style="vertical-align:middle;border:0;padding:0;background:transparent;">
        <div style="display:inline-block;font-size:10px;letter-spacing:0.08em;text-transform:uppercase;font-weight:700;color:#A5B4FC;background:rgba(88,101,242,0.18);border:1px solid rgba(88,101,242,0.45);padding:4px 10px;border-radius:999px;margin-bottom:10px;white-space:nowrap;">Community &amp; support</div>
        <div style="font-size:15px;font-weight:700;line-height:1.35;color:#F2F6FA;margin-bottom:4px;">Need help with this model? Chat with the Embedl team and other engineers on <span style="color:#A5B4FC;white-space:nowrap;">Discord</span>.</div>
        <div style="font-size:13px;color:#9BA7B5;">Quantization gotchas, hardware questions, fine-tuning tips — bring them all.</div>
      </td>
      <td width="1%" style="vertical-align:middle;border:0;padding:0 0 0 18px;white-space:nowrap;text-align:right;background:transparent;">
        <a href="https://discord.gg/MTbMWdKqE" class="embedl-discord-btn" style="display:inline-block;font-size:13px;font-weight:600;padding:9px 14px;border-radius:6px;border:1px solid #5865F2;background:#5865F2;color:#FFFFFF;text-decoration:none;box-shadow:0 6px 18px rgba(88,101,242,0.35);">Join our Discord →</a>
      </td>
    </tr>
  </table>
</div>
<!-- embedl-discord-banner:end -->

	---
	license: other
	license_name: embedl-models-community-licence-1.0
	license_link: https://github.com/embedl/embedl-models/blob/main/LICENSE
	base_model:
	- Roboflow/rf-detr-base
	quantized_from:
	- Roboflow/rf-detr-base
	tags:
	- object-detection
	- quantization
	- onnx
	- tensorrt
	- edge
	- embedl
	gated: true
	extra_gated_heading: "Access Embedl RF-DETR Base"
	extra_gated_description: "To access this model, please review and accept the terms below. Your contact information is collected solely to manage access and, with your explicit consent, to notify you about updated or new optimized models from Embedl."
	extra_gated_button_content: "Agree and request access"
	extra_gated_prompt: "By requesting access you agree to the Embedl Models Community Licence and the upstream RF-DETR License"
	extra_gated_fields:
	Company: text
	I agree to the Embedl Models Community Licence and upstream RF-DETR License: checkbox
	I consent to being contacted by Embedl about products and services (optional): checkbox
	---
	<!-- embedl-banner:start -->
	<style>
	.embedl-btn-primary { transition: background 160ms ease, box-shadow 160ms ease; }
	.embedl-btn-primary:hover { background: #4FDCE4 !important; box-shadow: 0 8px 22px rgba(45,212,221,0.45) !important; }
	.embedl-btn-secondary { transition: background 160ms ease; }
	.embedl-btn-secondary:hover { background: rgba(45,212,221,0.15) !important; }
	.embedl-headline { font-size: clamp(11px, 2.15vw, 15px) !important; }
	.embedl-btn-primary, .embedl-btn-secondary {
	font-size: clamp(11px, 1.65vw, 13px) !important;
	padding: clamp(6px, 1.1vw, 9px) clamp(10px, 1.6vw, 14px) !important;
	}
	</style>
	<div style="background:radial-gradient(600px 220px at 0% 50%,rgba(45,212,221,0.22) 0%,rgba(45,212,221,0) 60%),radial-gradient(400px 180px at 100% 100%,rgba(45,212,221,0.10) 0%,rgba(45,212,221,0) 55%),linear-gradient(135deg,#0B1626 0%,#142338 100%);border:1px solid rgba(45,212,221,0.28);border-radius:12px;padding:22px 24px;margin:0 0 24px 0;color:#F2F6FA;box-shadow:0 4px 16px rgba(11,22,38,0.18);overflow:hidden;box-sizing:border-box;max-width:100%;">
	<table style="width:100%;border-collapse:collapse;border:0;background:transparent;">
	<tr style="background:transparent;">
	<td style="vertical-align:middle;border:0;padding:0;background:transparent;">
	<div style="display:inline-block;font-size:10px;letter-spacing:0.08em;text-transform:uppercase;font-weight:700;color:#2DD4DD;background:rgba(45,212,221,0.15);border:1px solid rgba(45,212,221,0.35);padding:4px 10px;border-radius:999px;margin-bottom:10px;white-space:nowrap;">Optimized by Embedl</div>
	<div class="embedl-headline" style="font-size:15px;font-weight:700;line-height:1.35;color:#F2F6FA;margin-bottom:4px;">Need to <span style="color:#2DD4DD;white-space:nowrap;">fine-tune</span>, hit <span style="color:#2DD4DD;white-space:nowrap;">performance targets</span>, or deploy on <span style="color:#2DD4DD;white-space:nowrap;">specific hardware</span>?</div>
	<div style="font-size:13px;color:#9BA7B5;">We've got you covered.</div>
	</td>
	<td width="1%" style="vertical-align:middle;border:0;padding:0 0 0 18px;white-space:nowrap;text-align:right;background:transparent;">
	<a href="https://www.embedl.com/models" class="embedl-btn-secondary" style="display:inline-block;font-size:13px;font-weight:600;padding:9px 14px;border-radius:6px;border:1px solid #2DD4DD;color:#2DD4DD;text-decoration:none;margin-right:8px;">Learn more</a>
	<a href="https://www.embedl.com/contact" class="embedl-btn-primary" style="display:inline-block;font-size:13px;font-weight:600;padding:9px 14px;border-radius:6px;border:1px solid #2DD4DD;background:#2DD4DD;color:#0B1626;text-decoration:none;box-shadow:0 6px 18px rgba(45,212,221,0.28);">Get in touch →</a>
	</td>
	</tr>
	</table>
	</div>
	<!-- embedl-banner:end -->

	# Embedl RF-DETR Base (Quantized for TensorRT)

	<!-- embedl-run-stickers:start -->
	<p class="dark:hidden not-prose" style="margin:8px 0 16px 0;">
	<a href="https://embedl.com/embedl/rf-detr-base/thor?tab=run&utm_source=huggingface&utm_medium=model-card&utm_campaign=run-sticker&utm_content=embedl%2Frf-detr-base-quantized-tensorrt" title="Run rf-detr-base-quantized-tensorrt on AGX Thor: 15.6 ms, free on Embedl's device farm, embedl.com"><img src="https://storage.googleapis.com/embedl-model-media/sha256/d3785dda0e2578ca69369f4fca2061826c6c9b1c0b7b48c28f8a9942897c9c6e.png" alt="Run on AGX Thor: 15.6 ms. Free on Embedl's device farm (embedl.com)" width="270" height="108" style="display:inline-block;margin:0 2px 4px 0;"></a>
	</p>
	<p class="hidden dark:block not-prose" style="margin:8px 0 16px 0;">
	<a href="https://embedl.com/embedl/rf-detr-base/thor?tab=run&utm_source=huggingface&utm_medium=model-card&utm_campaign=run-sticker&utm_content=embedl%2Frf-detr-base-quantized-tensorrt" title="Run rf-detr-base-quantized-tensorrt on AGX Thor: 15.6 ms, free on Embedl's device farm, embedl.com"><img src="https://storage.googleapis.com/embedl-model-media/sha256/9746b05f167ada95151260c44d1498a1eaaaccd6afca099dd5ae75c51c585011.png" alt="Run on AGX Thor: 15.6 ms. Free on Embedl's device farm (embedl.com)" width="270" height="124" style="display:inline-block;margin:0 2px 4px 0;"></a>
	</p>
	<!-- embedl-run-stickers:end -->

	Deployable INT8-quantized version of [`Roboflow/rf-detr-base`](https://huggingface.co/Roboflow/rf-detr-base),
	optimized with [embedl-deploy](https://github.com/embedl/embedl-deploy)
	for low-latency NVIDIA TensorRT inference on edge GPUs.

	## Upstream Model

	<a href="https://hfviewer.com/Roboflow/rf-detr-base?utm_source=huggingface&utm_medium=embedded_model_card&utm_campaign=Roboflow__rf-detr-base_card" target="_blank" rel="noopener">
	<img
	src="https://hfviewer.com/api/card.svg?source=Roboflow%2Frf-detr-base&v=20260501clipcard"
	alt="Open Roboflow/rf-detr-base in hfviewer"
	width="100%"
	/>
	</a>

	## Highlights

	- Mixed-precision INT8/FP16 quantization with hardware-aware
	optimizations from [embedl-deploy](https://github.com/embedl/embedl-deploy):
	INT8 backbone, FP16 projector and decoder.
	- Drop-in replacement for `Roboflow/rf-detr-base` in TensorRT pipelines:
	same image input (560×560), same output semantics.
	- Measured on the Jetson with TensorRT: within 0.87 mAP points of the
	FP16 TensorRT baseline on all 5,000 COCO val2017 images, and 1.11× to
	1.36× faster (see the tables below).
	- Saved at ONNX opset 19, so TensorRT 10.x (JetPack 6 and 7) builds it
	as intended; see [TensorRT 10 compatibility](#tensorrt-10-compatibility).
	- Faster than `trtexec --best` on supported NVIDIA hardware (earlier
	revision, Orin Nano Super; see Performance below).
	- Includes both ONNX (for TensorRT) and PT2
	(`torch.export`-loadable) artifacts plus runnable inference scripts.

	## Quick Start

	```bash
	pip install huggingface_hub onnxruntime-gpu pillow numpy
	python -c "from huggingface_hub import snapshot_download; snapshot_download('embedl/rf-detr-base-quantized-tensorrt', local_dir='.')"
	python infer_trt.py --image path/to/image.jpg # TensorRT
	# or
	python infer_pt2.py --image path/to/image.jpg # pure PyTorch via torch.export
	```

	## Files

	\| File \| Purpose \|
	\|---\|---\|
	\| `embedl_rf-detr-base_int8.onnx` \| INT8/FP16 ONNX with Q/DQ nodes, ready for TensorRT (opset 19). \|
	\| `embedl_rf-detr-base_int8.pt2` \| The same model for PyTorch (`torch.export` ExportedProgram): INT8 backbone, projector and decoder unquantized. \|
	\| `infer_trt.py` \| Build a TRT engine from the ONNX and run sample inference. \|
	\| `infer_pt2.py` \| Load the `.pt2` with `torch.export.load` and run sample inference. \|

	## Accuracy

	COCO val2017, all 5,000 images, pycocotools bbox mAP. Each row is a
	TensorRT 10.16.2.10 engine built and scored on the Jetson itself
	(JetPack 7.2): the image resized to a square 560×560, ImageNet
	normalization, top-300 over query × class. The baseline is upstream's
	own ONNX export (`rfdetr` 1.8.3) built with `--fp16`.

	\| Device \| TensorRT FP16 mAP@50-95 \| Embedl INT8 mAP@50-95 \| Difference \| TensorRT FP16 mAP@50 \| Embedl INT8 mAP@50 \|
	\|---\|---\|---\|---\|---\|---\|
	\| Jetson AGX Orin \| 53.29 \| 52.42 \| −0.87 \| 72.14 \| 71.60 \|
	\| Jetson Orin NX \| 53.30 \| 52.43 \| −0.87 \| 72.21 \| 71.55 \|
	\| Jetson Orin Nano Super \| 53.28 \| 52.43 \| −0.85 \| 72.19 \| 71.56 \|
	\| Jetson AGX Thor \| 53.29 \| 52.43 \| −0.87 \| 72.20 \| 71.44 \|

	In PyTorch (the `.pt2` model on the CPU: INT8 backbone, projector and
	decoder unquantized), COCO val2017, all 5,000 images, the same
	preprocessing and top-300 post-processing:

	\| Model \| mAP@50-95 \| mAP@50 \|
	\|---\|---\|---\|
	\| `Roboflow/rf-detr-base` FP32 (ours) \| 53.30% \| 72.17% \|
	\| Embedl RF-DETR Base INT8 \| 52.35% \| 71.46% \|

	In the same run, the earlier `.pt2` (INT8 throughout, revision
	[`5492ae0e`](https://huggingface.co/embedl/rf-detr-base-quantized-tensorrt/tree/5492ae0e296d03ee6686fb02c096669c16cdeab6))
	scores 52.15% / 71.24%.

	## Performance

	GPU compute time per image, median: `trtexec`, CUDA Graph, spin wait,
	no data transfers (`--noDataTransfers`), 2 s warm-up and at least 1,000
	timed runs, after a build with `--builderOptimizationLevel=3`
	(`--fp16`, or `--fp16 --int8` for Embedl INT8). TensorRT 10.16.2.10 on
	JetPack 7.2, batch 1, GPU clock locked at its maximum. Both builds were
	timed in the same job on the same board.

	\| Device \| TensorRT FP16 \| Embedl INT8 \| Speedup vs FP16 \|
	\|---\|---\|---\|---\|
	\| Jetson AGX Orin \| 5.99 ms \| 5.22 ms \| 1.15× \|
	\| Jetson Orin NX \| 12.26 ms \| 10.61 ms \| 1.16× \|
	\| Jetson Orin Nano Super \| 16.39 ms \| 12.02 ms \| 1.36× \|
	\| Jetson AGX Thor \| 2.23 ms \| 2.01 ms \| 1.11× \|

	### Earlier measurement: revision 5492ae0e on the Jetson Orin Nano Super

	Latency measured with TensorRT + `trtexec`, GPU compute time only
	(`--noDataTransfers`), CUDA Graph + Spin Wait enabled, clocks locked
	(max-performance `nvpmodel` mode + `jetson_clocks` on Jetson).

	<a href="https://huggingface.co/spaces/embedl/Edge-Inference-Benchmarks" target="_blank" rel="noopener"><img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/rf-detr-base__orin-nano-super-palo-alto__latency.svg" alt="RF-DETR Base latency on NVIDIA Jetson Orin Nano Super"></a>

	<a href="https://huggingface.co/spaces/embedl/Edge-Inference-Benchmarks" target="_blank" rel="noopener"><img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/rf-detr-base__orin-nano-super-palo-alto__memory.svg" alt="RF-DETR Base peak memory on NVIDIA Jetson Orin Nano Super"></a>

	### NVIDIA Jetson Orin Nano Super

	\| Configuration \| Mean Latency \| Speedup vs FP16 \|
	\|---\|---\|---\|
	\| TensorRT FP16 \| 14.83 ms \| 1.00x \|
	\| TensorRT --best (unconstrained) \| 14.87 ms \| 1.00x \|
	\| Embedl Deploy INT8 \| 11.77 ms \| 1.26x \|

	## Example Detections

	Rendered from the earlier revision (5492ae0e).

	Both panels in every figure below are decoded from the **actual
	TensorRT engines running on the Jetson Orin Nano Super**: inputs fed
	with `trtexec --loadInputs`, raw `dets`/`labels` outputs exported and
	drawn offline. Small per-detection score differences between the two
	precisions are expected for a two-stage DETR (proposal selection
	reorders under precision changes); the detection sets match.

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/detections_street.png" alt="RF-DETR Base FP16 vs Embedl INT8 detections - street scene">

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/detections_livingroom.png" alt="RF-DETR Base FP16 vs Embedl INT8 detections - living room">

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/detections_cats.png" alt="RF-DETR Base FP16 vs Embedl INT8 detections - COCO cats">

	Confidence-threshold sweep from 0.80 down to 0.30, both engines in
	sync; detections appear at nearly identical thresholds:

	<img src="https://huggingface.co/datasets/embedl/documentation-images/resolve/main/rf-detr-base-quantized-tensorrt/threshold_sweep_street.gif" alt="Synced confidence-threshold sweep, TensorRT FP16 vs Embedl INT8">

	## TensorRT 10 compatibility

	Revision of November 2026. The earlier ONNX (revision
	[`5492ae0e`](https://huggingface.co/embedl/rf-detr-base-quantized-tensorrt/tree/5492ae0e296d03ee6686fb02c096669c16cdeab6))
	was exported at ONNX opset 20, where the `GridSample` operator spells its
	bilinear mode `"linear"`. TensorRT's ONNX parser before 11.2 knows only the
	older spelling `"bilinear"` and builds any other mode as nearest-neighbour
	without a warning
	([onnx-tensorrt 10.16](https://github.com/onnx/onnx-tensorrt/blob/6bde6596eca84abf9afe9b4d21c18c32800fd3ad/onnxOpImporters.cpp#L5951-L5964),
	[11.2](https://github.com/onnx/onnx-tensorrt/blob/f7246cce0c88351aa8249a8d26d6113af4d5f7b6/onnxOpImporters.cpp#L6041-L6053)).
	RF-DETR's decoder samples its image features with `GridSample`, so with
	TensorRT 10.16 (JetPack 7.2) that file lost about 4.5 mAP points (48.8 on
	the Jetson AGX Orin); TensorRT 10.3 (JetPack 6) has the same parser code.

	This revision is the same model saved at opset 19, where `GridSample` is
	spelled `"bilinear"` and `Gelu` is written as `x · 0.5 · (1 + erf(x / √2))`.
	The opset change alone leaves the arithmetic as it was: built without its
	Q/DQ nodes in FP16, the graph scores 53.29 mAP on the Jetson AGX Orin, the
	same as upstream's export. Two further changes bring the INT8 build within
	1 mAP point of FP16:

	- the projector and the decoder (including the detection heads) run in
	FP16 rather than INT8;
	- the backbone's activation scales were re-calibrated (minimum squared
	error on 128 COCO train2017 images).

	The `.pt2` file carries the same model for PyTorch: the same weights and
	re-calibrated backbone scales, with the projector and decoder unquantized.

	## Creating Your Own Optimized Models

	This artifact was produced with
	[embedl-deploy](https://github.com/embedl/embedl-deploy),
	Embedl's open-source PyTorch-to-TensorRT deployment library. You can
	apply the same workflow to your own models - see
	[the documentation](https://github.com/embedl/embedl-deploy#readme)
	for installation and usage.

	## License

	\| Component \| License \|
	\|---\|---\|
	\| Optimized model artifacts (this repo) \| [Embedl Models Community Licence v1.0](https://github.com/embedl/embedl-models/blob/main/LICENSE) (no redistribution as a hosted service) \|
	\| Upstream architecture and weights \| [RF-DETR License](https://github.com/roboflow/rf-detr/blob/main/LICENSE) \|

	## Contact

	We offer engineering support for on-prem/edge deployments and partner
	co-marketing opportunities. Reach out at
	[contact@embedl.com](mailto:contact@embedl.com), or open an issue on
	[GitHub](https://github.com/embedl/embedl-deploy).

	<!-- embedl-discord-banner:start -->
	<style>
	.embedl-discord-btn { transition: background 160ms ease, box-shadow 160ms ease; }
	.embedl-discord-btn:hover { background: #6C77F5 !important; box-shadow: 0 8px 22px rgba(88,101,242,0.55) !important; }
	</style>
	<div style="background:radial-gradient(600px 220px at 0% 50%,rgba(88,101,242,0.22) 0%,rgba(88,101,242,0) 60%),radial-gradient(400px 180px at 100% 100%,rgba(88,101,242,0.10) 0%,rgba(88,101,242,0) 55%),linear-gradient(135deg,#0B1626 0%,#142338 100%);border:1px solid rgba(88,101,242,0.35);border-radius:12px;padding:22px 24px;margin:24px 0 0 0;color:#F2F6FA;box-shadow:0 4px 16px rgba(11,22,38,0.18);overflow:hidden;box-sizing:border-box;max-width:100%;">
	<table style="width:100%;border-collapse:collapse;border:0;background:transparent;">
	<tr style="background:transparent;">
	<td style="vertical-align:middle;border:0;padding:0;background:transparent;">
	<div style="display:inline-block;font-size:10px;letter-spacing:0.08em;text-transform:uppercase;font-weight:700;color:#A5B4FC;background:rgba(88,101,242,0.18);border:1px solid rgba(88,101,242,0.45);padding:4px 10px;border-radius:999px;margin-bottom:10px;white-space:nowrap;">Community & support</div>
	<div style="font-size:15px;font-weight:700;line-height:1.35;color:#F2F6FA;margin-bottom:4px;">Need help with this model? Chat with the Embedl team and other engineers on <span style="color:#A5B4FC;white-space:nowrap;">Discord</span>.</div>
	<div style="font-size:13px;color:#9BA7B5;">Quantization gotchas, hardware questions, fine-tuning tips — bring them all.</div>
	</td>
	<td width="1%" style="vertical-align:middle;border:0;padding:0 0 0 18px;white-space:nowrap;text-align:right;background:transparent;">
	<a href="https://discord.gg/MTbMWdKqE" class="embedl-discord-btn" style="display:inline-block;font-size:13px;font-weight:600;padding:9px 14px;border-radius:6px;border:1px solid #5865F2;background:#5865F2;color:#FFFFFF;text-decoration:none;box-shadow:0 6px 18px rgba(88,101,242,0.35);">Join our Discord →</a>
	</td>
	</tr>
	</table>
	</div>
	<!-- embedl-discord-banner:end -->