RTMDet (CSPNeXt, tiny / S / M / L / X; nano and M person detectors) (fastforks)

RTMDet (Lyu et al. 2022, OpenMMLab): real-time one-stage detector, all convolutions: a CSPNeXt backbone (5 x 5 depthwise convs), a CSPNeXt-PAFPN neck and a head that predicts, per cell of the stride 8 / 16 / 32 maps, a score per class and the distances to the four box edges; class-wise NMS. COCO checkpoints (80 classes) from mmdetection and the person-only detectors mmpose ships as the box source of RTMPose (same backbone family as fastforks.pose's RTMPose). The image is fitted into 640 x 640 (nano: 320 x 320); vendored from mmdetection (no mmdet / mmcv dependency). The portable module() is the whole network (class logits and decoded boxes per cell, before NMS).

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/open-mmlab/mmdetection at 44ebd17b145c.

Task: detection. Tier: stable. Outputs: boxes, scores, labels. Backends: portable, kernels.

Input: one RGB image. Output: boxes, class labels, scores and instance masks.

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
rtmdet-l-coco@1 rtmdet_l_8xb32-300e_coco_20220719_112030-5a0be7c4.pth 5a0be7c4a812 Apache-2.0 yes
rtmdet-m-coco@1 rtmdet_m_8xb32-300e_coco_20220719_112220-229f527c.pth 229f527ca884 Apache-2.0 yes
rtmdet-m-person@1 rtmdet_m_8xb32-100e_coco-obj365-person-235e8209.pth 235e820939cb Apache-2.0 yes
rtmdet-nano-person@1 rtmdet_nano_8xb32-100e_coco-obj365-person-05d8511e.pth 05d8511e7b3f Apache-2.0 yes
rtmdet-s-coco@1 rtmdet_s_8xb32-300e_coco_20220905_161602-387a891e.pth 387a891e157c Apache-2.0 yes
rtmdet-tiny-coco@1 rtmdet_tiny_8xb32-300e_coco_20220902_112414-78e30dcc.pth 78e30dcce0c6 Apache-2.0 yes
rtmdet-x-coco@1 rtmdet_x_8xb32-300e_coco_20220715_230555-cc79b9ae.pth cc79b9ae0970 Apache-2.0 yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0. Licence files: LICENSE.

Training data: COCO 2017 (train, instance annotations), ImageNet-1k (ILSVRC 2012 classification) (COCO 2017 train (300 epochs); the CSPNeXt backbones start from ImageNet-1k pretraining (configs/rtmdet/README.md)).

Licence evidence: https://github.com/open-mmlab/mmdetection/blob/44ebd17b145c2372c4b700bfb9cb20dbd28ab64a/configs/rtmdet/README.md

Input size

any H x W >= 32, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): rtmdet-l-coco 3000x4000 (0.6 GiB peak allocated, 0.6 reserved); rtmdet-m-coco 3000x4000 (0.5 GiB peak allocated, 0.5 reserved); rtmdet-m-person 3000x4000 (0.5 GiB peak allocated, 0.5 reserved); rtmdet-nano-person 3000x4000 (0.4 GiB peak allocated, 0.5 reserved); rtmdet-s-coco 3000x4000 (0.5 GiB peak allocated, 0.5 reserved); rtmdet-tiny-coco 3000x4000 (0.5 GiB peak allocated, 0.5 reserved); rtmdet-x-coco 3000x4000 (0.7 GiB peak allocated, 0.7 reserved)

Speed and memory

Detection, synthetic, 800x1344

model time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
rtmdet-l-coco 19.5 20.4 9.60 5.74 3.4x 1.7x 332 / 356 183 / 230 178 / 284
rtmdet-m-coco 14.7 17.8 7.69 4.49 3.3x 1.7x 168 / 198 114 / 148 122 / 200
rtmdet-m-person 14.1 16.3 7.39 4.12 3.4x 1.8x 168 / 198 114 / 146 119 / 202
rtmdet-nano-person 12.9 16.0 7.45 2.11 6.1x 3.5x 12 / 30 9 / 28 11 / 52
rtmdet-s-coco 12.1 14.5 6.21 2.97 4.1x 2.1x 73 / 92 53 / 82 48 / 96
rtmdet-tiny-coco 10.7 13.8 6.04 2.74 3.9x 2.2x 48 / 64 40 / 56 40 / 84
rtmdet-x-coco 32.2 24.2 13.3 8.55 2.8x 1.6x 585 / 604 267 / 316 267 / 378

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
rtmdet-l-coco 0.68 0.63 2.07 0.32 1.27 0.36
rtmdet-m-coco 0.46 0.34 1.81 0.31 1.34 0.34
rtmdet-m-person 0.51 0.23 1.72 0.03 0.52 0.21
rtmdet-nano-person 0.14 0.19 4.13 2.69 2.20 0.60
rtmdet-s-coco 0.27 0.28 1.81 0.58 1.58 0.21
rtmdet-tiny-coco 0.22 0.31 2.12 1.24 1.54 0.26
rtmdet-x-coco 1.00 0.77 2.38 0.61 1.25 0.29

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
rtmdet-l-coco 800x1344 9.60 ms 10.6 ms 3.59 ms 5.74 ms 17.2 s 25.0 s 183 / 230 178 / 226 178 / 284 283 / 318
rtmdet-m-coco 800x1344 7.69 ms 4.78 ms 2.39 ms 4.49 ms 15.7 s 22.4 s 114 / 148 115 / 130 122 / 200 156 / 212
rtmdet-m-person 800x1344 7.39 ms 4.44 ms 2.38 ms 4.12 ms 15.5 s 24.8 s 114 / 146 115 / 128 119 / 202 153 / 190
rtmdet-nano-person 800x1344 7.45 ms 4.23 ms 0.769 ms 2.11 ms 16.3 s 22.6 s 9 / 28 48 / 64 11 / 52 49 / 58
rtmdet-s-coco 800x1344 6.21 ms 3.56 ms 1.19 ms 2.97 ms 14.3 s 20.1 s 53 / 82 76 / 98 48 / 96 93 / 130
rtmdet-tiny-coco 800x1344 6.04 ms 3.43 ms 0.962 ms 2.74 ms 13.7 s 19.3 s 40 / 56 64 / 94 40 / 84 80 / 122
rtmdet-x-coco 800x1344 13.3 ms 11.8 ms 6.17 ms 8.55 ms 18.6 s 27.8 s 267 / 316 271 / 292 267 / 378 471 / 504

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

pycocotools COCOeval of the top 100 detections per image. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable kernels note
rtmdet-tiny-coco COCO val2017, every 10th image, box mAP 40.51 39.95 40.48 40.52
rtmdet-s-coco COCO val2017, every 10th image, box mAP 45.36 44.58 45.27 45.37
rtmdet-m-coco COCO val2017, every 10th image, box mAP 49.11 49.14 49.11 49.14
rtmdet-l-coco COCO val2017, every 10th image, box mAP 52.37 52.39 52.37 52.41
rtmdet-x-coco COCO val2017, every 10th image, box mAP 52.79 52.79 52.79 52.85
rtmdet-nano-person COCO val2017, every 10th image, box AP of the person class 36.86 36.79 36.91 36.94
rtmdet-m-person COCO val2017, every 10th image, box AP of the person class 60.20 60.21 60.20 60.21

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, 2026-11-10.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0037 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.16 and p99 <= 0.64 (absolute error of the output boxes).

Limitations

  • The image is resized like OpenCV 4's 8-bit INTER_LINEAR (what mmdetection calls); OpenCV 5 resizes differently by up to two grey levels, so a reference run on OpenCV 5 differs slightly.
  • Boxes are not clamped to the image, as upstream: at the right or bottom border they can reach a few pixels into the padding.
  • The person detectors' released config names COCO only; the training config stored in the checkpoints (and the file name) also lists Objects365.

Usage

from fastforks import detection

model = detection.load("rtmdet-l-coco")  # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg")
result.boxes, result.scores, result.labels  # [N, 4], [N], [N]
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: rtmdet-l-coco, rtmdet-m-coco, rtmdet-m-person, rtmdet-nano-person, rtmdet-s-coco, rtmdet-tiny-coco, rtmdet-x-coco.

Kernels backend

detection.load("rtmdet-l-coco", mode="kernels") runs our CUDA kernels (cudnn_hacc, fvm_conv, fvm_extra, fvm_mb, fvm_r3). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including fastforks/rtmdet