LW-DETR (ViT, tiny / small / medium / large / xlarge) (fastforks)

LW-DETR (Chen et al. 2024, Baidu): real-time end-to-end detection with a plain ViT encoder (interleaved window and global attention), a C2f projector and a 3-layer deformable DETR decoder over the encoder's best proposals (no NMS); boxes, scores and COCO classes at the input size. Images are resized to 640x640 (aspect ratio ignored). The portable module() is the whole network at its input size (class logits and normalised boxes).

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/transformers at c4c4605f9618.

Author weight repos: https://huggingface.co/AnnaZhang/lwdetr_large_60e_coco, https://huggingface.co/AnnaZhang/lwdetr_medium_60e_coco, https://huggingface.co/AnnaZhang/lwdetr_small_60e_coco, https://huggingface.co/AnnaZhang/lwdetr_tiny_60e_coco, https://huggingface.co/AnnaZhang/lwdetr_xlarge_60e_coco.

Task: detection. Tier: experimental. Outputs: boxes, scores, labels. Backends: portable, kernels.

Input: one RGB image. Output: boxes, class labels, scores and instance masks.

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
lw-detr-large-coco@1 lwdetr-large.config.json
lwdetr-large.model.safetensors
4ce0d981459d
c9d4a1066e29
Apache-2.0 yes
lw-detr-medium-coco@1 lwdetr-medium.config.json
lwdetr-medium.model.safetensors
d04681b8a578
1707a8b1112d
Apache-2.0 yes
lw-detr-small-coco@1 lwdetr-small.config.json
lwdetr-small.model.safetensors
0fda7252545d
152597c899b3
Apache-2.0 yes
lw-detr-tiny-coco@1 lwdetr-tiny.config.json
lwdetr-tiny.model.safetensors
71b79e99f9bf
42dece9592fe
Apache-2.0 yes
lw-detr-xlarge-coco@1 lwdetr-xlarge.config.json
lwdetr-xlarge.model.safetensors
a2d2679802fa
2918313e5081
Apache-2.0 yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.

Training data: COCO 2017 (train, instance annotations), Objects365 (train, box annotations) (COCO 2017 train after detection pretraining on Objects365; the ViT encoders start from CAEv2 self-supervised pretraining (paper arXiv:2406.03459 section 4).).

Licence evidence: https://huggingface.co/AnnaZhang/lwdetr_tiny_60e_coco/blob/4b636b514dcf623f6eafc9e1ab63b8ad5c513925/README.md

Input size

any H x W >= 32, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): lw-detr-large-coco 3000x4000 (0.5 GiB peak allocated, 0.5 reserved); lw-detr-medium-coco 3000x4000 (0.5 GiB peak allocated, 0.5 reserved); lw-detr-small-coco 3000x4000 (0.5 GiB peak allocated, 0.5 reserved); lw-detr-tiny-coco 3000x4000 (0.5 GiB peak allocated, 0.5 reserved); lw-detr-xlarge-coco 3000x4000 (0.7 GiB peak allocated, 0.7 reserved)

Speed and memory

Detection, synthetic, 800x1344

model time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
lw-detr-large-coco – 22.8 10.2 5.64 4.0x 1.8x 355 / 382 204 / 230 192 / 254
lw-detr-medium-coco – 23.3 8.86 3.83 6.1x 2.3x 228 / 274 128 / 136 128 / 180
lw-detr-small-coco – 19.9 8.82 2.80 7.1x 3.1x 129 / 166 97 / 116 99 / 140
lw-detr-tiny-coco – 16.4 7.45 2.20 7.5x 3.4x 114 / 132 92 / 112 94 / 134
lw-detr-xlarge-coco – 25.5 12.1 9.44 2.7x 1.3x 762 / 850 344 / 360 333 / 408

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
lw-detr-large-coco 3.82 3.20 3.04 0.72 0.36 0.34
lw-detr-medium-coco 3.91 2.73 2.93 0.64 0.35 0.33
lw-detr-small-coco 4.14 2.72 2.91 0.63 0.35 0.33
lw-detr-tiny-coco 3.80 2.69 2.91 0.64 0.35 0.33
lw-detr-xlarge-coco 4.72 2.91 3.05 0.71 0.37 0.34

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
lw-detr-large-coco 800x1344 10.2 ms 5.62 ms 3.45 ms 5.64 ms 29.6 s 56.0 s 204 / 230 195 / 208 192 / 254 270 / 296
lw-detr-medium-coco 800x1344 8.86 ms 3.74 ms 2.30 ms 3.83 ms 22.4 s 63.3 s 128 / 136 147 / 174 128 / 180 185 / 212
lw-detr-small-coco 800x1344 8.82 ms 3.63 ms 1.36 ms 2.80 ms 23.0 s 63.1 s 97 / 116 116 / 134 99 / 140 121 / 130
lw-detr-tiny-coco 800x1344 7.45 ms 2.98 ms 0.999 ms 2.20 ms 21.6 s 60.3 s 92 / 112 112 / 130 94 / 134 116 / 128
lw-detr-xlarge-coco 800x1344 12.1 ms 10.7 ms 6.58 ms 9.44 ms 30.5 s 57.5 s 344 / 360 339 / 358 333 / 408 544 / 566

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

pycocotools COCOeval of the top 100 detections per image. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 portable kernels note
lw-detr-tiny-coco COCO val2017, every 20th image, box mAP 49.05 48.81 48.99
lw-detr-large-coco COCO val2017, every 20th image, box mAP 60.67 61.01 60.74

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks aee4b565, 2026-11-13.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.28 and p99 <= 0.93 (absolute error of the output boxes).

Limitations

  • Requires the optional transformers library (the network is its LwDetrForObjectDetection).
  • The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port (AnnaZhang/lwdetr_*_60e_coco weights) of the official LW-DETR (Apache-2.0); the official repository code itself was not run.

Usage

from fastforks import detection

model = detection.load("lw-detr-large-coco")  # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg")
result.boxes, result.scores, result.labels  # [N, 4], [N], [N]
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: lw-detr-large-coco, lw-detr-medium-coco, lw-detr-small-coco, lw-detr-tiny-coco, lw-detr-xlarge-coco.

Kernels backend

detection.load("lw-detr-large-coco", mode="kernels") runs our CUDA kernels (cudnn_hacc). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including fastforks/lwdetr

Paper for fastforks/lwdetr