OWLv2 (open-vocabulary, CLIP B/16 and L/14) (fastforks)
OWLv2 (Minderer et al. 2023, Google): open-vocabulary detection by self-training OWL-ViT on web image-text data; a CLIP ViT image tower predicts one box per patch and scores it against CLIP text embeddings of free-text class names. Pass text=[...] at call time (default: the 80 COCO class names); the image is padded to a square and resized to 960 (B/16) or 1008 (L/14) px. The portable module() is the image tower and heads with the COCO names as constant queries.
This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/transformers at c4c4605f9618.
Author weight repos: https://huggingface.co/google/owlv2-base-patch16-ensemble, https://huggingface.co/google/owlv2-large-patch14-ensemble.
Task: detection. Tier: experimental. Outputs: boxes, scores, labels. Backends: portable, kernels.
Input: one RGB image. Output: boxes, class labels, scores and instance masks.
Variants
Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.
| variant | weights (author link) | SHA-256 | weights licence | commercial use |
|---|---|---|---|---|
owlv2-b16@1 |
owlv2-b16.config.json owlv2-b16.safetensors clip-bpe.vocab.json clip-bpe.merges.txt |
ba9df8c25a4be1e130b9e404e089ad92ba369fd691f7c803 |
Apache-2.0 | yes |
owlv2-l14@1 |
owlv2-l14.config.json owlv2-l14.safetensors clip-bpe.vocab.json clip-bpe.merges.txt |
3d45ac1f2698d1c2261503c5e089ad92ba369fd691f7c803 |
Apache-2.0 | yes |
Licence
Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.
Training data: WebLI (Google web image-text data, not released), Objects365 (train, box annotations) (Self-training on WebLI image-text pairs with pseudo-boxes from an OWL-ViT annotator (itself trained on Objects365 and Visual Genome), then the weight ensemble of that model and its fine-tune on Objects365 + Visual Genome; Visual Genome has no dataset id here (OWLv2 paper, arXiv:2306.09683; the ensemble checkpoints).).
Licence evidence: https://huggingface.co/google/owlv2-base-patch16-ensemble/blob/cfd3195ba4ea9592eec887ded089f4c08eff231d/README.md
Input size
any H x W >= 32, padded to a multiple of 32
Tested up to (portable, fp16, RTX 5060 Ti 16 GB): owlv2-b16 3000x4000 (1.1 GiB peak allocated, 1.6 reserved); owlv2-l14 3000x4000 (1.7 GiB peak allocated, 2.3 reserved)
Speed and memory
Detection, synthetic, 800x1344
| model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
|---|---|---|---|---|---|---|---|---|---|
owlv2-b16 |
β | 92.3 | 31.8 | 23.8 | 3.9x | 1.3x | 921 / 960 | 497 / 570 | 607 / 732 |
owlv2-l14 |
β | 212 | 155 | 110 | 1.9x | 1.4x | 2550 / 2606 | 1239 / 1282 | 1552 / 1726 |
Startup and first call
| model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
|---|---|---|---|---|---|---|
owlv2-b16 |
8.22 | 2.81 | 3.00 | 0.74 | 0.32 | 0.27 |
owlv2-l14 |
9.55 | 3.18 | 3.16 | 1.18 | 0.42 | 0.36 |
torch.compile and TensorRT
| model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
|---|---|---|---|---|---|---|---|---|---|---|---|
owlv2-b16 |
800x1344 | 31.8 ms | 28.5 ms | 18.4 ms | 23.8 ms | 10.6 s | 25.9 s | 497 / 570 | 487 / 530 | 607 / 732 | 655 / 726 |
owlv2-l14 |
800x1344 | 155 ms | 141 ms | 95.9 ms | 110 ms | 12.4 s | 33.8 s | 1239 / 1282 | 1190 / 1242 | 1552 / 1726 | 1786 / 1820 |
Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
Accuracy
On public datasets
pycocotools COCOeval of the top 100 detections per image. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | kernels | note |
|---|---|---|---|---|---|---|
owlv2-b16 |
COCO val2017, every 10th image, box mAP | 46.14 | 46.04 | 46.11 | 46.14 | |
owlv2-l14 |
COCO val2017, every 10th image, box mAP | 47.75 | 47.80 | 47.81 | 47.88 |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks a981ff86, 2026-11-12.
Parity with the original code
Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.15 and p99 <= 0.39 (absolute error of the output boxes).
Limitations
- Requires the optional transformers library (the network is its
Owlv2ForObjectDetection). - The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (google/owlv2-*-ensemble); the official scenic (JAX) code was not run.
- Text queries are used as given (no prompt templates or prompt ensembling as in the paper's COCO / LVIS numbers); each is cut to 16 tokens as the transformers processor does.
- No objectness scores and no image-guided (one-shot) detection.
- One box per patch token (3600 for B/16, 5184 for L/14) without NMS, as the original post-processing; overlapping boxes of the same object are common at low thresholds.
Usage
from fastforks import detection
model = detection.load("owlv2-b16") # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg")
result.boxes, result.scores, result.labels # [N, 4], [N], [N]
result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: owlv2-b16, owlv2-l14.
Kernels backend
detection.load("owlv2-b16", mode="kernels") runs our CUDA kernels (fvm_conv). It needs a prebuilt kernel for your GPU architecture (downloaded from https://huggingface.co/fastforks/kernels on first use) and a recent NVIDIA driver; if there is none it raises BackendUnavailableError with the reason and never falls back silently. mode="auto" picks kernels when it can run and logs the choice.
Model tree for fastforks/owlv2
Base model
google/owlv2-base-patch16-ensemble