--- library_name: "fastforks" pipeline_tag: "object-detection" license: "apache-2.0" tags: - "detection" - "object-detection" - "instance-segmentation" - "cuda" - "pytorch" - "fastforks" - "fastforks-card" base_model: - "google/owlv2-base-patch16-ensemble" - "google/owlv2-large-patch14-ensemble" --- # OWLv2 (open-vocabulary, CLIP B/16 and L/14) (fastforks) OWLv2 (Minderer et al. 2023, Google): open-vocabulary detection by self-training OWL-ViT on web image-text data; a CLIP ViT image tower predicts one box per patch and scores it against CLIP text embeddings of free-text class names. Pass `text=[...]` at call time (default: the 80 COCO class names); the image is padded to a square and resized to 960 (B/16) or 1008 (L/14) px. The portable `module()` is the image tower and heads with the COCO names as constant queries. This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: . Kernels for the `kernels` backend: . Original code: at `c4c4605f9618`. Author weight repos: , . Task: detection. Tier: experimental. Outputs: `boxes`, `scores`, `labels`. Backends: `portable`, `kernels`. Input: one RGB image. Output: boxes, class labels, scores and instance masks. ## Variants Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/detection/models/owlv2/card.toml). | variant | weights (author link) | SHA-256 | weights licence | commercial use | |---|---|---|---|---| | `owlv2-b16@1` | [owlv2-b16.config.json](https://huggingface.co/google/owlv2-base-patch16-ensemble/resolve/cfd3195ba4ea9592eec887ded089f4c08eff231d/config.json)
[owlv2-b16.safetensors](https://huggingface.co/google/owlv2-base-patch16-ensemble/resolve/cfd3195ba4ea9592eec887ded089f4c08eff231d/model.safetensors)
[clip-bpe.vocab.json](https://huggingface.co/google/owlv2-base-patch16-ensemble/resolve/cfd3195ba4ea9592eec887ded089f4c08eff231d/vocab.json)
[clip-bpe.merges.txt](https://huggingface.co/google/owlv2-base-patch16-ensemble/resolve/cfd3195ba4ea9592eec887ded089f4c08eff231d/merges.txt) | `ba9df8c25a4b`
`e1e130b9e404`
`e089ad92ba36`
`9fd691f7c803` | Apache-2.0 | yes | | `owlv2-l14@1` | [owlv2-l14.config.json](https://huggingface.co/google/owlv2-large-patch14-ensemble/resolve/95e26936e865f87db1742128404b3c035d47d89d/config.json)
[owlv2-l14.safetensors](https://huggingface.co/google/owlv2-large-patch14-ensemble/resolve/95e26936e865f87db1742128404b3c035d47d89d/model.safetensors)
[clip-bpe.vocab.json](https://huggingface.co/google/owlv2-base-patch16-ensemble/resolve/cfd3195ba4ea9592eec887ded089f4c08eff231d/vocab.json)
[clip-bpe.merges.txt](https://huggingface.co/google/owlv2-base-patch16-ensemble/resolve/cfd3195ba4ea9592eec887ded089f4c08eff231d/merges.txt) | `3d45ac1f2698`
`d1c2261503c5`
`e089ad92ba36`
`9fd691f7c803` | Apache-2.0 | yes | ## Licence Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0. Training data: WebLI (Google web image-text data, not released), Objects365 (train, box annotations) (Self-training on WebLI image-text pairs with pseudo-boxes from an OWL-ViT annotator (itself trained on Objects365 and Visual Genome), then the weight ensemble of that model and its fine-tune on Objects365 + Visual Genome; Visual Genome has no dataset id here (OWLv2 paper, arXiv:2306.09683; the ensemble checkpoints).). Licence evidence: ## Input size any H x W >= 32, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `owlv2-b16` 3000x4000 (1.1 GiB peak allocated, 1.6 reserved); `owlv2-l14` 3000x4000 (1.7 GiB peak allocated, 2.3 reserved) ## Speed and memory ### Detection, synthetic, 800x1344 | model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `owlv2-b16` | – | 92.3 | 31.8 | 23.8 | 3.9x | 1.3x | 921 / 960 | 497 / 570 | 607 / 732 | | `owlv2-l14` | – | 212 | 155 | 110 | 1.9x | 1.4x | 2550 / 2606 | 1239 / 1282 | 1552 / 1726 | ### Startup and first call | model | load (s): original | portable | kernels | first call (s): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | | `owlv2-b16` | 8.22 | 2.81 | 3.00 | 0.74 | 0.32 | 0.27 | | `owlv2-l14` | 9.55 | 3.18 | 3.16 | 1.18 | 0.42 | 0.36 | ### torch.compile and TensorRT | model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `owlv2-b16` | 800x1344 | 31.8 ms | 28.5 ms | 18.4 ms | 23.8 ms | 10.6 s | 25.9 s | 497 / 570 | 487 / 530 | 607 / 732 | 655 / 726 | | `owlv2-l14` | 800x1344 | 155 ms | 141 ms | 95.9 ms | 110 ms | 12.4 s | 33.8 s | 1239 / 1282 | 1190 / 1242 | 1552 / 1726 | 1786 / 1820 | Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB). Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's. ## Accuracy ### On public datasets pycocotools COCOeval of the top 100 detections per image. Original = the original code named under Details, same checkpoint and protocol. | variant | dataset | original fp32 | original fp16 | portable | kernels | note | |---|---|---|---|---|---|---| | `owlv2-b16` | COCO val2017, every 10th image, box mAP | 46.14 | 46.04 | 46.11 | 46.14 | | | `owlv2-l14` | COCO val2017, every 10th image, box mAP | 47.75 | 47.80 | 47.81 | 47.88 | | Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks a981ff86, 2026-11-12. ### Parity with the original code Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.15 and p99 <= 0.39 (absolute error of the output `boxes`). ## Limitations - Requires the optional transformers library (the network is its `Owlv2ForObjectDetection`). - The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (google/owlv2-*-ensemble); the official scenic (JAX) code was not run. - Text queries are used as given (no prompt templates or prompt ensembling as in the paper's COCO / LVIS numbers); each is cut to 16 tokens as the transformers processor does. - No objectness scores and no image-guided (one-shot) detection. - One box per patch token (3600 for B/16, 5184 for L/14) without NMS, as the original post-processing; overlapping boxes of the same object are common at low thresholds. ## Usage ```python from fastforks import detection model = detection.load("owlv2-b16") # mode="auto": the preferred backend from recorded measurements result = model("photo.jpg") result.boxes, result.scores, result.labels # [N, 4], [N], [N] result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16') ``` Install with `pip install fastforks` (see ). Other variants: `owlv2-b16`, `owlv2-l14`. ## Kernels backend `detection.load("owlv2-b16", mode="kernels")` runs our CUDA kernels (`fvm_conv`). It needs a prebuilt kernel for your GPU architecture (downloaded from on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.