README.md

---
pipeline_tag: zero-shot-object-detection
license: apache-2.0
base_model: google/owlvit-base-patch16
library_name: zeromodels
tags:
- keras
- zeromodels
- owlvit
- open-vocabulary
- zero-shot-object-detection
- object-detection
- arxiv:2205.06230
- pytorch
- jax
- tf
---

## ***See [our collection](https://huggingface.co/collections/zeromodels/owl-vit-6a8eaf7f225e1deeba2c22ba) for all versions of OWL-ViT.***

# Run OWL-ViT with Keras 3: JAX, PyTorch, or TensorFlow

[![GitHub](https://img.shields.io/badge/GitHub-ZeroModels-black?logo=github)](https://github.com/IMvision12/ZeroModels) [![Docs](https://img.shields.io/badge/Docs-OWL--ViT-blue)](https://imvision12.github.io/ZeroModels/owlvit/) [![Collection](https://img.shields.io/badge/HF-OWL--ViT%20collection-yellow)](https://huggingface.co/collections/zeromodels/owl-vit-6a8eaf7f225e1deeba2c22ba)

# zeromodels/owlvit-base-patch16

Paper: [Simple Open-Vocabulary Object Detection with Vision Transformers (arXiv:2205.06230)](https://arxiv.org/abs/2205.06230) · [HF Papers](https://huggingface.co/papers/2205.06230)

OWL-ViT detects objects described by free text, with no fixed class list. It starts from a CLIP-style vision and text encoder, then drops CLIP's pooling and attaches a lightweight box head to every patch token. Each patch becomes a detection candidate, scored by cosine similarity against your text queries.

For more details on the model, please go to Google's original [model card](https://huggingface.co/google/owlvit-base-patch16).

Pure-**Keras 3** conversion of [`google/owlvit-base-patch16`](https://huggingface.co/google/owlvit-base-patch16) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.

This is an **open-vocabulary object detection** checkpoint (`OwlViTDetect`): pass free-text prompts at inference time.

## ✨ Quick start

```python
import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.models.owlvit import (
    OwlViTDetect,
    OwlViTProcessor,
    OwlViTImageProcessor,
)

model = OwlViTDetect.from_weights("zeromodels/owlvit-base-patch16")
processor = OwlViTProcessor.from_weights("zeromodels/owlvit-base-patch16")
image_processor = OwlViTImageProcessor.from_weights("zeromodels/owlvit-base-patch16")

image = Image.open("your_image.jpg").convert("RGB")
prompts = ["a photo of a mug", "a photo of a knife"]
inputs = processor(text=[prompts], images=image)
output = model(
    {
        "input_ids": inputs["input_ids"],
        "pixel_values": inputs["pixel_values"],
    }
)
results = image_processor.post_process_object_detection(
    output,
    threshold=0.1,
    target_sizes=[(image.height, image.width)],
    text_labels=[prompts],
)[0]
for score, name, box in zip(
    results["scores"], results["text_labels"], results["boxes"]
):
    print(f"{name}: {float(score):.3f} {box}")
```

Load any OWL-ViT / OWLv2 variant the same way with `from_weights("zeromodels/<variant>")` (use `OwlViTDetect` for this repo):

| Variant | Hub | Family |
|---|---|---|
| `owlvit-base-patch32` | [`zeromodels/owlvit-base-patch32`](https://huggingface.co/zeromodels/owlvit-base-patch32) | OWL-ViT |
| `owlvit-base-patch16` | [`zeromodels/owlvit-base-patch16`](https://huggingface.co/zeromodels/owlvit-base-patch16) | OWL-ViT |
| `owlvit-large-patch14` | [`zeromodels/owlvit-large-patch14`](https://huggingface.co/zeromodels/owlvit-large-patch14) | OWL-ViT |
| `owlv2-base-patch16` | [`zeromodels/owlv2-base-patch16`](https://huggingface.co/zeromodels/owlv2-base-patch16) | OWLv2 |
| `owlv2-base-patch16-ensemble` | [`zeromodels/owlv2-base-patch16-ensemble`](https://huggingface.co/zeromodels/owlv2-base-patch16-ensemble) | OWLv2 |
| `owlv2-base-patch16-finetuned` | [`zeromodels/owlv2-base-patch16-finetuned`](https://huggingface.co/zeromodels/owlv2-base-patch16-finetuned) | OWLv2 |
| `owlv2-large-patch14` | [`zeromodels/owlv2-large-patch14`](https://huggingface.co/zeromodels/owlv2-large-patch14) | OWLv2 |
| `owlv2-large-patch14-ensemble` | [`zeromodels/owlv2-large-patch14-ensemble`](https://huggingface.co/zeromodels/owlv2-large-patch14-ensemble) | OWLv2 |
| `owlv2-large-patch14-finetuned` | [`zeromodels/owlv2-large-patch14-finetuned`](https://huggingface.co/zeromodels/owlv2-large-patch14-finetuned) | OWLv2 |

## Tips

- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `Processor.from_weights(...)` so image size matches the variant.
- Open-vocab thresholds are often much lower than closed-set detectors (try `0.1`).
- OWLv2 pads to square before resize; pass the original `(height, width)` as `target_sizes` carefully (see the OWLv2 docs for the padding trap).
- See [OWL-ViT docs](https://imvision12.github.io/ZeroModels/owlvit/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `OwlViTDetect.from_weights("hf:google/owlvit-base-patch16")`.

## Special Thanks

A huge thank you to the Google OWL-ViT authors for creating and releasing these models.

License: Apache 2.0.