File size: 5,150 Bytes
875ccfb a155511 6352875 875ccfb 6352875 875ccfb a155511 875ccfb a155511 875ccfb f87a6eb a155511 f87a6eb a155511 6352875 a155511 6352875 875ccfb a155511 875ccfb a155511 6352875 a155511 6352875 a155511 875ccfb a155511 6352875 a155511 6352875 a155511 6352875 a155511 6352875 a155511 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | ---
pipeline_tag: zero-shot-object-detection
license: apache-2.0
base_model: google/owlvit-large-patch14
library_name: zeromodels
tags:
- keras
- zeromodels
- owlvit
- open-vocabulary
- zero-shot-object-detection
- object-detection
- arxiv:2205.06230
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/owl-vit-6a8eaf7f225e1deeba2c22ba) for all versions of OWL-ViT.***
# Run OWL-ViT with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/owlvit/) [](https://huggingface.co/collections/zeromodels/owl-vit-6a8eaf7f225e1deeba2c22ba)
# zeromodels/owlvit-large-patch14
Paper: [Simple Open-Vocabulary Object Detection with Vision Transformers (arXiv:2205.06230)](https://arxiv.org/abs/2205.06230) · [HF Papers](https://huggingface.co/papers/2205.06230)
OWL-ViT detects objects described by free text, with no fixed class list. It starts from a CLIP-style vision and text encoder, then drops CLIP's pooling and attaches a lightweight box head to every patch token. Each patch becomes a detection candidate, scored by cosine similarity against your text queries.
For more details on the model, please go to Google's original [model card](https://huggingface.co/google/owlvit-large-patch14).
Pure-**Keras 3** conversion of [`google/owlvit-large-patch14`](https://huggingface.co/google/owlvit-large-patch14) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is an **open-vocabulary object detection** checkpoint (`OwlViTDetect`): pass free-text prompts at inference time.
## ✨ Quick start
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from PIL import Image
from zeromodels.models.owlvit import (
OwlViTDetect,
OwlViTProcessor,
OwlViTImageProcessor,
)
model = OwlViTDetect.from_weights("zeromodels/owlvit-large-patch14")
processor = OwlViTProcessor.from_weights("zeromodels/owlvit-large-patch14")
image_processor = OwlViTImageProcessor.from_weights("zeromodels/owlvit-large-patch14")
image = Image.open("your_image.jpg").convert("RGB")
prompts = ["a photo of a mug", "a photo of a knife"]
inputs = processor(text=[prompts], images=image)
output = model(
{
"input_ids": inputs["input_ids"],
"pixel_values": inputs["pixel_values"],
}
)
results = image_processor.post_process_object_detection(
output,
threshold=0.1,
target_sizes=[(image.height, image.width)],
text_labels=[prompts],
)[0]
for score, name, box in zip(
results["scores"], results["text_labels"], results["boxes"]
):
print(f"{name}: {float(score):.3f} {box}")
```
Load any OWL-ViT / OWLv2 variant the same way with `from_weights("zeromodels/<variant>")` (use `OwlViTDetect` for this repo):
| Variant | Hub | Family |
|---|---|---|
| `owlvit-base-patch32` | [`zeromodels/owlvit-base-patch32`](https://huggingface.co/zeromodels/owlvit-base-patch32) | OWL-ViT |
| `owlvit-base-patch16` | [`zeromodels/owlvit-base-patch16`](https://huggingface.co/zeromodels/owlvit-base-patch16) | OWL-ViT |
| `owlvit-large-patch14` | [`zeromodels/owlvit-large-patch14`](https://huggingface.co/zeromodels/owlvit-large-patch14) | OWL-ViT |
| `owlv2-base-patch16` | [`zeromodels/owlv2-base-patch16`](https://huggingface.co/zeromodels/owlv2-base-patch16) | OWLv2 |
| `owlv2-base-patch16-ensemble` | [`zeromodels/owlv2-base-patch16-ensemble`](https://huggingface.co/zeromodels/owlv2-base-patch16-ensemble) | OWLv2 |
| `owlv2-base-patch16-finetuned` | [`zeromodels/owlv2-base-patch16-finetuned`](https://huggingface.co/zeromodels/owlv2-base-patch16-finetuned) | OWLv2 |
| `owlv2-large-patch14` | [`zeromodels/owlv2-large-patch14`](https://huggingface.co/zeromodels/owlv2-large-patch14) | OWLv2 |
| `owlv2-large-patch14-ensemble` | [`zeromodels/owlv2-large-patch14-ensemble`](https://huggingface.co/zeromodels/owlv2-large-patch14-ensemble) | OWLv2 |
| `owlv2-large-patch14-finetuned` | [`zeromodels/owlv2-large-patch14-finetuned`](https://huggingface.co/zeromodels/owlv2-large-patch14-finetuned) | OWLv2 |
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `Processor.from_weights(...)` so image size matches the variant.
- Open-vocab thresholds are often much lower than closed-set detectors (try `0.1`).
- OWLv2 pads to square before resize; pass the original `(height, width)` as `target_sizes` carefully (see the OWLv2 docs for the padding trap).
- See [OWL-ViT docs](https://imvision12.github.io/ZeroModels/owlvit/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `OwlViTDetect.from_weights("hf:google/owlvit-large-patch14")`.
## Special Thanks
A huge thank you to the Google OWL-ViT authors for creating and releasing these models.
License: Apache 2.0.
|