| --- |
| pipeline_tag: zero-shot-image-classification |
| license: apache-2.0 |
| base_model: google/tipsv2-b14 |
| library_name: zeromodels |
| tags: |
| - keras |
| - zeromodels |
| - tipsv2 |
| - zero-shot-image-classification |
| - vision |
| - arxiv:2604.12012 |
| - pytorch |
| - jax |
| - tf |
| --- |
| |
| ## ***See [our collection](https://huggingface.co/collections/zeromodels/tipsv2-6a8eadf49c908c77e74e47eb) for all versions of TIPSv2.*** |
|
|
| # Run TIPSv2 with Keras 3: JAX, PyTorch, or TensorFlow |
|
|
| [](https://github.com/IMvision12/ZeroModels) [](https://huggingface.co/collections/zeromodels/tipsv2-6a8eadf49c908c77e74e47eb) |
|
|
| # zeromodels/tipsv2-b14 |
|
|
| Paper: [TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment (arXiv:2604.12012)](https://huggingface.co/papers/2604.12012) |
|
|
| TIPSv2 (Google DeepMind) is a CLIP/SigLIP-style dual encoder: a DINOv2-style ViT vision tower with register tokens plus a bidirectional text tower, aligned with a temperature-scaled contrastive objective. |
|
|
| For more details on the model, please go to the upstream [model card](https://huggingface.co/google/tipsv2-b14). |
|
|
| Pure-**Keras 3** conversion of [`google/tipsv2-b14`](https://huggingface.co/google/tipsv2-b14) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**. The full model and both towers load from this single repo. |
|
|
| ## ✨ Quick start (zero-shot) |
|
|
| ```python |
| import os |
| os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow" |
| |
| from PIL import Image |
| import numpy as np |
| import keras |
| from zeromodels.models.tipsv2 import Tipsv2Model, Tipsv2Processor |
| |
| model = Tipsv2Model.from_weights("zeromodels/tipsv2-b14") |
| processor = Tipsv2Processor.from_weights("zeromodels/tipsv2-b14") |
| |
| image = Image.open("your_image.jpg").convert("RGB") |
| texts = ["a photo of a cat", "a photo of a dog", "a photo of a car"] |
| inputs = processor(text=texts, images=np.array(image)) |
| out = model(inputs) |
| probs = keras.ops.softmax(out["logits_per_image"], axis=-1) |
| print(keras.ops.convert_to_numpy(probs)[0]) |
| ``` |
|
|
| Towers only: |
|
|
| ```python |
| from zeromodels.models.tipsv2 import Tipsv2VisionModel, Tipsv2TextModel |
| vision = Tipsv2VisionModel.from_weights("zeromodels/tipsv2-b14") |
| text = Tipsv2TextModel.from_weights("zeromodels/tipsv2-b14") |
| ``` |
|
|
| All TIPSv2 variants load the same way with `from_weights("zeromodels/<variant>")`: |
|
|
| | Variant | Hub | |
| |---|---| |
| | `tipsv2-b14` | [`zeromodels/tipsv2-b14`](https://huggingface.co/zeromodels/tipsv2-b14) | |
| | `tipsv2-l14` | [`zeromodels/tipsv2-l14`](https://huggingface.co/zeromodels/tipsv2-l14) | |
| | `tipsv2-so400m14` | [`zeromodels/tipsv2-so400m14`](https://huggingface.co/zeromodels/tipsv2-so400m14) | |
| | `tipsv2-g14` | [`zeromodels/tipsv2-g14`](https://huggingface.co/zeromodels/tipsv2-g14) | |
|
|
| ## Tips |
|
|
| - Set `KERAS_BACKEND` **before** importing Keras / zeromodels. |
| - The image processor rescales to `[0, 1]` (no mean/std normalization); input resolution is 448. |
| - Upstream checkpoints: `Tipsv2Model.from_weights("hf:google/tipsv2-b14")`. |
|
|
| ## Special Thanks |
|
|
| A huge thank you to the TIPSv2 authors (Google DeepMind) and the HF community. |
|
|
| License: Apache-2.0 (matches the upstream `google/tipsv2-b14` checkpoint). |
|
|
