license: openmdw-1.0
library_name: transformers
pipeline_tag: feature-extraction
tags:
- vision-language
- multimodal
- image-text-retrieval
- hyperbolic-embeddings
hyper3-clip-v1 is a hyperbolic vision–language model for visual search. It brings images and text into a shared embedding space for broad-to-specific retrieval and compositional matching.
- Broad-to-specific retrieval: Search for a broad concept such as “animal” to find different types of animals, then narrow your search with a specific description.
- Compositional matching: Find images by combinations of objects, attributes, and relationships—for example, “a red car beside a tree.” This helps distinguish similar scenes where a color, object, or relationship changes.
Results
| Model | SugarCrepe macro accuracy |
|---|---|
| hyper3-clip-v1 | 79.54% |
| Jina CLIP v2 | 75.02% |
| OpenAI CLIP ViT-B/16 | 73.06% |
Image retrieval — mAP (%) on evaluation subsets of each dataset.
| Dataset | hyper3-clip-v1 | OpenAI CLIP ViT-B/32 |
|---|---|---|
| Amazon Berkeley Objects — product types | 58.2 | 55.2 |
| DeepFashion In-Shop — same item | 63.5 | 35.2 |
| COCO — object categories | 55.4 | 53.2 |
Quick start
Complete the short access form on this page, then install and sign in.
pip install "torch>=2.2" "transformers>=4.49,<5" "timm>=1.0" \
"safetensors>=0.4" "Pillow>=10" "huggingface_hub>=0.34"
hf auth login
For SentenceTransformers, also run pip install "sentence-transformers>=5.5.1,<6".
This example ranks three descriptions of a sofa. The example image downloads automatically.
from PIL import Image
from huggingface_hub import hf_hub_download
from transformers import AutoModel
model_id = "hyper3labs/hyper3-clip-v1"
model = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval()
image = Image.open(hf_hub_download(model_id, "examples/grey-velvet-sofa.jpg")).convert("RGB")
descriptions = ["a grey velvet sofa", "a blue velvet sofa", "a wooden chair"]
scores = model.score(image, descriptions)
for score, text in sorted(zip(scores.tolist(), descriptions), reverse=True):
print(f"{score:.3f} {text}")
The default is Lorentz scoring. Scores are negative; higher means a closer match.
To try your own, use image = Image.open("your-image.jpg").convert("RGB") and change the descriptions — broad words such as “furniture” work alongside specific ones.
Scoring
Use scoring="lorentz", "cosine", or "cone" with model.score(image, descriptions, scoring=...).
| Scoring | What it measures | Use |
|---|---|---|
| Lorentz (default) | Closeness in hyperbolic space; gives the same ranking as hyperbolic distance | General image–text retrieval |
| Cosine | Similarity of embedding directions | Standard vector similarity search |
| Cone | How well an image fits a description's general-to-specific region | Directional and compositional matching |
Higher is better for all three. Their numerical scales differ. The SugarCrepe result above uses cone scoring.
Use with Qdrant
Install the client and start a local Qdrant server:
pip install "qdrant-client>=1.19,<2"
docker run --rm -p 127.0.0.1:6333:6333 qdrant/qdrant:v1.19.1
In Python, continue from the quick start:
from qdrant_client import QdrantClient
index = model.qdrant(QdrantClient(url="http://localhost:6333"), "hyper3_images")
index.create_collection()
index.upsert([image], ids=[1], payloads=[{"label": "grey velvet sofa"}])
for hit in index.search("a grey sofa", limit=5):
print(hit.id, hit.score, hit.payload["label"])
The helper stores one search vector and two geometry values per image. Qdrant finds candidates with its index, then ranks them by Lorentz score in the same request. Only the final results are returned.
For a hosted deployment, pass your server URL and API key to QdrantClient. index.search accepts query_filter for filtering and candidates (default 200) to tune shortlist recall. For image-to-image search, pass an encode_image_lorentz embedding to index.search_embedding. Create payload indexes for your filters before bulk ingestion. Use a separate collection for each model revision.
License and community
hyper3-clip-v1 is free to use under OpenMDW-1.0. The short access form helps us understand how people use the model.