README.md

---
license: openmdw-1.0
pipeline_tag: feature-extraction
tags:
- vision-language
- multimodal
- image-text-retrieval
- hyperbolic-embeddings
- clip
- research
- scratch-training
---

# Hyper3-CLIP

Hyper3-CLIP is a hyperbolic vision-language model from hyper³labs. Its 512-d
image and text projections are lifted onto a Lorentz hyperboloid of learned
curvature and trained with entailment objectives over a rule-based caption
hierarchy, so that broader concepts and their more specific descriptions are
related by explicit part-whole and parent-child terms rather than by cosine
similarity alone.

This repository publishes the raw PyTorch checkpoint for the completed
500k-step scratch run reported in the paper. It is separate from the
pretrained-base [Hyper3-CLIP v1](https://huggingface.co/hyper3labs/hyper3-clip-v1)
SentenceTransformers release, which is a different model line; do not mix
results between the two.

- Paper: [Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training](https://arxiv.org/abs/2608.29313) (arXiv:2608.29313)
- Code: [github.com/Hyper3Labs/hyper3-clip](https://github.com/Hyper3Labs/hyper3-clip)
- Venue: oral at the [Beyond Euclidean](https://sites.google.com/view/beyondeuclidean/home) workshop, ECCV 2026

## What the model does

During training, each caption is decomposed into a small hierarchy of queries:
the full caption, sentence fragments, localized part descriptions, and extracted
phrases. Each query conditions an 8-head cross-attention pooling over the image
patch tokens and instantiates its own visual node in the shared hyperbolic
space. Those nodes are supervised through three entailment relations, alongside
the base contrastive and grounded box-level entailment objective.

Query-conditioned pooling is active only during training. At inference the model
is a standard dual encoder and the pooling module adds no computation.

## Artifact

- Checkpoint: `checkpoint_final.pt`
- Config: `config.yaml`
- Training metadata: `metadata.json`
- Base objective: `uncha` (`model.objective`), with query-conditioned pooling
  supplied by the `model.beta_clip_*` keys and `data.beta_clip`; see
  `config.yaml` for the full loss configuration
- Vision backbone: `vit_base_patch16_224`, randomly initialized (`vision_pretrained: false`)
- Text architecture/tokenizer: `openai/clip-vit-base-patch32`, randomly initialized (`text_pretrained: false`)
- Embedding dimension: 512, Lorentz projection with learned curvature
- Queries: at most 6 per image, at most 5 localized parts per image
- Training steps: 500,000; global batch size 768; seed 31
- Run id recorded in `eval/summary_wide.csv`: `hier_beta_argent_vit_b_paper_scratch_500k_s31`

## Evaluation

The `eval/` directory holds the paper-comparable full benchmark table and the
raw wide summary row. Headline numbers from that evaluation:

- COCO I2T/T2I R@10: 84.30 / 73.19
- Flickr I2T/T2I R@10: 97.60 / 91.44
- ImageNet top-1: 46.98%
- WordNet hierarchy: TIE 3.1597, LCA 2.0786, Jaccard 0.8179
- PEP AUC/AP: 96.07 / 69.36

Against the UNCHA checkpoint this model builds on, all eight reported COCO and
Flickr R@5/R@10 retrieval settings improve, and multi-label mAP under a single
shared evaluator improves on VOC and COCO. Two results in the paper do not
improve and are worth knowing before you use this checkpoint: the ImageNet
hierarchy metrics are competitive with UNCHA rather than better, and zero-shot
classification is unusually sensitive to prompt wording, with large swings on
fine-grained datasets such as Food-101, CUB, and Flowers-102. Treat this as a
research checkpoint, not a polished production model.

## Loading

This is a raw training checkpoint. Use the hyper³labs
[`hyper3-clip`](https://github.com/Hyper3Labs/hyper3-clip) codebase and the
included `config.yaml` to instantiate the model, then load
`checkpoint_final.pt`.

```python
import torch

checkpoint = torch.load("checkpoint_final.pt", map_location="cpu", weights_only=False)
state_dict = checkpoint.get("model", checkpoint)
```

## Citation

```bibtex
@inproceedings{mahmood2026hyper3clip,
  title     = {Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training},
  author    = {Mahmood, Matin and Rueda-Toicen, Antonio and ElBassat, Mohamed and Elkerdany, Seifeldin and Wang, Weixing and de Melo, Gerard},
  booktitle = {Proceedings of the ECCV 2026 Workshop on Beyond Euclidean: Hyperbolic Deep Learning for Computer Vision},
  year      = {2026},
  eprint    = {2608.29313},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV}
}
```

## License And Attribution

The model materials in this repository are released under OpenMDW-1.0.
Redistributions should preserve `NOTICE`, `LICENSE`, and the model card when
practical.

Please cite and link to the original hyper³labs model repository when publishing
benchmarks, papers, derivative checkpoints, or public demos based on this model.