mnm-matin commited on
Commit
34d132c
·
verified ·
1 Parent(s): a7417bc

Refresh model card with paper figure and complete results

Browse files
Files changed (3) hide show
  1. .gitattributes +1 -0
  2. README.md +153 -84
  3. assets/hyper3clip-overview.png +3 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/hyper3clip-overview.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -13,83 +13,157 @@ tags:
13
 
14
  # Hyper3-CLIP
15
 
16
- Hyper3-CLIP is a hyperbolic vision-language model from hyper³labs. Its 512-d
17
- image and text projections are lifted onto a Lorentz hyperboloid of learned
18
- curvature and trained with entailment objectives over a rule-based caption
19
- hierarchy, so that broader concepts and their more specific descriptions are
20
- related by explicit part-whole and parent-child terms rather than by cosine
21
- similarity alone.
22
-
23
- This repository publishes the raw PyTorch checkpoint for the completed
24
- 500k-step scratch run reported in the paper. It is separate from the
25
- pretrained-base [Hyper3-CLIP v1](https://huggingface.co/hyper3labs/hyper3-clip-v1)
26
- SentenceTransformers release, which is a different model line; do not mix
27
- results between the two.
28
-
29
- - Paper: [Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training](https://arxiv.org/abs/2608.29313) (arXiv:2608.29313)
30
- - Code: [github.com/Hyper3Labs/hyper3-clip](https://github.com/Hyper3Labs/hyper3-clip)
31
- - Venue: oral at the [Beyond Euclidean](https://sites.google.com/view/beyondeuclidean/home) workshop, ECCV 2026
32
-
33
- ## What the model does
34
-
35
- During training, each caption is decomposed into a small hierarchy of queries:
36
- the full caption, sentence fragments, localized part descriptions, and extracted
37
- phrases. Each query conditions an 8-head cross-attention pooling over the image
38
- patch tokens and instantiates its own visual node in the shared hyperbolic
39
- space. Those nodes are supervised through three entailment relations, alongside
40
- the base contrastive and grounded box-level entailment objective.
41
-
42
- Query-conditioned pooling is active only during training. At inference the model
43
- is a standard dual encoder and the pooling module adds no computation.
44
-
45
- ## Artifact
46
-
47
- - Checkpoint: `checkpoint_final.pt`
48
- - Config: `config.yaml`
49
- - Training metadata: `metadata.json`
50
- - Base objective: `uncha` (`model.objective`), with query-conditioned pooling
51
- supplied by the `model.beta_clip_*` keys and `data.beta_clip`; see
52
- `config.yaml` for the full loss configuration
53
- - Vision backbone: `vit_base_patch16_224`, randomly initialized (`vision_pretrained: false`)
54
- - Text architecture/tokenizer: `openai/clip-vit-base-patch32`, randomly initialized (`text_pretrained: false`)
55
- - Embedding dimension: 512, Lorentz projection with learned curvature
56
- - Queries: at most 6 per image, at most 5 localized parts per image
57
- - Training steps: 500,000; global batch size 768; seed 31
58
- - Run id recorded in `eval/summary_wide.csv`: `hier_beta_argent_vit_b_paper_scratch_500k_s31`
59
-
60
- ## Evaluation
61
-
62
- The `eval/` directory holds the paper-comparable full benchmark table and the
63
- raw wide summary row. Headline numbers from that evaluation:
64
-
65
- - COCO I2T/T2I R@10: 84.30 / 73.19
66
- - Flickr I2T/T2I R@10: 97.60 / 91.44
67
- - ImageNet top-1: 46.98%
68
- - WordNet hierarchy: TIE 3.1597, LCA 2.0786, Jaccard 0.8179
69
- - PEP AUC/AP: 96.07 / 69.36
70
-
71
- Against the UNCHA checkpoint this model builds on, all eight reported COCO and
72
- Flickr R@5/R@10 retrieval settings improve, and multi-label mAP under a single
73
- shared evaluator improves on VOC and COCO. Two results in the paper do not
74
- improve and are worth knowing before you use this checkpoint: the ImageNet
75
- hierarchy metrics are competitive with UNCHA rather than better, and zero-shot
76
- classification is unusually sensitive to prompt wording, with large swings on
77
- fine-grained datasets such as Food-101, CUB, and Flowers-102. Treat this as a
78
- research checkpoint, not a polished production model.
79
-
80
- ## Loading
81
-
82
- This is a raw training checkpoint. Use the hyper³labs
83
- [`hyper3-clip`](https://github.com/Hyper3Labs/hyper3-clip) codebase and the
84
- included `config.yaml` to instantiate the model, then load
85
- `checkpoint_final.pt`.
86
-
87
- ```python
88
- import torch
89
-
90
- checkpoint = torch.load("checkpoint_final.pt", map_location="cpu", weights_only=False)
91
- state_dict = checkpoint.get("model", checkpoint)
92
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93
 
94
  ## Citation
95
 
@@ -105,11 +179,6 @@ state_dict = checkpoint.get("model", checkpoint)
105
  }
106
  ```
107
 
108
- ## License And Attribution
109
-
110
- The model materials in this repository are released under OpenMDW-1.0.
111
- Redistributions should preserve `NOTICE`, `LICENSE`, and the model card when
112
- practical.
113
 
114
- Please cite and link to the original hyper³labs model repository when publishing
115
- benchmarks, papers, derivative checkpoints, or public demos based on this model.
 
13
 
14
  # Hyper3-CLIP
15
 
16
+ > For visual search and application development, we recommend [Hyper3-CLIP V1](https://huggingface.co/hyper3labs/hyper3-clip-v1).
17
+
18
+ Hyper3-CLIP is a hyperbolic version of CLIP from hyper³labs. It combines query-conditioned visual pooling with hyperbolic hierarchical representation learning.
19
+
20
+ [Paper](https://arxiv.org/abs/2608.29313) · [Code](https://github.com/Hyper3Labs/hyper3-clip) · [Hyper3-CLIP V1](https://huggingface.co/hyper3labs/hyper3-clip-v1)
21
+
22
+ Presented as an oral at the ECCV 2026 [Beyond Euclidean](https://sites.google.com/view/beyondeuclidean/home) workshop.
23
+
24
+ ## How it works
25
+
26
+ To train the model, we construct lightweight query hierarchies from text, comprising full captions, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions the pooling of visual patches, and the resulting representations support image-text, whole-part, and parent-child entailment losses.
27
+
28
+ [![Hyper3-CLIP overview: caption queries attend to image patches and form visual and text nodes in hyperbolic space.](assets/hyper3clip-overview.png)](https://arxiv.org/html/2608.29313v1#S2.F1)
29
+
30
+ Query-conditioned pooling is active only during training; at inference, Hyper3-CLIP is a standard dual encoder, and the pooling module adds no inference-time computation.
31
+
32
+ ## Model
33
+
34
+ - ViT-B/16 image encoder and CLIP-B/32 text-encoder architecture, trained from scratch.
35
+ - Trained on GRIT for 500,000 steps.
36
+
37
+ ## Results
38
+
39
+ All downstream results are zero-shot evaluations of frozen checkpoints.
40
+
41
+ ### Image-text retrieval
42
+
43
+ Hyper3-CLIP improves over UNCHA across all eight considered COCO/Flickr retrieval settings.
44
+
45
+ [Paper Table 2](https://arxiv.org/html/2608.29313v1#S3.T2), ViT-B/16. Published baseline values are from UNCHA Table 2; Hyper3-CLIP is evaluated with the same retrieval and hierarchy metrics. Each pair reports **R@5 / R@10**. Higher is better.
46
+
47
+ **Image → text**
48
+
49
+ | Model | COCO | Flickr30K |
50
+ | --- | ---: | ---: |
51
+ | CLIP | 71.4 / 81.5 | 93.6 / 96.9 |
52
+ | MERU | 72.3 / 82.0 | 93.5 / 96.2 |
53
+ | ATMG | 62.9 / 74.0 | 85.1 / 92.2 |
54
+ | HyCoCLIP | 72.0 / 82.0 | 92.6 / 95.4 |
55
+ | UNCHA | 72.7 / 82.7 | 91.4 / 95.9 |
56
+ | Hyper3-CLIP | **75.7** / **84.3** | **95.4** / **97.6** |
57
+
58
+ **Text → image**
59
+
60
+ | Model | COCO | Flickr30K |
61
+ | --- | ---: | ---: |
62
+ | CLIP | 57.4 / 68.5 | 83.5 / 89.9 |
63
+ | MERU | 57.4 / 68.6 | 84.0 / 90.0 |
64
+ | ATMG | 51.2 / 62.6 | 78.0 / 85.3 |
65
+ | HyCoCLIP | 58.4 / 69.3 | 84.9 / 90.3 |
66
+ | UNCHA | 60.0 / 71.0 | 84.9 / 91.2 |
67
+ | Hyper3-CLIP | **63.0** / **73.2** | **86.4** / **91.4** |
68
+
69
+ ### ImageNet hierarchy
70
+
71
+ [Paper Table 2](https://arxiv.org/html/2608.29313v1#S3.T2), ViT-B/16. Higher is better except TIE and LCA.
72
+
73
+ | Model | TIE ↓ | LCA ↓ | J ↑ | H-P ↑ | H-R ↑ |
74
+ | --- | ---: | ---: | ---: | ---: | ---: |
75
+ | CLIP | 3.60 | 2.21 | 0.79 | 0.85 | 0.85 |
76
+ | MERU | 3.63 | 2.22 | 0.78 | 0.85 | 0.85 |
77
+ | ATMG | 4.19 | 2.48 | 0.75 | 0.83 | 0.83 |
78
+ | HyCoCLIP | 3.17 | 2.05 | 0.81 | 0.87 | 0.87 |
79
+ | UNCHA | **2.94** | **1.96** | **0.83** | **0.88** | **0.88** |
80
+ | Hyper3-CLIP | 3.16 | 2.08 | 0.82 | 0.87 | **0.88** |
81
+
82
+ ### Zero-shot classification
83
+
84
+ [Paper Table 3](https://arxiv.org/html/2608.29313v1#S3.T3). Values are mean-per-class accuracy. All models use ViT-B/16. Food-101, CUB, and Flowers, marked with *, use the common Photo prompt, `a photo of a {}.`; the remaining datasets use the baseline evaluation prompts.
85
+
86
+ | Dataset | HyCoCLIP | UNCHA | PHyCLIP | Hyper3-CLIP |
87
+ | --- | ---: | ---: | ---: | ---: |
88
+ | ImageNet | 45.80 | **48.80** | 44.31 | 46.98 |
89
+ | CIFAR-10 | 88.80 | 90.40 | 89.33 | **91.14** |
90
+ | CIFAR-100 | 60.10 | 63.20 | 59.05 | **65.64** |
91
+ | SUN397 | 57.20 | 57.70 | 55.32 | **59.39** |
92
+ | Caltech-101 | 81.30 | **83.90** | 76.35 | 83.42 |
93
+ | STL-10 | 95.00 | 95.70 | 94.84 | **96.56** |
94
+ | Food-101* | 61.14 | 64.10 | 62.40 | **67.31** |
95
+ | CUB* | 17.76 | 18.74 | 17.97 | **19.36** |
96
+ | Cars | 11.60 | 14.00 | 10.89 | **16.12** |
97
+ | Aircraft | 3.70 | 3.80 | 3.24 | **4.33** |
98
+ | Pets | 56.80 | 57.10 | 54.18 | **62.57** |
99
+ | Flowers* | 28.79 | 30.75 | 28.27 | **39.66** |
100
+ | DTD | 29.40 | **30.30** | 25.50 | 29.41 |
101
+ | EuroSAT | 35.80 | **41.30** | 36.29 | 28.52 |
102
+ | RESISC45 | 45.60 | **52.70** | 48.22 | 48.94 |
103
+ | Country211 | **6.50** | 6.10 | 5.56 | 6.06 |
104
+ | Average | 45.33 | 47.41 | 44.48 | **47.84** |
105
+
106
+ <details>
107
+ <summary>Prompt comparison (paper Table 6)</summary>
108
+
109
+ The 16-dataset average changes only the three prompt-sensitive columns; all other zero-shot columns are held fixed. Values are mean-per-class accuracy.
110
+
111
+ **Food-101 / CUB / Flowers average**
112
+
113
+ | Model | Official | Photo | Δ |
114
+ | --- | ---: | ---: | ---: |
115
+ | HyCoCLIP | 33.09 | 35.90 | +2.81 |
116
+ | UNCHA | 33.90 | 37.86 | +3.96 |
117
+ | Hyper3-CLIP | 3.26 | **42.11** | **+38.85** |
118
+
119
+ **16-dataset average**
120
+
121
+ | Model | Official | Photo | Δ |
122
+ | --- | ---: | ---: | ---: |
123
+ | HyCoCLIP | 44.82 | 45.33 | +0.51 |
124
+ | UNCHA | 46.69 | 47.41 | +0.72 |
125
+ | Hyper3-CLIP | 40.55 | **47.84** | **+7.29** |
126
+
127
+ [Prompt templates and Table 6](https://arxiv.org/html/2608.29313v1#S4.SS2).
128
+
129
+ </details>
130
+
131
+ ### Multi-label classification
132
+
133
+ Under a shared evaluator, Hyper3-CLIP improves over UNCHA by 2.69 mAP on VOC and 3.35 mAP on COCO.
134
+
135
+ [Paper Table 4](https://arxiv.org/html/2608.29313v1#S3.T4), ViT-B/16. Values are mAP; Avg. is the mean of VOC and COCO. Higher is better.
136
+
137
+ | Model | VOC | COCO | Avg. |
138
+ | --- | ---: | ---: | ---: |
139
+ | CLIP | 68.55 | 44.89 | 56.72 |
140
+ | MERU | 71.29 | 45.33 | 58.31 |
141
+ | HyCoCLIP | 74.89 | 50.73 | 62.81 |
142
+ | UNCHA | 75.51 | 50.93 | 63.22 |
143
+ | PHyCLIP | 73.29 | 49.64 | 61.47 |
144
+ | Hyper3-CLIP | **78.20** | **54.28** | **66.24** |
145
+
146
+ <details>
147
+ <summary>Query-conditioned objective ablation (paper Table 5)</summary>
148
+
149
+ To isolate their effects, each variant is first trained with only the base objective for 80k steps, then continues training to 100k steps with the query entailment terms specified in the table. Data, batch size, optimizer schedule, and seed are fixed; the control rows keep all query entailment terms enabled and modify only the named control.
150
+
151
+ | Variant | Image-text | Visual hierarchy | Text hierarchy | AP | ΔAP | AUROC | ΔR@10 |
152
+ | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
153
+ | No query losses | – | �� | – | 76.21 | 0.00 | 98.34 | 0.00 |
154
+ | Query image-text only | ✓ | – | – | 75.34 | -0.87 | 98.24 | -0.46 |
155
+ | Visual hierarchy only | – | ✓ | – | 71.85 | -4.36 | 98.01 | -0.77 |
156
+ | Text hierarchy only | – | – | ✓ | 78.86 | +2.65 | 98.44 | -0.09 |
157
+ | Full objective | ✓ | ✓ | ✓ | **79.85** | **+3.64** | **98.55** | +0.06 |
158
+ | Mean-pooled visual nodes | ✓ | ✓ | ✓ | 75.97 | -0.25 | 98.32 | **+0.15** |
159
+ | Reversed visual order (query → image) | ✓ | ✓ | ✓ | 76.06 | -0.15 | 98.29 | -0.09 |
160
+ | Shuffled text parents | ✓ | ✓ | ✓ | 77.22 | +1.00 | 98.40 | -0.93 |
161
+
162
+ We report Average Precision (AP) and AUROC over 4,000 positive and 100,000 negative pairs from 1,000 images. Avg. R@10 is the mean of the four COCO/Flickr text- and image-retrieval R@10 values. ΔAP and ΔR@10 are measured relative to the no-query baseline.
163
+
164
+ [Objective components and Table 5](https://arxiv.org/html/2608.29313v1#S4.SS1).
165
+
166
+ </details>
167
 
168
  ## Citation
169
 
 
179
  }
180
  ```
181
 
182
+ ## License
 
 
 
 
183
 
184
+ [OpenMDW-1.0](https://huggingface.co/hyper3labs/hyper3-clip/blob/main/LICENSE) · [NOTICE](https://huggingface.co/hyper3labs/hyper3-clip/blob/main/NOTICE).
 
assets/hyper3clip-overview.png ADDED

Git LFS Details

  • SHA256: e627280397a50c7f9f1d2978cb9ae96546f03c15809253e09aa6372c8d0fe0c0
  • Pointer size: 131 Bytes
  • Size of remote file: 555 kB