juns commited on
Commit
9fa07da
·
verified ·
1 Parent(s): 66884e7

Release CS-CLIP ViT-B/32 checkpoint

Browse files
README.md ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ library_name: open_clip
4
+ pipeline_tag: zero-shot-image-classification
5
+ base_model: openai/clip-vit-base-patch32
6
+ datasets:
7
+ - juns/CS-CLIP-data
8
+ tags:
9
+ - clip
10
+ - open_clip
11
+ - vision-language
12
+ - compositional-reasoning
13
+ ---
14
+
15
+ # CS-CLIP (ViT-B/32)
16
+
17
+ CS-CLIP is CLIP ViT-B/32 fine-tuned on scene graph-guided hard negatives for robust compositional reasoning.
18
+ It is from the paper [CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning](https://arxiv.org/abs/2609.08242) (Findings of EMNLP 2026).
19
+
20
+ - Code: https://github.com/statjuns2/CS-CLIP
21
+ - Training data and benchmark files: [juns/CS-CLIP-data](https://huggingface.co/datasets/juns/CS-CLIP-data)
22
+
23
+ The architecture is unchanged OpenAI CLIP ViT-B/32 (QuickGELU), stored in fp32. This repository holds both the OpenCLIP weights (`open_clip_model.safetensors`) and the Transformers weights (`model.safetensors`).
24
+
25
+ ## Usage
26
+
27
+ OpenCLIP:
28
+
29
+ ```python
30
+ import torch, open_clip
31
+ from PIL import Image
32
+
33
+ model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:juns/CS-CLIP")
34
+ tokenizer = open_clip.get_tokenizer("hf-hub:juns/CS-CLIP")
35
+ model.eval()
36
+
37
+ image = preprocess(Image.open("example.jpg")).unsqueeze(0)
38
+ text = tokenizer(["a man riding a brown horse", "a brown man riding a horse"])
39
+ with torch.no_grad():
40
+ sims = model.encode_image(image, normalize=True) @ model.encode_text(text, normalize=True).T
41
+ ```
42
+
43
+ Transformers:
44
+
45
+ ```python
46
+ from transformers import CLIPModel, CLIPProcessor
47
+
48
+ model = CLIPModel.from_pretrained("juns/CS-CLIP")
49
+ processor = CLIPProcessor.from_pretrained("juns/CS-CLIP")
50
+ inputs = processor(text=["a man riding a brown horse", "a brown man riding a horse"],
51
+ images=image, return_tensors="pt", padding=True)
52
+ logits_per_image = model(**inputs).logits_per_image
53
+ ```
54
+
55
+ ## Results
56
+
57
+ Results of this checkpoint, reproduced with the evaluation scripts in the code repository:
58
+
59
+ | SugarCrepe Rel / Obj / Att | Swap Obj / Att | Add Obj / Att | Avg | VG-Rel | VG-Att | COCO I2T / T2I R@1 | GQA two-choice |
60
+ |---|---|---|---|---|---|---|---|
61
+ | 83.71 / 94.31 / 87.94 | 78.05 / 81.98 | 90.83 / 89.02 | 88.59 | 85.96 | 77.36 | 53.20 / 42.17 | 71.27 |
62
+
63
+ The paper's Table 1 reports the mean over three training seeds (SugarCrepe Avg 88.6 ± 0.2); this checkpoint is the seed-120 run.
64
+
65
+ ## Training
66
+
67
+ - **Initialization.** OpenAI CLIP ViT-B/32.
68
+ - **Data.** 1,941,340 caption / hard-negative pairs on COCO train2014 images ([juns/CS-CLIP-data](https://huggingface.co/datasets/juns/CS-CLIP-data)).
69
+ - **Objective.** Structure-CLIP's in-batch margin ranking loss (margin 0.2) plus 5 × a hard-negative hinge loss (margin 0.1).
70
+ - **Optimization.** AdamW with a constant learning rate of 2e-6, weight decay 0.1, batch size 128 and 2 epochs, in fp32.
71
+
72
+ ## Limitations
73
+
74
+ Like other CLIP derivatives, the model may inherit biases from CLIP's web-scale pre-training data, and it was fine-tuned only on COCO.
75
+
76
+ ## Citation
77
+
78
+ ```bibtex
79
+ @inproceedings{jeong2026csclip,
80
+ title = {{CS-CLIP}: Compositional Scene Graph-guided {CLIP} for Robust Compositional Reasoning},
81
+ author = {Jeong, SeongJun and Jung, Minjoon and Choi, Woo Suk and Jang, Youwon and Zhang, Byoung-Tak},
82
+ booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
83
+ year = {2026}
84
+ }
85
+ ```
config.json ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "openai/clip-vit-base-patch32",
3
+ "architectures": [
4
+ "CLIPModel"
5
+ ],
6
+ "initializer_factor": 1.0,
7
+ "logit_scale_init_value": 2.6592,
8
+ "model_type": "clip",
9
+ "projection_dim": 512,
10
+ "text_config": {
11
+ "bos_token_id": 0,
12
+ "dropout": 0.0,
13
+ "eos_token_id": 2,
14
+ "model_type": "clip_text_model"
15
+ },
16
+ "torch_dtype": "float32",
17
+ "transformers_version": "4.44.2",
18
+ "vision_config": {
19
+ "dropout": 0.0,
20
+ "model_type": "clip_vision_model"
21
+ }
22
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7f725ca0e25021be0da848e61b8b087e5c6446ac5d99a8d9d93b2ee7a5bd403
3
+ size 605156676
open_clip_config.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_cfg": {
3
+ "embed_dim": 512,
4
+ "quick_gelu": true,
5
+ "vision_cfg": {
6
+ "image_size": 224,
7
+ "layers": 12,
8
+ "width": 768,
9
+ "patch_size": 32
10
+ },
11
+ "text_cfg": {
12
+ "context_length": 77,
13
+ "vocab_size": 49408,
14
+ "width": 512,
15
+ "heads": 8,
16
+ "layers": 12
17
+ }
18
+ },
19
+ "preprocess_cfg": {
20
+ "mean": [
21
+ 0.48145466,
22
+ 0.4578275,
23
+ 0.40821073
24
+ ],
25
+ "std": [
26
+ 0.26862954,
27
+ 0.26130258,
28
+ 0.27577711
29
+ ],
30
+ "interpolation": "bicubic",
31
+ "resize_mode": "shortest"
32
+ }
33
+ }
open_clip_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ccc441a6a5f7c31b93c222f5dbc65e72aedc29bd2805136bc40473dde4881124
3
+ size 605143316
preprocessor_config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "crop_size": {
3
+ "height": 224,
4
+ "width": 224
5
+ },
6
+ "do_center_crop": true,
7
+ "do_convert_rgb": true,
8
+ "do_normalize": true,
9
+ "do_rescale": true,
10
+ "do_resize": true,
11
+ "image_mean": [
12
+ 0.48145466,
13
+ 0.4578275,
14
+ 0.40821073
15
+ ],
16
+ "image_processor_type": "CLIPImageProcessor",
17
+ "image_std": [
18
+ 0.26862954,
19
+ 0.26130258,
20
+ 0.27577711
21
+ ],
22
+ "processor_class": "CLIPProcessor",
23
+ "resample": 3,
24
+ "rescale_factor": 0.00392156862745098,
25
+ "size": {
26
+ "shortest_edge": 224
27
+ }
28
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<|startoftext|>",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "<|endoftext|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "<|endoftext|>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "unk_token": {
24
+ "content": "<|endoftext|>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ }
30
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "added_tokens_decoder": {
4
+ "49406": {
5
+ "content": "<|startoftext|>",
6
+ "lstrip": false,
7
+ "normalized": true,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "49407": {
13
+ "content": "<|endoftext|>",
14
+ "lstrip": false,
15
+ "normalized": false,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ }
20
+ },
21
+ "bos_token": "<|startoftext|>",
22
+ "clean_up_tokenization_spaces": true,
23
+ "do_lower_case": true,
24
+ "eos_token": "<|endoftext|>",
25
+ "errors": "replace",
26
+ "model_max_length": 77,
27
+ "pad_token": "<|endoftext|>",
28
+ "processor_class": "CLIPProcessor",
29
+ "tokenizer_class": "CLIPTokenizer",
30
+ "unk_token": "<|endoftext|>"
31
+ }
vocab.json ADDED
The diff for this file is too large to render. See raw diff