CS-CLIP / README.md
juns's picture
Release CS-CLIP ViT-B/32 checkpoint
9fa07da verified
|
Raw History Blame Contribute Delete
3.28 kB
---
license: mit
library_name: open_clip
pipeline_tag: zero-shot-image-classification
base_model: openai/clip-vit-base-patch32
datasets:
- juns/CS-CLIP-data
tags:
- clip
- open_clip
- vision-language
- compositional-reasoning
---
# CS-CLIP (ViT-B/32)
CS-CLIP is CLIP ViT-B/32 fine-tuned on scene graph-guided hard negatives for robust compositional reasoning.
It is from the paper [CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning](https://arxiv.org/abs/2609.08242) (Findings of EMNLP 2026).
- Code: https://github.com/statjuns2/CS-CLIP
- Training data and benchmark files: [juns/CS-CLIP-data](https://huggingface.co/datasets/juns/CS-CLIP-data)
The architecture is unchanged OpenAI CLIP ViT-B/32 (QuickGELU), stored in fp32. This repository holds both the OpenCLIP weights (`open_clip_model.safetensors`) and the Transformers weights (`model.safetensors`).
## Usage
OpenCLIP:
```python
import torch, open_clip
from PIL import Image
model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:juns/CS-CLIP")
tokenizer = open_clip.get_tokenizer("hf-hub:juns/CS-CLIP")
model.eval()
image = preprocess(Image.open("example.jpg")).unsqueeze(0)
text = tokenizer(["a man riding a brown horse", "a brown man riding a horse"])
with torch.no_grad():
sims = model.encode_image(image, normalize=True) @ model.encode_text(text, normalize=True).T
```
Transformers:
```python
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("juns/CS-CLIP")
processor = CLIPProcessor.from_pretrained("juns/CS-CLIP")
inputs = processor(text=["a man riding a brown horse", "a brown man riding a horse"],
images=image, return_tensors="pt", padding=True)
logits_per_image = model(**inputs).logits_per_image
```
## Results
Results of this checkpoint, reproduced with the evaluation scripts in the code repository:
| SugarCrepe Rel / Obj / Att | Swap Obj / Att | Add Obj / Att | Avg | VG-Rel | VG-Att | COCO I2T / T2I R@1 | GQA two-choice |
|---|---|---|---|---|---|---|---|
| 83.71 / 94.31 / 87.94 | 78.05 / 81.98 | 90.83 / 89.02 | 88.59 | 85.96 | 77.36 | 53.20 / 42.17 | 71.27 |
The paper's Table 1 reports the mean over three training seeds (SugarCrepe Avg 88.6 ± 0.2); this checkpoint is the seed-120 run.
## Training
- **Initialization.** OpenAI CLIP ViT-B/32.
- **Data.** 1,941,340 caption / hard-negative pairs on COCO train2014 images ([juns/CS-CLIP-data](https://huggingface.co/datasets/juns/CS-CLIP-data)).
- **Objective.** Structure-CLIP's in-batch margin ranking loss (margin 0.2) plus 5 × a hard-negative hinge loss (margin 0.1).
- **Optimization.** AdamW with a constant learning rate of 2e-6, weight decay 0.1, batch size 128 and 2 epochs, in fp32.
## Limitations
Like other CLIP derivatives, the model may inherit biases from CLIP's web-scale pre-training data, and it was fine-tuned only on COCO.
## Citation
```bibtex
@inproceedings{jeong2026csclip,
title = {{CS-CLIP}: Compositional Scene Graph-guided {CLIP} for Robust Compositional Reasoning},
author = {Jeong, SeongJun and Jung, Minjoon and Choi, Woo Suk and Jang, Youwon and Zhang, Byoung-Tak},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026}
}
```