CS-CLIP (ViT-B/32)

CS-CLIP is CLIP ViT-B/32 fine-tuned on scene graph-guided hard negatives for robust compositional reasoning. It is from the paper CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning (Findings of EMNLP 2026).

The architecture is unchanged OpenAI CLIP ViT-B/32 (QuickGELU), stored in fp32. This repository holds both the OpenCLIP weights (open_clip_model.safetensors) and the Transformers weights (model.safetensors).

Usage

OpenCLIP:

import torch, open_clip
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:juns/CS-CLIP")
tokenizer = open_clip.get_tokenizer("hf-hub:juns/CS-CLIP")
model.eval()

image = preprocess(Image.open("example.jpg")).unsqueeze(0)
text = tokenizer(["a man riding a brown horse", "a brown man riding a horse"])
with torch.no_grad():
    sims = model.encode_image(image, normalize=True) @ model.encode_text(text, normalize=True).T

Transformers:

from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("juns/CS-CLIP")
processor = CLIPProcessor.from_pretrained("juns/CS-CLIP")
inputs = processor(text=["a man riding a brown horse", "a brown man riding a horse"],
                   images=image, return_tensors="pt", padding=True)
logits_per_image = model(**inputs).logits_per_image

Results

Results of this checkpoint, reproduced with the evaluation scripts in the code repository:

SugarCrepe Rel / Obj / Att Swap Obj / Att Add Obj / Att Avg VG-Rel VG-Att COCO I2T / T2I R@1 GQA two-choice
83.71 / 94.31 / 87.94 78.05 / 81.98 90.83 / 89.02 88.59 85.96 77.36 53.20 / 42.17 71.27

The paper's Table 1 reports the mean over three training seeds (SugarCrepe Avg 88.6 ± 0.2); this checkpoint is the seed-120 run.

Training

  • Initialization. OpenAI CLIP ViT-B/32.
  • Data. 1,941,340 caption / hard-negative pairs on COCO train2014 images (juns/CS-CLIP-data).
  • Objective. Structure-CLIP's in-batch margin ranking loss (margin 0.2) plus 5 × a hard-negative hinge loss (margin 0.1).
  • Optimization. AdamW with a constant learning rate of 2e-6, weight decay 0.1, batch size 128 and 2 epochs, in fp32.

Limitations

Like other CLIP derivatives, the model may inherit biases from CLIP's web-scale pre-training data, and it was fine-tuned only on COCO.

Citation

@inproceedings{jeong2026csclip,
  title     = {{CS-CLIP}: Compositional Scene Graph-guided {CLIP} for Robust Compositional Reasoning},
  author    = {Jeong, SeongJun and Jung, Minjoon and Choi, Woo Suk and Jang, Youwon and Zhang, Byoung-Tak},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}
Downloads last month
-
Safetensors
Model size
0.2B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for juns/CS-CLIP

Finetuned
(163)
this model

Dataset used to train juns/CS-CLIP

Paper for juns/CS-CLIP