Instructions to use juns/CS-CLIP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- OpenCLIP
How to use juns/CS-CLIP with OpenCLIP:
import open_clip model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:juns/CS-CLIP') tokenizer = open_clip.get_tokenizer('hf-hub:juns/CS-CLIP') - Notebooks
- Google Colab
- Kaggle
|
Download README.md from juns/CS-CLIP: direct link, hf CLI and curl.
- Browser
- Download file 3.28 kB
-
https://huggingface.co/juns/CS-CLIP/resolve/main/README.md
- Command line
-
hf download hf://juns/CS-CLIP/README.md
-
curl -L -o README.md https://huggingface.co/juns/CS-CLIP/resolve/main/README.md
3.28 kB
| license: mit | |
| library_name: open_clip | |
| pipeline_tag: zero-shot-image-classification | |
| base_model: openai/clip-vit-base-patch32 | |
| datasets: | |
| - juns/CS-CLIP-data | |
| tags: | |
| - clip | |
| - open_clip | |
| - vision-language | |
| - compositional-reasoning | |
| # CS-CLIP (ViT-B/32) | |
| CS-CLIP is CLIP ViT-B/32 fine-tuned on scene graph-guided hard negatives for robust compositional reasoning. | |
| It is from the paper [CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning](https://arxiv.org/abs/2609.08242) (Findings of EMNLP 2026). | |
| - Code: https://github.com/statjuns2/CS-CLIP | |
| - Training data and benchmark files: [juns/CS-CLIP-data](https://huggingface.co/datasets/juns/CS-CLIP-data) | |
| The architecture is unchanged OpenAI CLIP ViT-B/32 (QuickGELU), stored in fp32. This repository holds both the OpenCLIP weights (`open_clip_model.safetensors`) and the Transformers weights (`model.safetensors`). | |
| ## Usage | |
| OpenCLIP: | |
| ```python | |
| import torch, open_clip | |
| from PIL import Image | |
| model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:juns/CS-CLIP") | |
| tokenizer = open_clip.get_tokenizer("hf-hub:juns/CS-CLIP") | |
| model.eval() | |
| image = preprocess(Image.open("example.jpg")).unsqueeze(0) | |
| text = tokenizer(["a man riding a brown horse", "a brown man riding a horse"]) | |
| with torch.no_grad(): | |
| sims = model.encode_image(image, normalize=True) @ model.encode_text(text, normalize=True).T | |
| ``` | |
| Transformers: | |
| ```python | |
| from transformers import CLIPModel, CLIPProcessor | |
| model = CLIPModel.from_pretrained("juns/CS-CLIP") | |
| processor = CLIPProcessor.from_pretrained("juns/CS-CLIP") | |
| inputs = processor(text=["a man riding a brown horse", "a brown man riding a horse"], | |
| images=image, return_tensors="pt", padding=True) | |
| logits_per_image = model(**inputs).logits_per_image | |
| ``` | |
| ## Results | |
| Results of this checkpoint, reproduced with the evaluation scripts in the code repository: | |
| | SugarCrepe Rel / Obj / Att | Swap Obj / Att | Add Obj / Att | Avg | VG-Rel | VG-Att | COCO I2T / T2I R@1 | GQA two-choice | | |
| |---|---|---|---|---|---|---|---| | |
| | 83.71 / 94.31 / 87.94 | 78.05 / 81.98 | 90.83 / 89.02 | 88.59 | 85.96 | 77.36 | 53.20 / 42.17 | 71.27 | | |
| The paper's Table 1 reports the mean over three training seeds (SugarCrepe Avg 88.6 ± 0.2); this checkpoint is the seed-120 run. | |
| ## Training | |
| - **Initialization.** OpenAI CLIP ViT-B/32. | |
| - **Data.** 1,941,340 caption / hard-negative pairs on COCO train2014 images ([juns/CS-CLIP-data](https://huggingface.co/datasets/juns/CS-CLIP-data)). | |
| - **Objective.** Structure-CLIP's in-batch margin ranking loss (margin 0.2) plus 5 × a hard-negative hinge loss (margin 0.1). | |
| - **Optimization.** AdamW with a constant learning rate of 2e-6, weight decay 0.1, batch size 128 and 2 epochs, in fp32. | |
| ## Limitations | |
| Like other CLIP derivatives, the model may inherit biases from CLIP's web-scale pre-training data, and it was fine-tuned only on COCO. | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{jeong2026csclip, | |
| title = {{CS-CLIP}: Compositional Scene Graph-guided {CLIP} for Robust Compositional Reasoning}, | |
| author = {Jeong, SeongJun and Jung, Minjoon and Choi, Woo Suk and Jang, Youwon and Zhang, Byoung-Tak}, | |
| booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026}, | |
| year = {2026} | |
| } | |
| ``` | |