README.md

---
license: mit
library_name: open_clip
language:
- en
base_model: openai/clip-vit-base-patch32
base_model_relation: finetune
datasets:
- kbora/CS-CLIP-Training
- kbora/Half-Truths
tags:
- clip
- image-text-retrieval
- compositionality
- arxiv:2602.23906
---

# CS-CLIP: Component-Supervised CLIP

Model for **[Half-Truths Break Similarity-Based Retrieval](https://arxiv.org/abs/2602.23906)**, NeurIPS 2026.
Bora Kargi, Arnas Uselis, Seong Joon Oh

[Project page](https://kargibora.github.io/half-truths/) · [Code](https://github.com/kargibora/CS-CLIP) · [Evaluation data](https://huggingface.co/datasets/kbora/Half-Truths) · [Training annotations](https://huggingface.co/datasets/kbora/CS-CLIP-Training)

CS-CLIP fine-tunes CLIP using entity/relation units and their matched foils to improve image–text retrieval when a caption contains a plausible but incorrect detail. Both encoders are fine-tuned; inference uses the standard dual encoder.

## Use


The ViT-B/32 checkpoint loads directly with OpenCLIP:

```bash
pip install 'open_clip_torch>=2.24,<3'
```

```python
import open_clip
import torch
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = open_clip.create_model_from_pretrained(
    "hf-hub:kbora/CS-CLIP", device=device
)
model.eval()
tokenizer = open_clip.get_tokenizer("ViT-B-32")

image = preprocess(Image.open("image.jpg").convert("RGB")).unsqueeze(0).to(device)
captions = ["A dog runs on the grass.", "A dog runs on the grass beside a bicycle."]
with torch.inference_mode():
    image_features = model.encode_image(image, normalize=True)
    text_features = model.encode_text(tokenizer(captions).to(device), normalize=True)
    similarities = image_features @ text_features.T
print(similarities[0].cpu().tolist())
```

Replace `image.jpg` and the captions with your own inputs. Scores are cosine similarities.

## Checkpoint

- Backbone: OpenAI CLIP ViT-B/32, fine-tuned on COCO images and the released CS-CLIP annotations.
- Input: RGB images at 224 × 224; English text with a 77-token context window.
- Output: 512-dimensional image and text embeddings.
- Files: `open_clip_pytorch_model.bin` contains the encoder weights; `open_clip_config.json` defines the OpenCLIP architecture and image preprocessing.

The checkpoint contains the paper's fine-tuned CLIP encoder weights and loads directly with OpenCLIP.

## Evaluation

Half-Truth accuracy (%) on the released, human-verified evaluation sets:

| Model | COCO–Qwen | COCO–Mistral | CC3M–Qwen |
|---|---:|---:|---:|
| CLIP | 34.0 | 28.6 | 36.1 |
| CS-CLIP | 76.4 | 67.2 | 66.2 |

CS-CLIP achieves 57.8% average image-to-text accuracy across the paper's 16 compositional benchmarks. See the [repository](https://github.com/kargibora/CS-CLIP#evaluate-half-truths) for evaluation commands and metric definitions.

## Scope and limitations

The model is intended for research on image–text retrieval and compositional understanding. A similarity score does not establish that every detail in a caption is correct. Training foils are generated automatically and can be noisy. Fine-tuning also has a zero-shot classification trade-off: the paper reports an average of 59.9%, compared with 63.6% for pretrained CLIP.

## License

Released under the [MIT license](LICENSE). Generated training annotations are CC BY 4.0; source images and captions retain their original terms.

## Citation

```bibtex
@inproceedings{kargi2026halftruths,
  title={Half-Truths Break Similarity-Based Retrieval},
  author={Kargi, Bora and Uselis, Arnas and Oh, Seong Joon},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026},
  url={https://arxiv.org/abs/2602.23906}
}
```