| --- |
| license: mit |
| library_name: open_clip |
| language: |
| - en |
| base_model: openai/clip-vit-base-patch32 |
| base_model_relation: finetune |
| datasets: |
| - kbora/CS-CLIP-Training |
| - kbora/Half-Truths |
| tags: |
| - clip |
| - image-text-retrieval |
| - compositionality |
| - arxiv:2602.23906 |
| --- |
| |
| # CS-CLIP: Component-Supervised CLIP |
|
|
| Model for **[Half-Truths Break Similarity-Based Retrieval](https://arxiv.org/abs/2602.23906)**, NeurIPS 2026. |
| Bora Kargi, Arnas Uselis, Seong Joon Oh |
|
|
| [Project page](https://kargibora.github.io/half-truths/) · [Code](https://github.com/kargibora/CS-CLIP) · [Evaluation data](https://huggingface.co/datasets/kbora/Half-Truths) · [Training annotations](https://huggingface.co/datasets/kbora/CS-CLIP-Training) |
|
|
| CS-CLIP fine-tunes CLIP using entity/relation units and their matched foils to improve image–text retrieval when a caption contains a plausible but incorrect detail. Both encoders are fine-tuned; inference uses the standard dual encoder. |
|
|
| ## Use |
|
|
|
|
| The ViT-B/32 checkpoint loads directly with OpenCLIP: |
|
|
| ```bash |
| pip install 'open_clip_torch>=2.24,<3' |
| ``` |
|
|
| ```python |
| import open_clip |
| import torch |
| from PIL import Image |
| |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| model, preprocess = open_clip.create_model_from_pretrained( |
| "hf-hub:kbora/CS-CLIP", device=device |
| ) |
| model.eval() |
| tokenizer = open_clip.get_tokenizer("ViT-B-32") |
| |
| image = preprocess(Image.open("image.jpg").convert("RGB")).unsqueeze(0).to(device) |
| captions = ["A dog runs on the grass.", "A dog runs on the grass beside a bicycle."] |
| with torch.inference_mode(): |
| image_features = model.encode_image(image, normalize=True) |
| text_features = model.encode_text(tokenizer(captions).to(device), normalize=True) |
| similarities = image_features @ text_features.T |
| print(similarities[0].cpu().tolist()) |
| ``` |
|
|
| Replace `image.jpg` and the captions with your own inputs. Scores are cosine similarities. |
|
|
| ## Checkpoint |
|
|
| - Backbone: OpenAI CLIP ViT-B/32, fine-tuned on COCO images and the released CS-CLIP annotations. |
| - Input: RGB images at 224 × 224; English text with a 77-token context window. |
| - Output: 512-dimensional image and text embeddings. |
| - Files: `open_clip_pytorch_model.bin` contains the encoder weights; `open_clip_config.json` defines the OpenCLIP architecture and image preprocessing. |
|
|
| The checkpoint contains the paper's fine-tuned CLIP encoder weights and loads directly with OpenCLIP. |
|
|
| ## Evaluation |
|
|
| Half-Truth accuracy (%) on the released, human-verified evaluation sets: |
|
|
| | Model | COCO–Qwen | COCO–Mistral | CC3M–Qwen | |
| |---|---:|---:|---:| |
| | CLIP | 34.0 | 28.6 | 36.1 | |
| | CS-CLIP | 76.4 | 67.2 | 66.2 | |
|
|
| CS-CLIP achieves 57.8% average image-to-text accuracy across the paper's 16 compositional benchmarks. See the [repository](https://github.com/kargibora/CS-CLIP#evaluate-half-truths) for evaluation commands and metric definitions. |
|
|
| ## Scope and limitations |
|
|
| The model is intended for research on image–text retrieval and compositional understanding. A similarity score does not establish that every detail in a caption is correct. Training foils are generated automatically and can be noisy. Fine-tuning also has a zero-shot classification trade-off: the paper reports an average of 59.9%, compared with 63.6% for pretrained CLIP. |
|
|
| ## License |
|
|
| Released under the [MIT license](LICENSE). Generated training annotations are CC BY 4.0; source images and captions retain their original terms. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{kargi2026halftruths, |
| title={Half-Truths Break Similarity-Based Retrieval}, |
| author={Kargi, Bora and Uselis, Arnas and Oh, Seong Joon}, |
| booktitle={Advances in Neural Information Processing Systems}, |
| year={2026}, |
| url={https://arxiv.org/abs/2602.23906} |
| } |
| ``` |
|
|
