Caption Bottleneck Models (ECCV 2026)
Collection
7 items โข Updated
Text classifier of Caption Bottleneck Models (CaBM) (paper, ECCV 2026; code) for Oxford Flowers-102 (102 classes).
CaBM classifies an image only from text: a frozen vision-language model writes K censored captions of the image, this RoBERTa-base classifier scores each caption, and the per-caption logits are averaged.
| Captioner (frozen) | Qwen/Qwen3-VL-2B-Instruct |
| Captions per image (K) | 5 |
| Sampling temperatures | 0.7, 0.9, 1.1, 1.3, 1.5 |
| Text classifier | roberta-base, final-layer [CLS] -> linear head |
| Training | 30 epochs max, lr 2e-05, effective batch 48, label smoothing 0.1, LLRD 0.85, bottom 2 layers frozen, early stopping on val loss |
| Captions used | datasets/cabm-captions/flowers102 |
| Test top-1 (logit averaging) | 84.23% |
| Test top-5 | 96.05% |
git clone https://github.com/bariscagliyan/CaptionBottleneckModels && cd CaptionBottleneckModels && pip install -r requirements.txt
huggingface-cli download bariscagliyan/cabm-flowers102 --local-dir runs/cabm-flowers102
import torch
from cabm.classifier import load_trained
model, tok, id2label = load_trained('runs/cabm-flowers102', 'cuda' if torch.cuda.is_available() else 'cpu')
caps = ['<caption 1>', '<caption 2>'] # captions of ONE image from generate_captions.py
x = tok(caps, padding=True, truncation=True, max_length=512, return_tensors='pt').to(model.classifier.weight.device)
with torch.no_grad():
logits = model(**x)['logits'].mean(0)
print(id2label[int(logits.argmax())])
Captions for new images: generate_captions.py in the GitHub repo (same captioner, prompt and temperatures as above).
@inproceedings{cagliyan2026cabm,
title = {Caption Bottleneck Models},
author = {Cagliyan, Seref Baris and Ozdemir, Umut and Tapli, Merve and Akbas, Emre},
booktitle = {Computer Vision -- ECCV 2026},
series = {Lecture Notes in Computer Science},
volume = {17036},
pages = {445--461},
publisher = {Springer},
year = {2026}
}
Base model
FacebookAI/roberta-base