CaBM text classifier: CIFAR-100

Text classifier of Caption Bottleneck Models (CaBM) (paper, ECCV 2026; code) for CIFAR-100 (100 classes).

CaBM classifies an image only from text: a frozen vision-language model writes K censored captions of the image, this RoBERTa-base classifier scores each caption, and the per-caption logits are averaged.

Captioner (frozen) Qwen/Qwen3-VL-2B-Instruct
Captions per image (K) 5
Sampling temperatures 0.7, 0.9, 1.1, 1.3, 1.5
Text classifier roberta-base, final-layer [CLS] -> linear head
Training 30 epochs max, lr 2e-05, effective batch 48, label smoothing 0.1, LLRD 0.85, bottom 2 layers frozen, early stopping on val loss
Captions used datasets/cabm-captions/cifar100
Test top-1 (logit averaging) 81.14%
Test top-5 95.97%

Usage

git clone https://github.com/bariscagliyan/CaptionBottleneckModels && cd CaptionBottleneckModels && pip install -r requirements.txt
huggingface-cli download bariscagliyan/cabm-cifar100 --local-dir runs/cabm-cifar100
import torch
from cabm.classifier import load_trained
model, tok, id2label = load_trained('runs/cabm-cifar100', 'cuda' if torch.cuda.is_available() else 'cpu')
caps = ['<caption 1>', '<caption 2>']  # captions of ONE image from generate_captions.py
x = tok(caps, padding=True, truncation=True, max_length=512, return_tensors='pt').to(model.classifier.weight.device)
with torch.no_grad():
    logits = model(**x)['logits'].mean(0)
print(id2label[int(logits.argmax())])

Captions for new images: generate_captions.py in the GitHub repo (same captioner, prompt and temperatures as above).

Citation

@inproceedings{cagliyan2026cabm,
  title     = {Caption Bottleneck Models},
  author    = {Cagliyan, Seref Baris and Ozdemir, Umut and Tapli, Merve and Akbas, Emre},
  booktitle = {Computer Vision -- ECCV 2026},
  series    = {Lecture Notes in Computer Science},
  volume    = {17036},
  pages     = {445--461},
  publisher = {Springer},
  year      = {2026}
}
Downloads last month
2
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for bariscagliyan/cabm-cifar100

Finetuned
(2,421)
this model

Dataset used to train bariscagliyan/cabm-cifar100

Collection including bariscagliyan/cabm-cifar100

Paper for bariscagliyan/cabm-cifar100

Evaluation results

  • Top-1 accuracy (caption logit averaging) on CIFAR-100
    self-reported
    81.140