README.md

metadata

license: mit
pipeline_tag: image-to-text
tags:
  - image-captioning
  - zero-shot-image-captioning
  - reranking
  - training-free
  - multi-agent
datasets:
  - coco

FRAMA: frozen multi-agent adjudication for zero-shot image captioning

Inference code and assets for "A Reranking Reality Check for Zero-Shot Image Captioning: Leakage-Free Oracles and System One Adjudication". Code for every table of the paper, with the released beam dumps and agent scores: github.com/bradduy/FRAMA.

FRAMA leaves a zero-shot captioner frozen and only decides which of its beam candidates to return. Six frozen agents score each candidate, every score is z-normalised within the beam, and the candidate with the highest equal-weight sum wins. No parameter is trained or tuned.

Agent	Model	Signal
LM	IFCap decoder	length-normalised log-likelihood
bigG	OpenCLIP ViT-bigG/14, LAION-2B	image-caption cosine
BLIP	Salesforce/blip-itm-base-coco	image-text match probability
VQA	Salesforce/blip2-flan-t5-xl	P(yes) to "Does this figure show ...?", the System One verifier
MBR	none	mean CIDEr-D of the candidate against the other 19
ground	google/owlv2-base-patch16-ensemble	mean detection score of the caption's nouns

Quick start

hf download bradduy/FRAMA --local-dir FRAMA && cd FRAMA
pip install -r requirements.txt
python frama_infer.py --images photo1.jpg photo2.jpg --out captions.json
python frama_infer.py --image_dir my_photos/ --out captions.json --dump_beam

Output, one record per image:

{"image": "photo1.jpg", "caption": "a man riding a wave on top of a surfboard", "config": "five"}

With --dump_beam, each record also holds the five retrieved captions, the hard-prompt entities, and all 20 candidates with every agent score.

--config five, the default, uses bigG, BLIP, VQA, MBR and ground. It is the better choice for images outside COCO: NoCaps 82.2 against 81.2 CIDEr and Flickr30k 72.3 against 70.7 for the six-agent stack. --config six adds the LM agent and is best on COCO itself.

What runs

1. ViT-bigG/14 retrieves nine captions from the 566,747 COCO training captions; their embeddings ship in assets/.
2. BLIP-ITM keeps five; nouns occurring at least three times in them form the hard prompt.
3. The IFCap COCO decoder, CLIP ViT-B/32 with GPT-2, emits a beam of 20 candidates.
4. The six agents score the beam and the equal-weight z-sum selects the caption.

Stages run over all input images one after another and free their models before the next stage loads, so a single 16 GB GPU is enough: 96 COCO images take about 16 minutes on one Tesla T4, model loading included.

On first use the script downloads the IFCap source at a pinned commit and the coco-indomain-004.pt checkpoint, about 650 MB, from the IFCap authors' GitHub, checksum-verified, into ~/.cache/frama. The other models come from the Hugging Face Hub. Pass --ifcap_root and --ifcap_ckpt to reuse an existing IFCap checkout.

Results in the paper

Official COCO caption toolkit, CIDEr / SPICE, revised manuscript (R1, October 2026).

Benchmark	IFCap top-1	FRAMA, 6 agents	FRAMA, 5 agents
COCO Karpathy test	112.7 / 21.0	120.7 / 22.2	120.5 / 22.2
NoCaps val, overall	70.3 / 9.8	81.2 / 11.0	82.2 / 11.0
Flickr30k Karpathy test	63.7 / 17.0	70.7 / 18.3	72.3 / 18.5

Correction in R1. Releases before October 2026 computed the n-gram document frequencies of the MBR agent on the COCO Karpathy-test reference captions, which lets test statistics into the selection. The MBR agent now uses frequencies from the 566,747 COCO training captions of the captioner (assets/mbr_df_coco_train.json.gz). With the earlier frequencies the six-agent stack scored 120.8 / 81.3 / 70.5 CIDEr and the five-agent stack 120.5 / 82.3 / 72.1. The selected captions of every rule in the revised paper are in predictions_r1/.

Fidelity to the paper

verify_infer.py runs the script on the first 96 COCO Karpathy-test images and compares every stage with the files behind the paper, on one Tesla T4:

Stage	Identical to the paper
bigG top-9 retrieval	93 / 96; on all 5,000 test images 4,977 retrieve the same nine captions, 4,882 in the same order
five captions kept by BLIP-ITM, hard-prompt entities	96 / 96
beam of 20 candidates	79 / 96
agent scores on those beams	bigG, BLIP within 5e-6; VQA, MBR, ground exact; LM within 1.1e-3
selected caption on those beams, both configurations	79 / 79

Order swaps in retrieval are adjacent near-ties and vanish once BLIP-ITM re-sorts the nine captions. The other 17 beams differ through floating-point noise at the decoder input, which beam search amplifies. CLIP ViT-B/32 features recomputed here agree with the IFCap authors' stored features at cosine 0.999996; decoding from the stored features gives 93 of 96 identical beams. A new image has no stored features, so this level of drift is inherent to running the pipeline on new hardware.

Files

File	Content
frama_infer.py	the full pipeline
detector_ground_scorer.py	noun extraction for the grounding agent
verify_infer.py	stage-by-stage check against the paper's artifacts
assets/coco_train_captions.json	the retrieval corpus, COCO Karpathy-train captions
assets/coco_train_captions_bigG14_fp16.npy	their ViT-bigG/14 text embeddings, float16, 1.45 GB
assets/mbr_df_coco_train.json.gz	n-gram document frequencies of the MBR agent, from the COCO training captions
predictions_r1/<benchmark>/<rule>__<split>.json	the selected caption of every rule in the revised paper, with the reference captions, in the IFCap evaluation format
assets/mbr_df_coco_karpathy_test.json.gz	superseded: frequencies from the test references, used before the R1 revision; do not use

The MBR agent weighs n-grams with document frequencies computed over the COCO training captions of the captioner, a fixed text statistic that involves no test data. Its scores equal the released mbr_trainidf_coco.npy of the revised paper exactly; we rechecked this on 40 COCO beams after the change. The end-to-end comparison in the table above was run before the change of frequencies, and verify_infer.py now compares against the R1 files.

License

Code and assets in this repository: MIT. The COCO captions and the statistics derived from them follow the COCO terms of use, CC BY 4.0. IFCap, BLIP, BLIP-2, OpenCLIP and OWLv2 are downloaded from their authors under their own licenses.