--- license: mit pipeline_tag: image-to-text tags: - image-captioning - zero-shot-image-captioning - reranking - training-free - multi-agent datasets: - coco --- # FRAMA: frozen multi-agent adjudication for zero-shot image captioning Inference code and assets for **"A Reranking Reality Check for Zero-Shot Image Captioning: Leakage-Free Oracles and System One Adjudication"**. Code for every table of the paper, with the released beam dumps and agent scores: [github.com/bradduy/FRAMA](https://github.com/bradduy/FRAMA). FRAMA leaves a zero-shot captioner frozen and only decides which of its beam candidates to return. Six frozen agents score each candidate, every score is z-normalised within the beam, and the candidate with the highest equal-weight sum wins. No parameter is trained or tuned. | Agent | Model | Signal | |---|---|---| | LM | IFCap decoder | length-normalised log-likelihood | | bigG | OpenCLIP ViT-bigG/14, LAION-2B | image-caption cosine | | BLIP | `Salesforce/blip-itm-base-coco` | image-text match probability | | VQA | `Salesforce/blip2-flan-t5-xl` | P(yes) to "Does this figure show ...?", the System One verifier | | MBR | none | mean CIDEr-D of the candidate against the other 19 | | ground | `google/owlv2-base-patch16-ensemble` | mean detection score of the caption's nouns | ## Quick start ```bash hf download bradduy/FRAMA --local-dir FRAMA && cd FRAMA pip install -r requirements.txt python frama_infer.py --images photo1.jpg photo2.jpg --out captions.json python frama_infer.py --image_dir my_photos/ --out captions.json --dump_beam ``` Output, one record per image: ```json {"image": "photo1.jpg", "caption": "a man riding a wave on top of a surfboard", "config": "five"} ``` With `--dump_beam`, each record also holds the five retrieved captions, the hard-prompt entities, and all 20 candidates with every agent score. `--config five`, the default, uses bigG, BLIP, VQA, MBR and ground. It is the better choice for images outside COCO: NoCaps 82.2 against 81.2 CIDEr and Flickr30k 72.3 against 70.7 for the six-agent stack. `--config six` adds the LM agent and is best on COCO itself. ## What runs 1. ViT-bigG/14 retrieves nine captions from the 566,747 COCO training captions; their embeddings ship in `assets/`. 2. BLIP-ITM keeps five; nouns occurring at least three times in them form the hard prompt. 3. The IFCap COCO decoder, CLIP ViT-B/32 with GPT-2, emits a beam of 20 candidates. 4. The six agents score the beam and the equal-weight z-sum selects the caption. Stages run over all input images one after another and free their models before the next stage loads, so a single 16 GB GPU is enough: 96 COCO images take about 16 minutes on one Tesla T4, model loading included. On first use the script downloads the IFCap source at a pinned commit and the `coco-indomain-004.pt` checkpoint, about 650 MB, from the [IFCap authors' GitHub](https://github.com/boreng0817/IFCap), checksum-verified, into `~/.cache/frama`. The other models come from the Hugging Face Hub. Pass `--ifcap_root` and `--ifcap_ckpt` to reuse an existing IFCap checkout. ## Results in the paper Official COCO caption toolkit, CIDEr / SPICE, revised manuscript (R1, October 2026). | Benchmark | IFCap top-1 | FRAMA, 6 agents | FRAMA, 5 agents | |---|---|---|---| | COCO Karpathy test | 112.7 / 21.0 | 120.7 / 22.2 | 120.5 / 22.2 | | NoCaps val, overall | 70.3 / 9.8 | 81.2 / 11.0 | 82.2 / 11.0 | | Flickr30k Karpathy test | 63.7 / 17.0 | 70.7 / 18.3 | 72.3 / 18.5 | **Correction in R1.** Releases before October 2026 computed the n-gram document frequencies of the MBR agent on the COCO Karpathy-test reference captions, which lets test statistics into the selection. The MBR agent now uses frequencies from the 566,747 COCO training captions of the captioner (`assets/mbr_df_coco_train.json.gz`). With the earlier frequencies the six-agent stack scored 120.8 / 81.3 / 70.5 CIDEr and the five-agent stack 120.5 / 82.3 / 72.1. The selected captions of every rule in the revised paper are in `predictions_r1/`. ### Fidelity to the paper `verify_infer.py` runs the script on the first 96 COCO Karpathy-test images and compares every stage with the files behind the paper, on one Tesla T4: | Stage | Identical to the paper | |---|---| | bigG top-9 retrieval | 93 / 96; on all 5,000 test images 4,977 retrieve the same nine captions, 4,882 in the same order | | five captions kept by BLIP-ITM, hard-prompt entities | 96 / 96 | | beam of 20 candidates | 79 / 96 | | agent scores on those beams | bigG, BLIP within 5e-6; VQA, MBR, ground exact; LM within 1.1e-3 | | selected caption on those beams, both configurations | 79 / 79 | Order swaps in retrieval are adjacent near-ties and vanish once BLIP-ITM re-sorts the nine captions. The other 17 beams differ through floating-point noise at the decoder input, which beam search amplifies. CLIP ViT-B/32 features recomputed here agree with the IFCap authors' stored features at cosine 0.999996; decoding from the stored features gives 93 of 96 identical beams. A new image has no stored features, so this level of drift is inherent to running the pipeline on new hardware. ## Files | File | Content | |---|---| | `frama_infer.py` | the full pipeline | | `detector_ground_scorer.py` | noun extraction for the grounding agent | | `verify_infer.py` | stage-by-stage check against the paper's artifacts | | `assets/coco_train_captions.json` | the retrieval corpus, COCO Karpathy-train captions | | `assets/coco_train_captions_bigG14_fp16.npy` | their ViT-bigG/14 text embeddings, float16, 1.45 GB | | `assets/mbr_df_coco_train.json.gz` | n-gram document frequencies of the MBR agent, from the COCO training captions | | `predictions_r1//__.json` | the selected caption of every rule in the revised paper, with the reference captions, in the IFCap evaluation format | | `assets/mbr_df_coco_karpathy_test.json.gz` | superseded: frequencies from the test references, used before the R1 revision; do not use | The MBR agent weighs n-grams with document frequencies computed over the COCO training captions of the captioner, a fixed text statistic that involves no test data. Its scores equal the released `mbr_trainidf_coco.npy` of the revised paper exactly; we rechecked this on 40 COCO beams after the change. The end-to-end comparison in the table above was run before the change of frequencies, and `verify_infer.py` now compares against the R1 files. ## License Code and assets in this repository: MIT. The COCO captions and the statistics derived from them follow the COCO terms of use, CC BY 4.0. IFCap, BLIP, BLIP-2, OpenCLIP and OWLv2 are downloaded from their authors under their own licenses.