| --- |
| license: mit |
| pipeline_tag: image-to-text |
| tags: |
| - image-captioning |
| - zero-shot-image-captioning |
| - reranking |
| - training-free |
| - multi-agent |
| datasets: |
| - coco |
| --- |
| |
| # FRAMA: frozen multi-agent adjudication for zero-shot image captioning |
|
|
| Inference code and assets for **"A Reranking Reality Check for Zero-Shot Image Captioning: |
| Leakage-Free Oracles and System One Adjudication"**. Code for every table of the paper, with the |
| released beam dumps and agent scores: [github.com/bradduy/FRAMA](https://github.com/bradduy/FRAMA). |
|
|
| FRAMA leaves a zero-shot captioner frozen and only decides which of its beam candidates to return. |
| Six frozen agents score each candidate, every score is z-normalised within the beam, and the |
| candidate with the highest equal-weight sum wins. No parameter is trained or tuned. |
|
|
| | Agent | Model | Signal | |
| |---|---|---| |
| | LM | IFCap decoder | length-normalised log-likelihood | |
| | bigG | OpenCLIP ViT-bigG/14, LAION-2B | image-caption cosine | |
| | BLIP | `Salesforce/blip-itm-base-coco` | image-text match probability | |
| | VQA | `Salesforce/blip2-flan-t5-xl` | P(yes) to "Does this figure show ...?", the System One verifier | |
| | MBR | none | mean CIDEr-D of the candidate against the other 19 | |
| | ground | `google/owlv2-base-patch16-ensemble` | mean detection score of the caption's nouns | |
|
|
| ## Quick start |
|
|
| ```bash |
| hf download bradduy/FRAMA --local-dir FRAMA && cd FRAMA |
| pip install -r requirements.txt |
| python frama_infer.py --images photo1.jpg photo2.jpg --out captions.json |
| python frama_infer.py --image_dir my_photos/ --out captions.json --dump_beam |
| ``` |
|
|
| Output, one record per image: |
|
|
| ```json |
| {"image": "photo1.jpg", "caption": "a man riding a wave on top of a surfboard", "config": "five"} |
| ``` |
|
|
| With `--dump_beam`, each record also holds the five retrieved captions, the hard-prompt entities, |
| and all 20 candidates with every agent score. |
|
|
| `--config five`, the default, uses bigG, BLIP, VQA, MBR and ground. It is the better choice for |
| images outside COCO: NoCaps 82.2 against 81.2 CIDEr and Flickr30k 72.3 against 70.7 for the |
| six-agent stack. `--config six` adds the LM agent and is best on COCO itself. |
|
|
| ## What runs |
|
|
| 1. ViT-bigG/14 retrieves nine captions from the 566,747 COCO training captions; their embeddings |
| ship in `assets/`. |
| 2. BLIP-ITM keeps five; nouns occurring at least three times in them form the hard prompt. |
| 3. The IFCap COCO decoder, CLIP ViT-B/32 with GPT-2, emits a beam of 20 candidates. |
| 4. The six agents score the beam and the equal-weight z-sum selects the caption. |
|
|
| Stages run over all input images one after another and free their models before the next stage |
| loads, so a single 16 GB GPU is enough: 96 COCO images take about 16 minutes on one Tesla T4, model |
| loading included. |
|
|
| On first use the script downloads the IFCap source at a pinned commit and the `coco-indomain-004.pt` |
| checkpoint, about 650 MB, from the [IFCap authors' GitHub](https://github.com/boreng0817/IFCap), |
| checksum-verified, into `~/.cache/frama`. The other models come from the Hugging Face Hub. Pass |
| `--ifcap_root` and `--ifcap_ckpt` to reuse an existing IFCap checkout. |
|
|
| ## Results in the paper |
|
|
| Official COCO caption toolkit, CIDEr / SPICE, revised manuscript (R1, October 2026). |
|
|
| | Benchmark | IFCap top-1 | FRAMA, 6 agents | FRAMA, 5 agents | |
| |---|---|---|---| |
| | COCO Karpathy test | 112.7 / 21.0 | 120.7 / 22.2 | 120.5 / 22.2 | |
| | NoCaps val, overall | 70.3 / 9.8 | 81.2 / 11.0 | 82.2 / 11.0 | |
| | Flickr30k Karpathy test | 63.7 / 17.0 | 70.7 / 18.3 | 72.3 / 18.5 | |
|
|
| **Correction in R1.** Releases before October 2026 computed the n-gram document frequencies of the |
| MBR agent on the COCO Karpathy-test reference captions, which lets test statistics into the |
| selection. The MBR agent now uses frequencies from the 566,747 COCO training captions of the |
| captioner (`assets/mbr_df_coco_train.json.gz`). With the earlier frequencies the six-agent stack |
| scored 120.8 / 81.3 / 70.5 CIDEr and the five-agent stack 120.5 / 82.3 / 72.1. The selected |
| captions of every rule in the revised paper are in `predictions_r1/`. |
|
|
| ### Fidelity to the paper |
|
|
| `verify_infer.py` runs the script on the first 96 COCO Karpathy-test images and compares every |
| stage with the files behind the paper, on one Tesla T4: |
|
|
| | Stage | Identical to the paper | |
| |---|---| |
| | bigG top-9 retrieval | 93 / 96; on all 5,000 test images 4,977 retrieve the same nine captions, 4,882 in the same order | |
| | five captions kept by BLIP-ITM, hard-prompt entities | 96 / 96 | |
| | beam of 20 candidates | 79 / 96 | |
| | agent scores on those beams | bigG, BLIP within 5e-6; VQA, MBR, ground exact; LM within 1.1e-3 | |
| | selected caption on those beams, both configurations | 79 / 79 | |
|
|
| Order swaps in retrieval are adjacent near-ties and vanish once BLIP-ITM re-sorts the nine captions. |
| The other 17 beams differ through floating-point noise at the decoder input, which beam search |
| amplifies. CLIP ViT-B/32 features recomputed here agree with the IFCap authors' stored features at |
| cosine 0.999996; decoding from the stored features gives 93 of 96 identical beams. A new image has |
| no stored features, so this level of drift is inherent to running the pipeline on new hardware. |
|
|
| ## Files |
|
|
| | File | Content | |
| |---|---| |
| | `frama_infer.py` | the full pipeline | |
| | `detector_ground_scorer.py` | noun extraction for the grounding agent | |
| | `verify_infer.py` | stage-by-stage check against the paper's artifacts | |
| | `assets/coco_train_captions.json` | the retrieval corpus, COCO Karpathy-train captions | |
| | `assets/coco_train_captions_bigG14_fp16.npy` | their ViT-bigG/14 text embeddings, float16, 1.45 GB | |
| | `assets/mbr_df_coco_train.json.gz` | n-gram document frequencies of the MBR agent, from the COCO training captions | |
| | `predictions_r1/<benchmark>/<rule>__<split>.json` | the selected caption of every rule in the revised paper, with the reference captions, in the IFCap evaluation format | |
| | `assets/mbr_df_coco_karpathy_test.json.gz` | superseded: frequencies from the test references, used before the R1 revision; do not use | |
|
|
| The MBR agent weighs n-grams with document frequencies computed over the COCO training captions of |
| the captioner, a fixed text statistic that involves no test data. Its scores equal the released |
| `mbr_trainidf_coco.npy` of the revised paper exactly; we rechecked this on 40 COCO beams after the |
| change. The end-to-end comparison in the table above was run before the change of frequencies, and |
| `verify_infer.py` now compares against the R1 files. |
|
|
| ## License |
|
|
| Code and assets in this repository: MIT. The COCO captions and the statistics derived from them |
| follow the COCO terms of use, CC BY 4.0. IFCap, BLIP, BLIP-2, OpenCLIP and OWLv2 are downloaded |
| from their authors under their own licenses. |
|
|
