R1 revision: MBR agent with training-caption document frequencies

Earlier releases computed the MBR document frequencies on the COCO Karpathy-test references. frama_infer.py now uses assets/mbr_df_coco_train.json.gz, computed from the 566,747 COCO training captions; verify_infer.py compares against the R1 files; the README states the correction and the R1 numbers.

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 assets/coco_train_captions.json filter=lfs diff=lfs merge=lfs -text
+predictions_r1/** filter=lfs diff=lfs merge=lfs -text

README.md

@@ -49,7 +49,7 @@ With `--dump_beam`, each record also holds the five retrieved captions, the hard
 and all 20 candidates with every agent score.
 
 `--config five`, the default, uses bigG, BLIP, VQA, MBR and ground. It is the better choice for
-images outside COCO: NoCaps 82.3 against 81.3 CIDEr and Flickr30k 72.1 against 70.5 for the
+images outside COCO: NoCaps 82.2 against 81.2 CIDEr and Flickr30k 72.3 against 70.7 for the
 six-agent stack. `--config six` adds the LM agent and is best on COCO itself.
 
 ## What runs
@@ -71,13 +71,20 @@ checksum-verified, into `~/.cache/frama`. The other models come from the Hugging
 
 ## Results in the paper
 
-Official COCO caption toolkit, CIDEr / SPICE.
+Official COCO caption toolkit, CIDEr / SPICE, revised manuscript (R1, October 2026).
 
 | Benchmark | IFCap top-1 | FRAMA, 6 agents | FRAMA, 5 agents |
 |---|---|---|---|
-| COCO Karpathy test | 112.7 / 21.0 | 120.8 / 22.2 | 120.5 / 22.2 |
-| NoCaps val, overall | 70.3 / 9.8 | 81.3 / 11.0 | 82.3 / 11.0 |
-| Flickr30k Karpathy test | 63.7 / 17.0 | 70.5 / 18.2 | 72.1 / 18.5 |
+| COCO Karpathy test | 112.7 / 21.0 | 120.7 / 22.2 | 120.5 / 22.2 |
+| NoCaps val, overall | 70.3 / 9.8 | 81.2 / 11.0 | 82.2 / 11.0 |
+| Flickr30k Karpathy test | 63.7 / 17.0 | 70.7 / 18.3 | 72.3 / 18.5 |
+
+**Correction in R1.** Releases before October 2026 computed the n-gram document frequencies of the
+MBR agent on the COCO Karpathy-test reference captions, which lets test statistics into the
+selection. The MBR agent now uses frequencies from the 566,747 COCO training captions of the
+captioner (`assets/mbr_df_coco_train.json.gz`). With the earlier frequencies the six-agent stack
+scored 120.8 / 81.3 / 70.5 CIDEr and the five-agent stack 120.5 / 82.3 / 72.1. The selected
+captions of every rule in the revised paper are in `predictions_r1/`.
 
 ### Fidelity to the paper
 
@@ -107,10 +114,15 @@ no stored features, so this level of drift is inherent to running the pipeline o
 | `verify_infer.py` | stage-by-stage check against the paper's artifacts |
 | `assets/coco_train_captions.json` | the retrieval corpus, COCO Karpathy-train captions |
 | `assets/coco_train_captions_bigG14_fp16.npy` | their ViT-bigG/14 text embeddings, float16, 1.45 GB |
-| `assets/mbr_df_coco_karpathy_test.json.gz` | n-gram document frequencies used by the MBR agent |
-
-The MBR agent weighs n-grams with the document frequencies the paper used, computed over the COCO
-Karpathy-test reference captions. They act as a fixed text statistic for any new image.
+| `assets/mbr_df_coco_train.json.gz` | n-gram document frequencies of the MBR agent, from the COCO training captions |
+| `predictions_r1/<benchmark>/<rule>__<split>.json` | the selected caption of every rule in the revised paper, with the reference captions, in the IFCap evaluation format |
+| `assets/mbr_df_coco_karpathy_test.json.gz` | superseded: frequencies from the test references, used before the R1 revision; do not use |
+
+The MBR agent weighs n-grams with document frequencies computed over the COCO training captions of
+the captioner, a fixed text statistic that involves no test data. Its scores equal the released
+`mbr_trainidf_coco.npy` of the revised paper exactly; we rechecked this on 40 COCO beams after the
+change. The end-to-end comparison in the table above was run before the change of frequencies, and
+`verify_infer.py` now compares against the R1 files.
 
 ## License
 

assets/mbr_df_coco_train.json.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43314708f8202580d26a1ca15ed0a436f7e96d1b9646ad72d1d6d7d7026da095
+size 16969177

frama_infer.py

@@ -363,7 +363,7 @@ def _tok(s):
 
 
 class MbrCider:
-    """CIDEr-D between two candidates, with the paper's fixed n-gram document frequencies."""
+    """CIDEr-D between two candidates, with n-gram document frequencies from the training captions."""
 
     def __init__(self, path):
         d = json.load(gzip.open(path, "rt"))
@@ -395,7 +395,9 @@ class MbrCider:
 
 
 def stage_mbr(recs):
-    cider = MbrCider(os.path.join(ASSETS, "mbr_df_coco_karpathy_test.json.gz"))
+    # Document frequencies from the 566,747 COCO training captions of the captioner. Earlier releases
+    # used frequencies computed on the COCO Karpathy-test references, which leaks test information.
+    cider = MbrCider(os.path.join(ASSETS, "mbr_df_coco_train.json.gz"))
     for r in recs:
         v = [cider.vec(_tok(b["caption"])) for b in r["beam"]]
         K = len(v)

verify_infer.py

@@ -3,13 +3,14 @@ COCO Karpathy-test images in retrieval order.
 
   python verify_infer.py --ifcap_root <IFCap checkout with annotations and val2014> \
       --ref <folder with coco_beam_dump.json, features.npz, vqascore_coco.jsonl, ground_coco.jsonl,
-             mbr_self_coco.npy, stack6_all_equal__overall.json, stack5_noLM_blip_bigG_vqa_mbr_ground__overall.json> \
+             mbr_trainidf_coco.npy, stack6_all_equal__overall.json, stack5_noLM_blip_bigG_vqa_mbr_ground__overall.json> \
       --n 96
   python verify_infer.py --ifcap_root ... --retrieval_only --n 5000
 
 Retrieval is compared with caption_coco_image_coco_bigG_9.json, the BLIP-ITM step with
 ..._bigG_blip5.json and its entity file, the beam with the stored dump, every agent with its
-released score file, and the final pick with the paper's prediction files.
+released score file, and the final pick with the paper's prediction files. Since the R1 revision the MBR
+reference is mbr_trainidf_coco.npy and the prediction files are those of predictions_r1/coco/.
 """
 import argparse, json, os
 import numpy as np
@@ -51,7 +52,7 @@ else:
     fpos = {str(x): i for i, x in enumerate(npz["image_names"])}
     vqa = {json.loads(l)["image_name"]: json.loads(l)["vqascore"] for l in open(R("vqascore_coco.jsonl"))}
     gro = {json.loads(l)["image_name"]: json.loads(l) for l in open(R("ground_coco.jsonl"))}
-    mbr = np.load(R("mbr_self_coco.npy"))
+    mbr = np.load(R("mbr_trainidf_coco.npy"))  # R1: MBR document frequencies from the training captions
     picks = {c: {p["image_name"]: p["prediction"] for p in json.load(open(R(f)))}
              for c, f in (("six", "stack6_all_equal__overall.json"), ("five", "stack5_noLM_blip_bigG_vqa_mbr_ground__overall.json"))}