AhmedSSabir commited on
Commit
deddbf6
Β·
verified Β·
1 Parent(s): 6b5c0d8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +95 -80
README.md CHANGED
@@ -26,10 +26,16 @@ comes from existing COCO reference captions.
26
 
27
  **It also improves a recent vision-language model.** Without any retraining, choosing
28
  among Google's [Gemma 4 12B](https://huggingface.co/google/gemma-4-12B-it) nine beam
29
- captions raises CIDEr from 0.9624 to 0.9783 on 5,000 COCO karpathy split
30
- test images, and SPICE and BERTScore. Details and examples are under
 
31
  [Evaluation](#evaluation).
32
 
 
 
 
 
 
33
  ## Training
34
 
35
  ![Two-stage training process](training_process.png)
@@ -51,16 +57,29 @@ head, trained with binary cross-entropy on the visual semantic relatedness datas
51
  policy-gradient objective computed without sampling.
52
  5. Training: 4 epochs, 5,000 steps, 8 images (72 captions) per step. Adam
53
  learning rates are 1e-6 for BERT, 1e-5 for the CNN head and 1e-2 for `b, w`;
54
- seed 42. It took 7 mins on one RTX A5000.
55
- <!-- 6. Selection is on 734 held-out validation images (best validation CIDEr, epoch 4).
56
- The test sets were run once, after the choice was locked in `protocol_lock.json`. -->
57
 
58
- <!-- Learned values: beam bonus `b = [2.58, -0.22, -2.62, -3.49, -4.01, -3.78, -5.26,
59
  -5.89, -6.85]` and `w = 3.20`. The model keeps the first beam unless BERT-CNN is
60
- confident another candidate is better. It replaces it for 39% of images.
61
- -->
 
62
  ## Usage
63
 
 
 
 
 
 
 
 
 
 
 
 
 
64
  ```python
65
  from huggingface_hub import snapshot_download
66
  import sys
@@ -71,18 +90,19 @@ from inference import CaptionReranker
71
 
72
  reranker = CaptionReranker(path, device="cpu") # "cuda" for a GPU
73
  beams = [ # exactly nine captions, in beam-search order (best beam first)
74
- "a white toilet sitting next to a white bath tub",
75
- "a bathroom with a sink and a toilet",
76
- "a bathroom with two sinks and a toilet",
77
- "a white bath tub sitting next to a toilet",
78
- "a bath room with a toilet and a bath tub",
79
- "a white bath tub sitting next to a white toilet",
80
- "a white toilet sitting next to a white sink",
81
- "a bath room with a toilet a bath tub",
82
- "a bath room with a toilet a sink and a mirror",
83
  ]
84
- best, scores = reranker.rerank("washbasin bathtub sink", beams) # visual context from the image
85
- print(beams[best]) # -> a bathroom with a sink and a toilet
 
86
  ```
87
 
88
  The beam bonus means `rerank` needs the nine captions **in the generator's beam
@@ -91,37 +111,16 @@ BERT-CNN scores.
91
 
92
  ## Evaluation
93
 
94
- Paper benchmark: the paper's 9-beam candidates for **5,000 Karpathy test
95
- images** (45,000 captions). All rows were scored with the same scorers. BERTScore
96
- in this setup reads about 0.004 higher than the paper's, so compare B-S within
97
- this table.
98
-
99
- | Method | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE | BERTScore |
100
- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
101
- | Best Beam (beam 1) | **0.3509** | 0.2741 | 0.5573 | 1.1150 | 0.2051 | 0.9406 |
102
- | Retrained PyTorch BERT-CNN (stage I only) | 0.3258 | 0.2716 | 0.5486 | 1.0855 | 0.2056 | 0.9393 |
103
- | **This model** | 0.3504 | **0.2794** | **0.5625** | **1.1475** | **0.2119** | **0.9413** |
104
- <!-- | Paper, best reported row | 0.352 | 0.275 | 0.560 | 1.131 | 0.208 | 0.9366* | -->
105
-
106
- The BERT-CNN row above chooses among all nine candidates. The two-candidate setting is shown below.
107
-
108
- **Ablations on the same benchmark (CIDEr):**
109
-
110
- | Variant | CIDEr |
111
- | --- | ---: |
112
- | Beam bonus only (BERT-CNN frozen, w = 0) | 1.1150 |
113
- | No beam bonus | 1.1202 |
114
- | Sentence-similarity reward instead of CIDEr | 1.1236 |
115
- | CIDEr and similarity mix | 1.1341 |
116
- | CIDEr reward plus relatedness BCE loss | 1.1384 |
117
- | **CIDEr reward (this model)** | **1.1475** |
118
 
119
  **In-domain test:** BLIP 9-beam candidates for the same 5,000 images.
120
 
121
  | Method | CIDEr |
122
  | --- | ---: |
123
  | Best Beam | 1.0632 |
124
- | This model | **1.1470** |
125
 
126
  **Gemma 4 12B candidates.** `google/gemma-4-12B-it`, prompted for a short COCO-style
127
  caption with nine beams, on the same 5,000 images. This model was not retrained on them.
@@ -131,6 +130,8 @@ caption with nine beams, on the same 5,000 images. This model was not retrained
131
  | Gemma 4 12B, beam 1 of 9 | 0.2441 | 0.2739 | 0.5288 | 0.9624 | 0.2248 | 0.9438 |
132
  | **Gemma 4 12B + this model** | **0.2458** | **0.2766** | **0.5314** | **0.9783** | **0.2267** | **0.9448** |
133
 
 
 
134
 
135
  Gemma 4 examples, CIDEr per caption. The model replaces Gemma's top caption on 1,298 of
136
  5,000 images: CIDEr improves on 751, drops on 543 and ties on 4. The rows below are the
@@ -141,59 +142,73 @@ first three improvements and the first drop in a random draw from those images.
141
  | COCO 27585 | wall clock analog tree | A large black and gold clock stands against a blue sky. (0.70) | **A large ornate clock stands in front of a building. (1.25)** |
142
  | COCO 320370 | croquet ball soccer tree | A man in a red shirt stands on a green field. (0.61) | **A man in a red shirt stands on a green lawn. (0.70)** |
143
  | COCO 300233 | mixing bowl corn food | Ingredients and a food processor are arranged on a gray surface. (0.40) | **Ingredients and a food processor are laid out on a table. (0.47)** |
144
- | COCO 419228 (failure) | ice bear polar | **A white polar bear walks on rocks in an enclosure. (2.54)** | A large white polar bear walks on a rocky surface. (1.85)
145
 
 
146
 
147
- **Summary: gain over each captioner's own top beam (CIDEr).**
 
 
 
 
 
 
 
148
 
149
- | Captioner | Top beam | + This model | Gain [95% CI] |
150
- | --- | ---: | ---: | --- |
151
- | BLIP-base (training captioner) | 1.0632 | 1.1470 | +0.0838 [+0.073, +0.095] |
152
- | ViLBERT (paper's candidates) | 1.1150 | 1.1475 | +0.0325 [+0.021, +0.044] |
153
- | Qwen3-VL-8B | 0.5723 | 0.5927 | +0.0204 [+0.016, +0.025] |
154
- | Gemma 4 12B | 0.9624 | 0.9783 | +0.0159 [+0.009, +0.023] |
155
- | BLIP-large | 1.1509 | 1.1638 | +0.0129 [+0.006, +0.020] |
156
- | Florence-2-large-ft | 1.4099 | 1.4203 | +0.0104 [+0.002, +0.020] |
157
- | BLIP-2 OPT-2.7B COCO | 1.4034 | 1.3960 | -0.0074 [-0.015, +0.0002] (not significant) |
158
 
159
  The gain shrinks as the captioner gets stronger. This model was not retrained on any
160
  captioner in this table except BLIP-base.
161
 
162
- **Other captioners, nine beams each (top beam -> this model).**
163
-
164
- | Captioner | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE |
165
- | --- | ---: | ---: | ---: | ---: | ---: |
166
- | Qwen3-VL-8B | 0.0669 -> 0.0710 | 0.1872 -> 0.1910 | 0.3357 -> 0.3404 | 0.5723 -> 0.5927 | 0.1582 -> 0.1613 |
167
- | BLIP-large | 0.3004 -> 0.3035 | 0.2993 -> 0.2996 | 0.5378 -> 0.5389 | 1.1509 -> 1.1638 | 0.2381 -> 0.2395 |
168
- | Florence-2-large-ft | 0.4079 -> 0.4090 | 0.3214 -> 0.3220 | 0.6121 -> 0.6143 | 1.4099 -> 1.4203 | 0.2498 -> 0.2512 |
169
- | BLIP-2 OPT-2.7B COCO | 0.3952 -> 0.3919 | 0.3256 -> 0.3251 | 0.6061 -> 0.6056 | 1.4034 -> 1.3960 | 0.2630 -> 0.2628 |
170
-
171
-
172
-
173
- <!-- **Summary: gain over each captioner's own top beam (CIDEr).**
174
-
175
- | Captioner | Top beam | + This model | Gain [95% CI] |
176
- | --- | ---: | ---: | --- |
177
- | BLIP-base (training captioner) | 1.0632 | 1.1470 | +0.0838 [+0.073, +0.095] |
178
- | ViLBERT (paper's candidates) | 1.1150 | 1.1475 | +0.0325 [+0.021, +0.044] |
179
- | Gemma 4 12B | 0.9624 | 0.9783 | +0.0159 [+0.009, +0.023] |
180
- -->
181
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
182
 
183
  ## Limitations
184
 
185
-
186
- **Use it as a reranker, not a relatedness classifier.** Reward training shifts
187
  the scores upward. On the paper's 41,849 held-out relatedness pairs, accuracy at
188
  the 0.5 cutoff falls from 0.893 to 0.631, and ROC-AUC from 0.962 to 0.937. For
189
- relatedness probabilities, use the stage I like [ModrenBERT-CNN](https://huggingface.co/AhmedSSabir/ModernBERT-CNN-Caption-Reranker).
190
-
191
  - Agreement with Flickr8K expert ratings dropped slightly: Spearman 0.365, against
192
  0.380 for the stage I model on 5,822 pairs.
193
- - A reranker is a cheap way to improve an existing captioner, but it can only choose among its candidates, so it does not replace a stronger captioner.
194
- - On CIDEr rewards exact wording ("hotdog" vs "hot dog"). The reranker cannot fix errors shared by all nine candidates.
 
 
 
 
 
 
195
  - The best validation epoch was the last epoch, so longer training was not explored.
196
 
 
 
197
 
198
  ## Citation
199
 
 
26
 
27
  **It also improves a recent vision-language model.** Without any retraining, choosing
28
  among Google's [Gemma 4 12B](https://huggingface.co/google/gemma-4-12B-it) nine beam
29
+ captions raises CIDEr from 0.9624 to 0.9783 (+0.0159 [+0.009, +0.023]) on 5,000 COCO
30
+ test images, and SPICE and BERTScore rise too. The gain varies by captioner and is
31
+ absent on BLIP-2 OPT-2.7B COCO. Details and examples are under
32
  [Evaluation](#evaluation).
33
 
34
+ | Method | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE | BERTScore |
35
+ | --- | ---: | ---: | ---: | ---: | ---: | ---: |
36
+ | Baseline: Gemma 4 12B, top beam | 0.2441 | 0.2739 | 0.5288 | 0.9624 | 0.2248 | 0.9438 |
37
+ | **Gemma 4 12B + this model** | **0.2458** | **0.2766** | **0.5314** | **0.9783** | **0.2267** | **0.9448** |
38
+
39
  ## Training
40
 
41
  ![Two-stage training process](training_process.png)
 
57
  policy-gradient objective computed without sampling.
58
  5. Training: 4 epochs, 5,000 steps, 8 images (72 captions) per step. Adam
59
  learning rates are 1e-6 for BERT, 1e-5 for the CNN head and 1e-2 for `b, w`;
60
+ seed 42. It took 407 s on one RTX A5000.
61
+ 6. Selection is on 734 held-out validation images (best validation CIDEr, epoch 4).
62
+ The test sets were run once, after the choice was locked in `protocol_lock.json`.
63
 
64
+ Learned values: beam bonus `b = [2.58, -0.22, -2.62, -3.49, -4.01, -3.78, -5.26,
65
  -5.89, -6.85]` and `w = 3.20`. The model keeps the first beam unless BERT-CNN is
66
+ confident another candidate is better. On Gemma 4 12B candidates it replaces the first
67
+ beam on 26% of images.
68
+
69
  ## Usage
70
 
71
+ Example, COCO image 27585 with Gemma 4 12B beams and visual context `wall clock analog tree`.
72
+ The first beam says "A large black and gold clock stands against a blue sky", and the
73
+ model picks "A large ornate clock stands in front of a building".
74
+
75
+ Second example, COCO image 300233 with visual context `mixing bowl corn food`. The first
76
+ beam says "Ingredients and a food processor are arranged on a gray surface", and the
77
+ model picks "Ingredients and a food processor are laid out on a table".
78
+
79
+ The model reads the **visual context** (text labels extracted from the image) and
80
+ the captions. It does not read the image itself, and this repository does not
81
+ include the label extractor, so you supply the labels.
82
+
83
  ```python
84
  from huggingface_hub import snapshot_download
85
  import sys
 
90
 
91
  reranker = CaptionReranker(path, device="cpu") # "cuda" for a GPU
92
  beams = [ # exactly nine captions, in beam-search order (best beam first)
93
+ "A large black and gold clock stands against a blue sky.",
94
+ "A black and gold street clock stands against a blue sky.",
95
+ "A large black and gold clock stands under a blue sky.",
96
+ "A large black and gold street clock stands against a blue sky.",
97
+ "A large black and gold street clock stands under a blue sky.",
98
+ "A black and gold street clock stands under a blue sky.",
99
+ "A large ornate street clock stands against a blue sky.",
100
+ "A large ornate clock stands in front of a building.",
101
+ "A large black and gold clock stands in a city.",
102
  ]
103
+ visual_context = "wall clock analog tree" # text labels extracted from the image
104
+ best, scores = reranker.rerank(visual_context, beams)
105
+ print(beams[best]) # -> A large ornate clock stands in front of a building.
106
  ```
107
 
108
  The beam bonus means `rerank` needs the nine captions **in the generator's beam
 
111
 
112
  ## Evaluation
113
 
114
+ Every captioner below gives nine beam-search candidates per image on the same **5,000
115
+ Karpathy test images**, and is compared with its own top beam. All rows use the same
116
+ scorers.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
117
 
118
  **In-domain test:** BLIP 9-beam candidates for the same 5,000 images.
119
 
120
  | Method | CIDEr |
121
  | --- | ---: |
122
  | Best Beam | 1.0632 |
123
+ | This model | **1.1470** (+0.0838 [+0.073, +0.095]) |
124
 
125
  **Gemma 4 12B candidates.** `google/gemma-4-12B-it`, prompted for a short COCO-style
126
  caption with nine beams, on the same 5,000 images. This model was not retrained on them.
 
130
  | Gemma 4 12B, beam 1 of 9 | 0.2441 | 0.2739 | 0.5288 | 0.9624 | 0.2248 | 0.9438 |
131
  | **Gemma 4 12B + this model** | **0.2458** | **0.2766** | **0.5314** | **0.9783** | **0.2267** | **0.9448** |
132
 
133
+ Gains: CIDEr +0.0159 [+0.009, +0.023], SPICE +0.0019 [+0.001, +0.003], BERTScore
134
+ +0.0009 [+0.0007, +0.0012].
135
 
136
  Gemma 4 examples, CIDEr per caption. The model replaces Gemma's top caption on 1,298 of
137
  5,000 images: CIDEr improves on 751, drops on 543 and ties on 4. The rows below are the
 
142
  | COCO 27585 | wall clock analog tree | A large black and gold clock stands against a blue sky. (0.70) | **A large ornate clock stands in front of a building. (1.25)** |
143
  | COCO 320370 | croquet ball soccer tree | A man in a red shirt stands on a green field. (0.61) | **A man in a red shirt stands on a green lawn. (0.70)** |
144
  | COCO 300233 | mixing bowl corn food | Ingredients and a food processor are arranged on a gray surface. (0.40) | **Ingredients and a food processor are laid out on a table. (0.47)** |
145
+ | COCO 419228 (failure) | ice bear polar | **A white polar bear walks on rocks in an enclosure. (2.54)** | A large white polar bear walks on a rocky surface. (1.85) |
146
 
147
+ **Summary: CIDEr for each captioner's own top beam and with this model.**
148
 
149
+ | Captioner | Top beam | + This model |
150
+ | --- | ---: | ---: |
151
+ | BLIP-base (training captioner) | 1.0632 | **1.1470**\* |
152
+ | Qwen3-VL-8B | 0.5723 | **0.5927**\* |
153
+ | Gemma 4 12B | 0.9624 | **0.9783**\* |
154
+ | BLIP-large | 1.1509 | **1.1638**\* |
155
+ | Florence-2-large-ft | 1.4099 | **1.4203**\* |
156
+ | BLIP-2 OPT-2.7B COCO | **1.4034** | 1.3960 |
157
 
158
+ \* Significantly different from the top beam (paired bootstrap 95% interval over images excludes zero). Best result in bold.
 
 
 
 
 
 
 
 
159
 
160
  The gain shrinks as the captioner gets stronger. This model was not retrained on any
161
  captioner in this table except BLIP-base.
162
 
163
+ **All metrics, nine beams each.** Bold marks the better result for each captioner.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
164
 
165
+ | System | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE | BERTScore |
166
+ | --- | ---: | ---: | ---: | ---: | ---: | ---: |
167
+ | Qwen3-VL-8B, Best Beam | 0.0669 | 0.1872 | 0.3357 | 0.5723 | 0.1582 | - |
168
+ | Qwen3-VL-8B + This model | **0.0710** | **0.1910** | **0.3404** | **0.5927** | **0.1613** | - |
169
+ | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ |
170
+ | Gemma 4 12B, Best Beam | 0.2441 | 0.2739 | 0.5288 | 0.9624 | 0.2248 | 0.9438 |
171
+ | Gemma 4 12B + This model | **0.2458** | **0.2766** | **0.5314** | **0.9783** | **0.2267** | **0.9448** |
172
+ | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ |
173
+ | BLIP-large, Best Beam | 0.3004 | 0.2993 | 0.5378 | 1.1509 | 0.2381 | - |
174
+ | BLIP-large + This model | **0.3035** | **0.2996** | **0.5389** | **1.1638** | **0.2395** | - |
175
+ | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ |
176
+ | Florence-2-large-ft, Best Beam | 0.4079 | 0.3214 | 0.6121 | 1.4099 | 0.2498 | - |
177
+ | Florence-2-large-ft + This model | **0.4090** | **0.3220** | **0.6143** | **1.4203** | **0.2512** | - |
178
+ | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ | ━━━━━━━━ |
179
+ | BLIP-2 OPT-2.7B COCO, Best Beam | **0.3952** | **0.3256** | **0.6061** | **1.4034** | **0.2630** | - |
180
+ | BLIP-2 OPT-2.7B COCO + This model | 0.3919 | 0.3251 | 0.6056 | 1.3960 | 0.2628 | - |
181
+
182
+ BERTScore was not computed for Qwen3-VL-8B, BLIP-large, Florence-2 or BLIP-2. Forced nine-beam decoding changes the captioner itself: Florence-2's top beam here
183
+ (1.4099) is below its default three-beam output (1.4449), so reranking its nine beams
184
+ does not beat that default. Florence-2's nine beams also repeat, about six distinct
185
+ captions per image.
186
+
187
+ **CIDEr and factual accuracy can differ.** CIDEr measures overlap with human wording,
188
+ so a higher score does not always mean a more accurate caption. Human labeling of the
189
+ visual context is too expensive at this scale, so the labels are predicted automatically
190
+ and can be noisy.
191
 
192
  ## Limitations
193
 
194
+ - **Use it as a reranker, not a relatedness classifier.** Reward training shifts
 
195
  the scores upward. On the paper's 41,849 held-out relatedness pairs, accuracy at
196
  the 0.5 cutoff falls from 0.893 to 0.631, and ROC-AUC from 0.962 to 0.937. For
197
+ relatedness probabilities, use the stage I BERT-CNN.
 
198
  - Agreement with Flickr8K expert ratings dropped slightly: Spearman 0.365, against
199
  0.380 for the stage I model on 5,822 pairs.
200
+ - The beam bonus was learned for 9-beam candidates. Other beam widths may need
201
+ retraining. It was trained on BLIP candidates; gains carry over to other captioners
202
+ but are smaller there, and absent on BLIP-2 OPT-2.7B COCO.
203
+ - A reranker is a cheap way to improve an existing captioner, but it can only choose
204
+ among its candidates, so it does not replace a stronger captioner. On the same images,
205
+ COCO-tuned captioners score higher on their own
206
+ (single caption, CIDEr: Florence-2-large-ft 1.445, BLIP-2 OPT-2.7B COCO 1.369). On their
207
+ nine-beam candidates this model adds a small gain for Florence-2 and none for BLIP-2.
208
  - The best validation epoch was the last epoch, so longer training was not explored.
209
 
210
+ The repository includes inference-only weights, the tokenizer, model and inference
211
+ code, training settings, and evaluation results (`evaluation/`).
212
 
213
  ## Citation
214