Update README.md
Browse files
README.md
CHANGED
|
@@ -26,10 +26,16 @@ comes from existing COCO reference captions.
|
|
| 26 |
|
| 27 |
**It also improves a recent vision-language model.** Without any retraining, choosing
|
| 28 |
among Google's [Gemma 4 12B](https://huggingface.co/google/gemma-4-12B-it) nine beam
|
| 29 |
-
captions raises CIDEr from 0.9624 to 0.9783 on 5,000 COCO
|
| 30 |
-
test images, and SPICE and BERTScore.
|
|
|
|
| 31 |
[Evaluation](#evaluation).
|
| 32 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
## Training
|
| 34 |
|
| 35 |

|
|
@@ -51,16 +57,29 @@ head, trained with binary cross-entropy on the visual semantic relatedness datas
|
|
| 51 |
policy-gradient objective computed without sampling.
|
| 52 |
5. Training: 4 epochs, 5,000 steps, 8 images (72 captions) per step. Adam
|
| 53 |
learning rates are 1e-6 for BERT, 1e-5 for the CNN head and 1e-2 for `b, w`;
|
| 54 |
-
seed 42. It took
|
| 55 |
-
|
| 56 |
-
The test sets were run once, after the choice was locked in `protocol_lock.json`.
|
| 57 |
|
| 58 |
-
|
| 59 |
-5.89, -6.85]` and `w = 3.20`. The model keeps the first beam unless BERT-CNN is
|
| 60 |
-
confident another candidate is better.
|
| 61 |
-
|
|
|
|
| 62 |
## Usage
|
| 63 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
```python
|
| 65 |
from huggingface_hub import snapshot_download
|
| 66 |
import sys
|
|
@@ -71,18 +90,19 @@ from inference import CaptionReranker
|
|
| 71 |
|
| 72 |
reranker = CaptionReranker(path, device="cpu") # "cuda" for a GPU
|
| 73 |
beams = [ # exactly nine captions, in beam-search order (best beam first)
|
| 74 |
-
"
|
| 75 |
-
"
|
| 76 |
-
"
|
| 77 |
-
"
|
| 78 |
-
"
|
| 79 |
-
"
|
| 80 |
-
"
|
| 81 |
-
"
|
| 82 |
-
"
|
| 83 |
]
|
| 84 |
-
|
| 85 |
-
|
|
|
|
| 86 |
```
|
| 87 |
|
| 88 |
The beam bonus means `rerank` needs the nine captions **in the generator's beam
|
|
@@ -91,37 +111,16 @@ BERT-CNN scores.
|
|
| 91 |
|
| 92 |
## Evaluation
|
| 93 |
|
| 94 |
-
|
| 95 |
-
images**
|
| 96 |
-
|
| 97 |
-
this table.
|
| 98 |
-
|
| 99 |
-
| Method | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE | BERTScore |
|
| 100 |
-
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 101 |
-
| Best Beam (beam 1) | **0.3509** | 0.2741 | 0.5573 | 1.1150 | 0.2051 | 0.9406 |
|
| 102 |
-
| Retrained PyTorch BERT-CNN (stage I only) | 0.3258 | 0.2716 | 0.5486 | 1.0855 | 0.2056 | 0.9393 |
|
| 103 |
-
| **This model** | 0.3504 | **0.2794** | **0.5625** | **1.1475** | **0.2119** | **0.9413** |
|
| 104 |
-
<!-- | Paper, best reported row | 0.352 | 0.275 | 0.560 | 1.131 | 0.208 | 0.9366* | -->
|
| 105 |
-
|
| 106 |
-
The BERT-CNN row above chooses among all nine candidates. The two-candidate setting is shown below.
|
| 107 |
-
|
| 108 |
-
**Ablations on the same benchmark (CIDEr):**
|
| 109 |
-
|
| 110 |
-
| Variant | CIDEr |
|
| 111 |
-
| --- | ---: |
|
| 112 |
-
| Beam bonus only (BERT-CNN frozen, w = 0) | 1.1150 |
|
| 113 |
-
| No beam bonus | 1.1202 |
|
| 114 |
-
| Sentence-similarity reward instead of CIDEr | 1.1236 |
|
| 115 |
-
| CIDEr and similarity mix | 1.1341 |
|
| 116 |
-
| CIDEr reward plus relatedness BCE loss | 1.1384 |
|
| 117 |
-
| **CIDEr reward (this model)** | **1.1475** |
|
| 118 |
|
| 119 |
**In-domain test:** BLIP 9-beam candidates for the same 5,000 images.
|
| 120 |
|
| 121 |
| Method | CIDEr |
|
| 122 |
| --- | ---: |
|
| 123 |
| Best Beam | 1.0632 |
|
| 124 |
-
| This model | **1.1470** |
|
| 125 |
|
| 126 |
**Gemma 4 12B candidates.** `google/gemma-4-12B-it`, prompted for a short COCO-style
|
| 127 |
caption with nine beams, on the same 5,000 images. This model was not retrained on them.
|
|
@@ -131,6 +130,8 @@ caption with nine beams, on the same 5,000 images. This model was not retrained
|
|
| 131 |
| Gemma 4 12B, beam 1 of 9 | 0.2441 | 0.2739 | 0.5288 | 0.9624 | 0.2248 | 0.9438 |
|
| 132 |
| **Gemma 4 12B + this model** | **0.2458** | **0.2766** | **0.5314** | **0.9783** | **0.2267** | **0.9448** |
|
| 133 |
|
|
|
|
|
|
|
| 134 |
|
| 135 |
Gemma 4 examples, CIDEr per caption. The model replaces Gemma's top caption on 1,298 of
|
| 136 |
5,000 images: CIDEr improves on 751, drops on 543 and ties on 4. The rows below are the
|
|
@@ -141,59 +142,73 @@ first three improvements and the first drop in a random draw from those images.
|
|
| 141 |
| COCO 27585 | wall clock analog tree | A large black and gold clock stands against a blue sky. (0.70) | **A large ornate clock stands in front of a building. (1.25)** |
|
| 142 |
| COCO 320370 | croquet ball soccer tree | A man in a red shirt stands on a green field. (0.61) | **A man in a red shirt stands on a green lawn. (0.70)** |
|
| 143 |
| COCO 300233 | mixing bowl corn food | Ingredients and a food processor are arranged on a gray surface. (0.40) | **Ingredients and a food processor are laid out on a table. (0.47)** |
|
| 144 |
-
| COCO 419228 (failure) | ice bear polar | **A white polar bear walks on rocks in an enclosure. (2.54)** | A large white polar bear walks on a rocky surface. (1.85)
|
| 145 |
|
|
|
|
| 146 |
|
| 147 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
|
| 149 |
-
|
| 150 |
-
| --- | ---: | ---: | --- |
|
| 151 |
-
| BLIP-base (training captioner) | 1.0632 | 1.1470 | +0.0838 [+0.073, +0.095] |
|
| 152 |
-
| ViLBERT (paper's candidates) | 1.1150 | 1.1475 | +0.0325 [+0.021, +0.044] |
|
| 153 |
-
| Qwen3-VL-8B | 0.5723 | 0.5927 | +0.0204 [+0.016, +0.025] |
|
| 154 |
-
| Gemma 4 12B | 0.9624 | 0.9783 | +0.0159 [+0.009, +0.023] |
|
| 155 |
-
| BLIP-large | 1.1509 | 1.1638 | +0.0129 [+0.006, +0.020] |
|
| 156 |
-
| Florence-2-large-ft | 1.4099 | 1.4203 | +0.0104 [+0.002, +0.020] |
|
| 157 |
-
| BLIP-2 OPT-2.7B COCO | 1.4034 | 1.3960 | -0.0074 [-0.015, +0.0002] (not significant) |
|
| 158 |
|
| 159 |
The gain shrinks as the captioner gets stronger. This model was not retrained on any
|
| 160 |
captioner in this table except BLIP-base.
|
| 161 |
|
| 162 |
-
**
|
| 163 |
-
|
| 164 |
-
| Captioner | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE |
|
| 165 |
-
| --- | ---: | ---: | ---: | ---: | ---: |
|
| 166 |
-
| Qwen3-VL-8B | 0.0669 -> 0.0710 | 0.1872 -> 0.1910 | 0.3357 -> 0.3404 | 0.5723 -> 0.5927 | 0.1582 -> 0.1613 |
|
| 167 |
-
| BLIP-large | 0.3004 -> 0.3035 | 0.2993 -> 0.2996 | 0.5378 -> 0.5389 | 1.1509 -> 1.1638 | 0.2381 -> 0.2395 |
|
| 168 |
-
| Florence-2-large-ft | 0.4079 -> 0.4090 | 0.3214 -> 0.3220 | 0.6121 -> 0.6143 | 1.4099 -> 1.4203 | 0.2498 -> 0.2512 |
|
| 169 |
-
| BLIP-2 OPT-2.7B COCO | 0.3952 -> 0.3919 | 0.3256 -> 0.3251 | 0.6061 -> 0.6056 | 1.4034 -> 1.3960 | 0.2630 -> 0.2628 |
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
<!-- **Summary: gain over each captioner's own top beam (CIDEr).**
|
| 174 |
-
|
| 175 |
-
| Captioner | Top beam | + This model | Gain [95% CI] |
|
| 176 |
-
| --- | ---: | ---: | --- |
|
| 177 |
-
| BLIP-base (training captioner) | 1.0632 | 1.1470 | +0.0838 [+0.073, +0.095] |
|
| 178 |
-
| ViLBERT (paper's candidates) | 1.1150 | 1.1475 | +0.0325 [+0.021, +0.044] |
|
| 179 |
-
| Gemma 4 12B | 0.9624 | 0.9783 | +0.0159 [+0.009, +0.023] |
|
| 180 |
-
-->
|
| 181 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 182 |
|
| 183 |
## Limitations
|
| 184 |
|
| 185 |
-
|
| 186 |
-
**Use it as a reranker, not a relatedness classifier.** Reward training shifts
|
| 187 |
the scores upward. On the paper's 41,849 held-out relatedness pairs, accuracy at
|
| 188 |
the 0.5 cutoff falls from 0.893 to 0.631, and ROC-AUC from 0.962 to 0.937. For
|
| 189 |
-
relatedness probabilities, use the stage I
|
| 190 |
-
|
| 191 |
- Agreement with Flickr8K expert ratings dropped slightly: Spearman 0.365, against
|
| 192 |
0.380 for the stage I model on 5,822 pairs.
|
| 193 |
-
-
|
| 194 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 195 |
- The best validation epoch was the last epoch, so longer training was not explored.
|
| 196 |
|
|
|
|
|
|
|
| 197 |
|
| 198 |
## Citation
|
| 199 |
|
|
|
|
| 26 |
|
| 27 |
**It also improves a recent vision-language model.** Without any retraining, choosing
|
| 28 |
among Google's [Gemma 4 12B](https://huggingface.co/google/gemma-4-12B-it) nine beam
|
| 29 |
+
captions raises CIDEr from 0.9624 to 0.9783 (+0.0159 [+0.009, +0.023]) on 5,000 COCO
|
| 30 |
+
test images, and SPICE and BERTScore rise too. The gain varies by captioner and is
|
| 31 |
+
absent on BLIP-2 OPT-2.7B COCO. Details and examples are under
|
| 32 |
[Evaluation](#evaluation).
|
| 33 |
|
| 34 |
+
| Method | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE | BERTScore |
|
| 35 |
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 36 |
+
| Baseline: Gemma 4 12B, top beam | 0.2441 | 0.2739 | 0.5288 | 0.9624 | 0.2248 | 0.9438 |
|
| 37 |
+
| **Gemma 4 12B + this model** | **0.2458** | **0.2766** | **0.5314** | **0.9783** | **0.2267** | **0.9448** |
|
| 38 |
+
|
| 39 |
## Training
|
| 40 |
|
| 41 |

|
|
|
|
| 57 |
policy-gradient objective computed without sampling.
|
| 58 |
5. Training: 4 epochs, 5,000 steps, 8 images (72 captions) per step. Adam
|
| 59 |
learning rates are 1e-6 for BERT, 1e-5 for the CNN head and 1e-2 for `b, w`;
|
| 60 |
+
seed 42. It took 407 s on one RTX A5000.
|
| 61 |
+
6. Selection is on 734 held-out validation images (best validation CIDEr, epoch 4).
|
| 62 |
+
The test sets were run once, after the choice was locked in `protocol_lock.json`.
|
| 63 |
|
| 64 |
+
Learned values: beam bonus `b = [2.58, -0.22, -2.62, -3.49, -4.01, -3.78, -5.26,
|
| 65 |
-5.89, -6.85]` and `w = 3.20`. The model keeps the first beam unless BERT-CNN is
|
| 66 |
+
confident another candidate is better. On Gemma 4 12B candidates it replaces the first
|
| 67 |
+
beam on 26% of images.
|
| 68 |
+
|
| 69 |
## Usage
|
| 70 |
|
| 71 |
+
Example, COCO image 27585 with Gemma 4 12B beams and visual context `wall clock analog tree`.
|
| 72 |
+
The first beam says "A large black and gold clock stands against a blue sky", and the
|
| 73 |
+
model picks "A large ornate clock stands in front of a building".
|
| 74 |
+
|
| 75 |
+
Second example, COCO image 300233 with visual context `mixing bowl corn food`. The first
|
| 76 |
+
beam says "Ingredients and a food processor are arranged on a gray surface", and the
|
| 77 |
+
model picks "Ingredients and a food processor are laid out on a table".
|
| 78 |
+
|
| 79 |
+
The model reads the **visual context** (text labels extracted from the image) and
|
| 80 |
+
the captions. It does not read the image itself, and this repository does not
|
| 81 |
+
include the label extractor, so you supply the labels.
|
| 82 |
+
|
| 83 |
```python
|
| 84 |
from huggingface_hub import snapshot_download
|
| 85 |
import sys
|
|
|
|
| 90 |
|
| 91 |
reranker = CaptionReranker(path, device="cpu") # "cuda" for a GPU
|
| 92 |
beams = [ # exactly nine captions, in beam-search order (best beam first)
|
| 93 |
+
"A large black and gold clock stands against a blue sky.",
|
| 94 |
+
"A black and gold street clock stands against a blue sky.",
|
| 95 |
+
"A large black and gold clock stands under a blue sky.",
|
| 96 |
+
"A large black and gold street clock stands against a blue sky.",
|
| 97 |
+
"A large black and gold street clock stands under a blue sky.",
|
| 98 |
+
"A black and gold street clock stands under a blue sky.",
|
| 99 |
+
"A large ornate street clock stands against a blue sky.",
|
| 100 |
+
"A large ornate clock stands in front of a building.",
|
| 101 |
+
"A large black and gold clock stands in a city.",
|
| 102 |
]
|
| 103 |
+
visual_context = "wall clock analog tree" # text labels extracted from the image
|
| 104 |
+
best, scores = reranker.rerank(visual_context, beams)
|
| 105 |
+
print(beams[best]) # -> A large ornate clock stands in front of a building.
|
| 106 |
```
|
| 107 |
|
| 108 |
The beam bonus means `rerank` needs the nine captions **in the generator's beam
|
|
|
|
| 111 |
|
| 112 |
## Evaluation
|
| 113 |
|
| 114 |
+
Every captioner below gives nine beam-search candidates per image on the same **5,000
|
| 115 |
+
Karpathy test images**, and is compared with its own top beam. All rows use the same
|
| 116 |
+
scorers.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
|
| 118 |
**In-domain test:** BLIP 9-beam candidates for the same 5,000 images.
|
| 119 |
|
| 120 |
| Method | CIDEr |
|
| 121 |
| --- | ---: |
|
| 122 |
| Best Beam | 1.0632 |
|
| 123 |
+
| This model | **1.1470** (+0.0838 [+0.073, +0.095]) |
|
| 124 |
|
| 125 |
**Gemma 4 12B candidates.** `google/gemma-4-12B-it`, prompted for a short COCO-style
|
| 126 |
caption with nine beams, on the same 5,000 images. This model was not retrained on them.
|
|
|
|
| 130 |
| Gemma 4 12B, beam 1 of 9 | 0.2441 | 0.2739 | 0.5288 | 0.9624 | 0.2248 | 0.9438 |
|
| 131 |
| **Gemma 4 12B + this model** | **0.2458** | **0.2766** | **0.5314** | **0.9783** | **0.2267** | **0.9448** |
|
| 132 |
|
| 133 |
+
Gains: CIDEr +0.0159 [+0.009, +0.023], SPICE +0.0019 [+0.001, +0.003], BERTScore
|
| 134 |
+
+0.0009 [+0.0007, +0.0012].
|
| 135 |
|
| 136 |
Gemma 4 examples, CIDEr per caption. The model replaces Gemma's top caption on 1,298 of
|
| 137 |
5,000 images: CIDEr improves on 751, drops on 543 and ties on 4. The rows below are the
|
|
|
|
| 142 |
| COCO 27585 | wall clock analog tree | A large black and gold clock stands against a blue sky. (0.70) | **A large ornate clock stands in front of a building. (1.25)** |
|
| 143 |
| COCO 320370 | croquet ball soccer tree | A man in a red shirt stands on a green field. (0.61) | **A man in a red shirt stands on a green lawn. (0.70)** |
|
| 144 |
| COCO 300233 | mixing bowl corn food | Ingredients and a food processor are arranged on a gray surface. (0.40) | **Ingredients and a food processor are laid out on a table. (0.47)** |
|
| 145 |
+
| COCO 419228 (failure) | ice bear polar | **A white polar bear walks on rocks in an enclosure. (2.54)** | A large white polar bear walks on a rocky surface. (1.85) |
|
| 146 |
|
| 147 |
+
**Summary: CIDEr for each captioner's own top beam and with this model.**
|
| 148 |
|
| 149 |
+
| Captioner | Top beam | + This model |
|
| 150 |
+
| --- | ---: | ---: |
|
| 151 |
+
| BLIP-base (training captioner) | 1.0632 | **1.1470**\* |
|
| 152 |
+
| Qwen3-VL-8B | 0.5723 | **0.5927**\* |
|
| 153 |
+
| Gemma 4 12B | 0.9624 | **0.9783**\* |
|
| 154 |
+
| BLIP-large | 1.1509 | **1.1638**\* |
|
| 155 |
+
| Florence-2-large-ft | 1.4099 | **1.4203**\* |
|
| 156 |
+
| BLIP-2 OPT-2.7B COCO | **1.4034** | 1.3960 |
|
| 157 |
|
| 158 |
+
\* Significantly different from the top beam (paired bootstrap 95% interval over images excludes zero). Best result in bold.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
|
| 160 |
The gain shrinks as the captioner gets stronger. This model was not retrained on any
|
| 161 |
captioner in this table except BLIP-base.
|
| 162 |
|
| 163 |
+
**All metrics, nine beams each.** Bold marks the better result for each captioner.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
|
| 165 |
+
| System | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE | BERTScore |
|
| 166 |
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 167 |
+
| Qwen3-VL-8B, Best Beam | 0.0669 | 0.1872 | 0.3357 | 0.5723 | 0.1582 | - |
|
| 168 |
+
| Qwen3-VL-8B + This model | **0.0710** | **0.1910** | **0.3404** | **0.5927** | **0.1613** | - |
|
| 169 |
+
| ββββββββββββββββββββββββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ |
|
| 170 |
+
| Gemma 4 12B, Best Beam | 0.2441 | 0.2739 | 0.5288 | 0.9624 | 0.2248 | 0.9438 |
|
| 171 |
+
| Gemma 4 12B + This model | **0.2458** | **0.2766** | **0.5314** | **0.9783** | **0.2267** | **0.9448** |
|
| 172 |
+
| ββββββββββββββββββββββββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ |
|
| 173 |
+
| BLIP-large, Best Beam | 0.3004 | 0.2993 | 0.5378 | 1.1509 | 0.2381 | - |
|
| 174 |
+
| BLIP-large + This model | **0.3035** | **0.2996** | **0.5389** | **1.1638** | **0.2395** | - |
|
| 175 |
+
| ββββββββββββββββββββββββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ |
|
| 176 |
+
| Florence-2-large-ft, Best Beam | 0.4079 | 0.3214 | 0.6121 | 1.4099 | 0.2498 | - |
|
| 177 |
+
| Florence-2-large-ft + This model | **0.4090** | **0.3220** | **0.6143** | **1.4203** | **0.2512** | - |
|
| 178 |
+
| ββββββββββββββββββββββββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ | ββββββββ |
|
| 179 |
+
| BLIP-2 OPT-2.7B COCO, Best Beam | **0.3952** | **0.3256** | **0.6061** | **1.4034** | **0.2630** | - |
|
| 180 |
+
| BLIP-2 OPT-2.7B COCO + This model | 0.3919 | 0.3251 | 0.6056 | 1.3960 | 0.2628 | - |
|
| 181 |
+
|
| 182 |
+
BERTScore was not computed for Qwen3-VL-8B, BLIP-large, Florence-2 or BLIP-2. Forced nine-beam decoding changes the captioner itself: Florence-2's top beam here
|
| 183 |
+
(1.4099) is below its default three-beam output (1.4449), so reranking its nine beams
|
| 184 |
+
does not beat that default. Florence-2's nine beams also repeat, about six distinct
|
| 185 |
+
captions per image.
|
| 186 |
+
|
| 187 |
+
**CIDEr and factual accuracy can differ.** CIDEr measures overlap with human wording,
|
| 188 |
+
so a higher score does not always mean a more accurate caption. Human labeling of the
|
| 189 |
+
visual context is too expensive at this scale, so the labels are predicted automatically
|
| 190 |
+
and can be noisy.
|
| 191 |
|
| 192 |
## Limitations
|
| 193 |
|
| 194 |
+
- **Use it as a reranker, not a relatedness classifier.** Reward training shifts
|
|
|
|
| 195 |
the scores upward. On the paper's 41,849 held-out relatedness pairs, accuracy at
|
| 196 |
the 0.5 cutoff falls from 0.893 to 0.631, and ROC-AUC from 0.962 to 0.937. For
|
| 197 |
+
relatedness probabilities, use the stage I BERT-CNN.
|
|
|
|
| 198 |
- Agreement with Flickr8K expert ratings dropped slightly: Spearman 0.365, against
|
| 199 |
0.380 for the stage I model on 5,822 pairs.
|
| 200 |
+
- The beam bonus was learned for 9-beam candidates. Other beam widths may need
|
| 201 |
+
retraining. It was trained on BLIP candidates; gains carry over to other captioners
|
| 202 |
+
but are smaller there, and absent on BLIP-2 OPT-2.7B COCO.
|
| 203 |
+
- A reranker is a cheap way to improve an existing captioner, but it can only choose
|
| 204 |
+
among its candidates, so it does not replace a stronger captioner. On the same images,
|
| 205 |
+
COCO-tuned captioners score higher on their own
|
| 206 |
+
(single caption, CIDEr: Florence-2-large-ft 1.445, BLIP-2 OPT-2.7B COCO 1.369). On their
|
| 207 |
+
nine-beam candidates this model adds a small gain for Florence-2 and none for BLIP-2.
|
| 208 |
- The best validation epoch was the last epoch, so longer training was not explored.
|
| 209 |
|
| 210 |
+
The repository includes inference-only weights, the tokenizer, model and inference
|
| 211 |
+
code, training settings, and evaluation results (`evaluation/`).
|
| 212 |
|
| 213 |
## Citation
|
| 214 |
|