File size: 4,258 Bytes
90316bb
 
 
 
d134a03
 
90316bb
 
 
 
 
 
 
35d45a8
90316bb
 
 
 
 
496d11e
d134a03
 
4be24b3
d134a03
496d11e
90316bb
 
 
 
 
 
 
 
35d45a8
90316bb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4df5eae
90316bb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4df5eae
90316bb
755afbd
 
 
 
9c3ab11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
---
language: en
library_name: pytorch
base_model: answerdotai/ModernBERT-base
datasets:
- AhmedSSabir/Textual-Image-Caption-Dataset
tags:
- modernbert
- image-captioning
- visual-relatedness
- text-classification
- custom-code
---
# ModernBERT-CNN Caption Reranker

ModernBERT-base with a CNN head that scores the relatedness between **image-context
words and a candidate caption**. The output is a probability from 0 to 1. Use it
to rank candidate captions or assign soft pseudo-labels to context/caption pairs.
The model does not read images or generate captions.
<!-- 
## Related paper and dataset

- **Published paper:** [Visual Semantic Relatedness Dataset for Image Captioning](https://huggingface.co/papers/2301.08784).
- **Source dataset:** [AhmedSSabir/Textual-Image-Caption-Dataset](https://huggingface.co/datasets/AhmedSSabir/Textual-Image-Caption-Dataset). Training and held-out classification splits were derived from this release as described below.
 -->
## Usage

Install the dependencies from `requirements.txt`, then run:

```python
from huggingface_hub import snapshot_download
import sys

path = snapshot_download("AhmedSSabir/ModernBERT-CNN-Caption-Reranker")
sys.path.insert(0, path)
from inference import RelatednessScorer

scorer = RelatednessScorer(path, device="cpu")  # use "cuda" for a GPU
scores = scorer.score("dog, ball, grass", [
    "A dog plays with a ball on the grass.",
    "A person cooks in a kitchen.",
])
print(scores)
```

This is a custom CNN architecture. Use the included inference helper; the
standard `AutoModel` loader is not configured for this head. Context words
should come from the image's detected objects or scene labels.

## Model and training

- Base: `answerdotai/ModernBERT-base`; all 22 encoder layers fine-tuned,
  embedding parameters frozen.
- Sequence length: 64 tokens for the combined context/caption input.
- Head: 32 convolution filters, kernel width 3, ReLU, local max pooling,
  time-major flattening, sigmoid score.
- Training: 330,790 pairs; validation: 41,275 pairs. Adam, learning rate 1e-5,
  batch size 16, seed 42; BF16/TF32 and compilation used during training.
- Early stopping after three epochs; **best validation checkpoint: epoch 1**,
  validation loss 0.256281.

## Evaluation

Caption reranking: **45,000 candidates, nine per image, over 5,000 images**.
Select the highest model score per image, with the earliest candidate winning
ties. No reference-based tuning was performed.

| Metric | Score |
| --- | ---: |
| BLEU-4 | 0.329180 |
| METEOR | 0.271592 |
| ROUGE-L | 0.552392 |
| CIDEr | 1.096723 |
| SPICE | 0.205827 |
| BERTScore F1 | 0.939820 |

Additional tests:

- **41,849 held-out pseudo-labeled pairs:** accuracy 0.890893, F1 0.904035,
  ROC-AUC 0.961563.
- **5,822 Flickr8K expert-rated pairs / 1,000 images:** Spearman 0.293677,
  Kendall tau-b 0.212962 using fixed combined image-derived textual context.

## Limitations

Training labels derive from semantic similarity scores thresholded at 0.3;
they are not human judgments. Output probabilities are not calibrated human
quality ratings, and human-rating agreement is modest. Object extraction
errors, truncated inputs and domain shift affect results. Evaluation is from
one training seed, and the derived training split and visual extraction
pipeline differ from the historical experiment in the original paper.

A small CLIP-teacher pilot did not improve agreement with teacher preferences on the validation 
set (51/64 pairs before and after training). Caption-reranking metrics and human-rated quality 
were not evaluated for this pilot, so its effect on those outcomes remains unknown.


## Citation

```bibtex
@InProceedings{Sabir_2023_CVPR,
  author    = {Sabir, Ahmed and Moreno-Noguer, Francesc and Padr{\'o}, Llu{\'i}s},
  title     = {Visual Semantic Relatedness Dataset for Image Captioning},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  month     = {June},
  year      = {2023},
  pages     = {5598--5606},
  doi       = {10.1109/CVPRW59228.2023.00592},
  url       = {https://openaccess.thecvf.com/content/CVPR2023W/O-DRUM/html/Sabir_Visual_Semantic_Relatedness_Dataset_for_Image_Captioning_CVPRW_2023_paper.html}
}
```