File size: 4,258 Bytes
90316bb d134a03 90316bb 35d45a8 90316bb 496d11e d134a03 4be24b3 d134a03 496d11e 90316bb 35d45a8 90316bb 4df5eae 90316bb 4df5eae 90316bb 755afbd 9c3ab11 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | ---
language: en
library_name: pytorch
base_model: answerdotai/ModernBERT-base
datasets:
- AhmedSSabir/Textual-Image-Caption-Dataset
tags:
- modernbert
- image-captioning
- visual-relatedness
- text-classification
- custom-code
---
# ModernBERT-CNN Caption Reranker
ModernBERT-base with a CNN head that scores the relatedness between **image-context
words and a candidate caption**. The output is a probability from 0 to 1. Use it
to rank candidate captions or assign soft pseudo-labels to context/caption pairs.
The model does not read images or generate captions.
<!--
## Related paper and dataset
- **Published paper:** [Visual Semantic Relatedness Dataset for Image Captioning](https://huggingface.co/papers/2301.08784).
- **Source dataset:** [AhmedSSabir/Textual-Image-Caption-Dataset](https://huggingface.co/datasets/AhmedSSabir/Textual-Image-Caption-Dataset). Training and held-out classification splits were derived from this release as described below.
-->
## Usage
Install the dependencies from `requirements.txt`, then run:
```python
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("AhmedSSabir/ModernBERT-CNN-Caption-Reranker")
sys.path.insert(0, path)
from inference import RelatednessScorer
scorer = RelatednessScorer(path, device="cpu") # use "cuda" for a GPU
scores = scorer.score("dog, ball, grass", [
"A dog plays with a ball on the grass.",
"A person cooks in a kitchen.",
])
print(scores)
```
This is a custom CNN architecture. Use the included inference helper; the
standard `AutoModel` loader is not configured for this head. Context words
should come from the image's detected objects or scene labels.
## Model and training
- Base: `answerdotai/ModernBERT-base`; all 22 encoder layers fine-tuned,
embedding parameters frozen.
- Sequence length: 64 tokens for the combined context/caption input.
- Head: 32 convolution filters, kernel width 3, ReLU, local max pooling,
time-major flattening, sigmoid score.
- Training: 330,790 pairs; validation: 41,275 pairs. Adam, learning rate 1e-5,
batch size 16, seed 42; BF16/TF32 and compilation used during training.
- Early stopping after three epochs; **best validation checkpoint: epoch 1**,
validation loss 0.256281.
## Evaluation
Caption reranking: **45,000 candidates, nine per image, over 5,000 images**.
Select the highest model score per image, with the earliest candidate winning
ties. No reference-based tuning was performed.
| Metric | Score |
| --- | ---: |
| BLEU-4 | 0.329180 |
| METEOR | 0.271592 |
| ROUGE-L | 0.552392 |
| CIDEr | 1.096723 |
| SPICE | 0.205827 |
| BERTScore F1 | 0.939820 |
Additional tests:
- **41,849 held-out pseudo-labeled pairs:** accuracy 0.890893, F1 0.904035,
ROC-AUC 0.961563.
- **5,822 Flickr8K expert-rated pairs / 1,000 images:** Spearman 0.293677,
Kendall tau-b 0.212962 using fixed combined image-derived textual context.
## Limitations
Training labels derive from semantic similarity scores thresholded at 0.3;
they are not human judgments. Output probabilities are not calibrated human
quality ratings, and human-rating agreement is modest. Object extraction
errors, truncated inputs and domain shift affect results. Evaluation is from
one training seed, and the derived training split and visual extraction
pipeline differ from the historical experiment in the original paper.
A small CLIP-teacher pilot did not improve agreement with teacher preferences on the validation
set (51/64 pairs before and after training). Caption-reranking metrics and human-rated quality
were not evaluated for this pilot, so its effect on those outcomes remains unknown.
## Citation
```bibtex
@InProceedings{Sabir_2023_CVPR,
author = {Sabir, Ahmed and Moreno-Noguer, Francesc and Padr{\'o}, Llu{\'i}s},
title = {Visual Semantic Relatedness Dataset for Image Captioning},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2023},
pages = {5598--5606},
doi = {10.1109/CVPRW59228.2023.00592},
url = {https://openaccess.thecvf.com/content/CVPR2023W/O-DRUM/html/Sabir_Visual_Semantic_Relatedness_Dataset_for_Image_Captioning_CVPRW_2023_paper.html}
}
```
|