|
Download README.md from AhmedSSabir/ModernBERT-CNN-Caption-Reranker: direct link, hf CLI and curl.
- Browser
- Download file 4.26 kB
-
https://huggingface.co/AhmedSSabir/ModernBERT-CNN-Caption-Reranker/resolve/main/README.md
- Command line
-
hf download hf://AhmedSSabir/ModernBERT-CNN-Caption-Reranker/README.md
-
curl -L -o README.md https://huggingface.co/AhmedSSabir/ModernBERT-CNN-Caption-Reranker/resolve/main/README.md
4.26 kB
| language: en | |
| library_name: pytorch | |
| base_model: answerdotai/ModernBERT-base | |
| datasets: | |
| - AhmedSSabir/Textual-Image-Caption-Dataset | |
| tags: | |
| - modernbert | |
| - image-captioning | |
| - visual-relatedness | |
| - text-classification | |
| - custom-code | |
| # ModernBERT-CNN Caption Reranker | |
| ModernBERT-base with a CNN head that scores the relatedness between **image-context | |
| words and a candidate caption**. The output is a probability from 0 to 1. Use it | |
| to rank candidate captions or assign soft pseudo-labels to context/caption pairs. | |
| The model does not read images or generate captions. | |
| <!-- | |
| ## Related paper and dataset | |
| - **Published paper:** [Visual Semantic Relatedness Dataset for Image Captioning](https://huggingface.co/papers/2301.08784). | |
| - **Source dataset:** [AhmedSSabir/Textual-Image-Caption-Dataset](https://huggingface.co/datasets/AhmedSSabir/Textual-Image-Caption-Dataset). Training and held-out classification splits were derived from this release as described below. | |
| --> | |
| ## Usage | |
| Install the dependencies from `requirements.txt`, then run: | |
| ```python | |
| from huggingface_hub import snapshot_download | |
| import sys | |
| path = snapshot_download("AhmedSSabir/ModernBERT-CNN-Caption-Reranker") | |
| sys.path.insert(0, path) | |
| from inference import RelatednessScorer | |
| scorer = RelatednessScorer(path, device="cpu") # use "cuda" for a GPU | |
| scores = scorer.score("dog, ball, grass", [ | |
| "A dog plays with a ball on the grass.", | |
| "A person cooks in a kitchen.", | |
| ]) | |
| print(scores) | |
| ``` | |
| This is a custom CNN architecture. Use the included inference helper; the | |
| standard `AutoModel` loader is not configured for this head. Context words | |
| should come from the image's detected objects or scene labels. | |
| ## Model and training | |
| - Base: `answerdotai/ModernBERT-base`; all 22 encoder layers fine-tuned, | |
| embedding parameters frozen. | |
| - Sequence length: 64 tokens for the combined context/caption input. | |
| - Head: 32 convolution filters, kernel width 3, ReLU, local max pooling, | |
| time-major flattening, sigmoid score. | |
| - Training: 330,790 pairs; validation: 41,275 pairs. Adam, learning rate 1e-5, | |
| batch size 16, seed 42; BF16/TF32 and compilation used during training. | |
| - Early stopping after three epochs; **best validation checkpoint: epoch 1**, | |
| validation loss 0.256281. | |
| ## Evaluation | |
| Caption reranking: **45,000 candidates, nine per image, over 5,000 images**. | |
| Select the highest model score per image, with the earliest candidate winning | |
| ties. No reference-based tuning was performed. | |
| | Metric | Score | | |
| | --- | ---: | | |
| | BLEU-4 | 0.329180 | | |
| | METEOR | 0.271592 | | |
| | ROUGE-L | 0.552392 | | |
| | CIDEr | 1.096723 | | |
| | SPICE | 0.205827 | | |
| | BERTScore F1 | 0.939820 | | |
| Additional tests: | |
| - **41,849 held-out pseudo-labeled pairs:** accuracy 0.890893, F1 0.904035, | |
| ROC-AUC 0.961563. | |
| - **5,822 Flickr8K expert-rated pairs / 1,000 images:** Spearman 0.293677, | |
| Kendall tau-b 0.212962 using fixed combined image-derived textual context. | |
| ## Limitations | |
| Training labels derive from semantic similarity scores thresholded at 0.3; | |
| they are not human judgments. Output probabilities are not calibrated human | |
| quality ratings, and human-rating agreement is modest. Object extraction | |
| errors, truncated inputs and domain shift affect results. Evaluation is from | |
| one training seed, and the derived training split and visual extraction | |
| pipeline differ from the historical experiment in the original paper. | |
| A small CLIP-teacher pilot did not improve agreement with teacher preferences on the validation | |
| set (51/64 pairs before and after training). Caption-reranking metrics and human-rated quality | |
| were not evaluated for this pilot, so its effect on those outcomes remains unknown. | |
| ## Citation | |
| ```bibtex | |
| @InProceedings{Sabir_2023_CVPR, | |
| author = {Sabir, Ahmed and Moreno-Noguer, Francesc and Padr{\'o}, Llu{\'i}s}, | |
| title = {Visual Semantic Relatedness Dataset for Image Captioning}, | |
| booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops}, | |
| month = {June}, | |
| year = {2023}, | |
| pages = {5598--5606}, | |
| doi = {10.1109/CVPRW59228.2023.00592}, | |
| url = {https://openaccess.thecvf.com/content/CVPR2023W/O-DRUM/html/Sabir_Visual_Semantic_Relatedness_Dataset_for_Image_Captioning_CVPRW_2023_paper.html} | |
| } | |
| ``` | |