Hugging Face's logo

AdhamAshraf
/
image-caption-generator

Image-to-Text
PyTorch
English
image-captioning
lstm
computer-vision
nlp
flickr8k
Model card Files
xet
 Community
image-caption-generator
File size: 4,175 Bytes
b77aba3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
---
license: mit
language:
- en
pipeline_tag: image-to-text
tags:
- image-captioning
- pytorch
- lstm
- computer-vision
- nlp
- flickr8k
metrics:
- bleu
- rouge
- meteor
---

# Image Caption Generator (ResNet50 + LSTM)

A ResNet50 (frozen, transfer learning) + LSTM decoder model that generates natural-language captions for images. Trained on [Flickr8k](https://www.kaggle.com/datasets/adityajn105/flickr8k).

- **Full project code, training pipeline, and documentation:** https://github.com/adhamashraf7788/Image-Caption-Generator
- **Live interactive demo (Hugging Face Space):** https://huggingface.co/spaces/AdhamAshraf/image_caption_generator

## Files in this repo

```
vocab.json                             # vocabulary (shared across both checkpoints)
base_resnet_lstm/
├── best_model.pt                      # baseline checkpoint
└── config.yaml                        # baseline training config
resnet_lstm_regularized/
├── best_model.pt                      # regularized checkpoint (recommended -- best results)
└── config.yaml                        # regularized training config
```

Two checkpoints are provided:

| Checkpoint | BLEU-4 (beam-3) | Notes |
|---|---|---|
| `base_resnet_lstm/best_model.pt` | 0.1364 | Initial baseline |
| `resnet_lstm_regularized/best_model.pt` | **0.1557** | Added LSTM output dropout, weight decay, gradient clipping — recommended |

Both checkpoints share the same `vocab.json` (identical vocabulary, 2,662 tokens).

## Architecture

```
Image → ResNet50 (frozen, ImageNet-pretrained) → 2048-d feature
      → Linear(2048 → 256) projection
      → fed as first input step to a 1-layer LSTM (hidden_dim=512)
      → LSTM generates caption word-by-word (beam search recommended, width 3)
```

Full architecture, preprocessing, and training details: see the [GitHub README](https://github.com/adhamashraf7788/Image-Caption-Generator#architecture).

## How to use

Requires the inference code from the [GitHub repo](https://github.com/adhamashraf7788/Image-Caption-Generator) (`src/inference/predict.py` and its dependencies) — these checkpoints are not standalone `transformers`-compatible weights, they're plain PyTorch `state_dict`s wrapped with config metadata.

```python
from huggingface_hub import hf_hub_download
from src.inference.predict import Predictor  # from the GitHub repo's src/

checkpoint_path = hf_hub_download(
    repo_id="AdhamAshraf/image-caption-generator",
    filename="resnet_lstm_regularized/best_model.pt",
)
vocab_path = hf_hub_download(
    repo_id="AdhamAshraf/image-caption-generator",
    filename="vocab.json",
)

predictor = Predictor(checkpoint_path=checkpoint_path, vocab_path=vocab_path, device="cpu")
caption = predictor.predict("path/to/image.jpg")
print(caption)
```

## Training data

[Flickr8k](https://www.kaggle.com/datasets/adityajn105/flickr8k) — 8,091 images, 5 human-written reference captions each. Split 80/10/10 (by image, not caption, to avoid leakage) using a fixed seed.

## Evaluation results (test set, 810 images)

| Metric | Baseline + greedy | Baseline + beam-3 | Regularized + greedy | **Regularized + beam-3** |
|---|---|---|---|---|
| BLEU-1 | 0.5127 | 0.5240 | 0.5444 | **0.5517** |
| BLEU-4 | 0.1221 | 0.1364 | 0.1435 | **0.1557** |
| ROUGE-L | 0.4177 | 0.4265 | 0.4434 | **0.4527** |
| METEOR | 0.3266 | 0.3267 | 0.3480 | **0.3528** |

Full evaluation methodology, qualitative examples, and failure-case analysis: see the [GitHub README](https://github.com/adhamashraf7788/Image-Caption-Generator#evaluation-metrics-and-results).

## Limitations

- Trained on a small (8k image) dataset; struggles with image content/styles underrepresented in Flickr8k (predominantly people, dogs, and outdoor scenes).
- Even the regularized model still shows some overfitting past its best epoch.
- Generated captions are sometimes fluent but not fully grounded in image-specific detail.

See the [GitHub README's Limitations section](https://github.com/adhamashraf7788/Image-Caption-Generator#known-limitations--next-steps) for a full discussion, including a documented failure case and how regularization + beam search improved it.