| --- |
| license: mit |
| language: |
| - en |
| pipeline_tag: image-to-text |
| tags: |
| - image-captioning |
| - pytorch |
| - lstm |
| - computer-vision |
| - nlp |
| - flickr8k |
| metrics: |
| - bleu |
| - rouge |
| - meteor |
| --- |
| |
| # Image Caption Generator (ResNet50 + LSTM) |
|
|
| A ResNet50 (frozen, transfer learning) + LSTM decoder model that generates natural-language captions for images. Trained on [Flickr8k](https://www.kaggle.com/datasets/adityajn105/flickr8k). |
|
|
| - **Full project code, training pipeline, and documentation:** https://github.com/adhamashraf7788/Image-Caption-Generator |
| - **Live interactive demo (Hugging Face Space):** https://huggingface.co/spaces/AdhamAshraf/image_caption_generator |
|
|
| ## Files in this repo |
|
|
| ``` |
| vocab.json # vocabulary (shared across both checkpoints) |
| base_resnet_lstm/ |
| ├── best_model.pt # baseline checkpoint |
| └── config.yaml # baseline training config |
| resnet_lstm_regularized/ |
| ├── best_model.pt # regularized checkpoint (recommended -- best results) |
| └── config.yaml # regularized training config |
| ``` |
|
|
| Two checkpoints are provided: |
|
|
| | Checkpoint | BLEU-4 (beam-3) | Notes | |
| |---|---|---| |
| | `base_resnet_lstm/best_model.pt` | 0.1364 | Initial baseline | |
| | `resnet_lstm_regularized/best_model.pt` | **0.1557** | Added LSTM output dropout, weight decay, gradient clipping — recommended | |
|
|
| Both checkpoints share the same `vocab.json` (identical vocabulary, 2,662 tokens). |
|
|
| ## Architecture |
|
|
| ``` |
| Image → ResNet50 (frozen, ImageNet-pretrained) → 2048-d feature |
| → Linear(2048 → 256) projection |
| → fed as first input step to a 1-layer LSTM (hidden_dim=512) |
| → LSTM generates caption word-by-word (beam search recommended, width 3) |
| ``` |
|
|
| Full architecture, preprocessing, and training details: see the [GitHub README](https://github.com/adhamashraf7788/Image-Caption-Generator#architecture). |
|
|
| ## How to use |
|
|
| Requires the inference code from the [GitHub repo](https://github.com/adhamashraf7788/Image-Caption-Generator) (`src/inference/predict.py` and its dependencies) — these checkpoints are not standalone `transformers`-compatible weights, they're plain PyTorch `state_dict`s wrapped with config metadata. |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| from src.inference.predict import Predictor # from the GitHub repo's src/ |
| |
| checkpoint_path = hf_hub_download( |
| repo_id="AdhamAshraf/image-caption-generator", |
| filename="resnet_lstm_regularized/best_model.pt", |
| ) |
| vocab_path = hf_hub_download( |
| repo_id="AdhamAshraf/image-caption-generator", |
| filename="vocab.json", |
| ) |
| |
| predictor = Predictor(checkpoint_path=checkpoint_path, vocab_path=vocab_path, device="cpu") |
| caption = predictor.predict("path/to/image.jpg") |
| print(caption) |
| ``` |
|
|
| ## Training data |
|
|
| [Flickr8k](https://www.kaggle.com/datasets/adityajn105/flickr8k) — 8,091 images, 5 human-written reference captions each. Split 80/10/10 (by image, not caption, to avoid leakage) using a fixed seed. |
|
|
| ## Evaluation results (test set, 810 images) |
|
|
| | Metric | Baseline + greedy | Baseline + beam-3 | Regularized + greedy | **Regularized + beam-3** | |
| |---|---|---|---|---| |
| | BLEU-1 | 0.5127 | 0.5240 | 0.5444 | **0.5517** | |
| | BLEU-4 | 0.1221 | 0.1364 | 0.1435 | **0.1557** | |
| | ROUGE-L | 0.4177 | 0.4265 | 0.4434 | **0.4527** | |
| | METEOR | 0.3266 | 0.3267 | 0.3480 | **0.3528** | |
|
|
| Full evaluation methodology, qualitative examples, and failure-case analysis: see the [GitHub README](https://github.com/adhamashraf7788/Image-Caption-Generator#evaluation-metrics-and-results). |
|
|
| ## Limitations |
|
|
| - Trained on a small (8k image) dataset; struggles with image content/styles underrepresented in Flickr8k (predominantly people, dogs, and outdoor scenes). |
| - Even the regularized model still shows some overfitting past its best epoch. |
| - Generated captions are sometimes fluent but not fully grounded in image-specific detail. |
|
|
| See the [GitHub README's Limitations section](https://github.com/adhamashraf7788/Image-Caption-Generator#known-limitations--next-steps) for a full discussion, including a documented failure case and how regularization + beam search improved it. |
