Spaces:
No application file
No application file
| title: Neural Image Captioning | |
| emoji: 🖼️ | |
| colorFrom: indigo | |
| colorTo: blue | |
| sdk: gradio | |
| sdk_version: 6.26.0 | |
| app_file: app/gradio_app.py | |
| pinned: false | |
| # 🖼️ Neural Image Caption Generation | |
|  | |
|  | |
|  | |
| ## Overview | |
| End-to-end image captioning system using deep learning, combining computer vision (ResNet-50) with NLP (LSTM) to generate natural language descriptions of images. | |
| ## Architecture | |
| ```mermaid | |
| graph LR | |
| A[Input Image] -->|Resize & Normalize| B[ResNet-50 Frozen] | |
| B -->|2048-d| C[Linear Projection] | |
| C -->|256-d| D[LSTM Decoder] | |
| D -->|Step-by-step| E[Caption] | |
| ``` | |
| This model follows the "Show and Tell" architecture by utilizing a pre-trained CNN to extract image features and passing those features as the initial state or input to an LSTM which decodes them into a natural language sequence. | |
| ## Dataset | |
| We use the **Flickr8k** dataset: | |
| - **8,092** images in total | |
| - **5** captions per image | |
| - **Splits**: | |
| - Train: 6,000 images | |
| - Val: 1,000 images | |
| - Test: 1,000 images | |
| - **Preprocessing**: Lowercase conversion, punctuation removal, and frequency thresholding. | |
| ## Project Structure | |
| ```text | |
| . | |
| ├── app/ | |
| │ ├── app.py (Streamlit App) | |
| │ └── gradio_app.py (Gradio App) | |
| ├── checkpoints/ | |
| ├── scripts/ | |
| │ ├── extract_features.py | |
| │ ├── train.py | |
| │ └── evaluate.py | |
| ├── src/ | |
| │ ├── data/ | |
| │ ├── evaluation/ | |
| │ ├── inference/ | |
| │ ├── models/ | |
| │ └── utils/ | |
| ├── tests/ | |
| │ ├── integration/ | |
| │ └── unit/ | |
| ├── Dockerfile | |
| ├── Dockerfile.gpu | |
| ├── docker-compose.yml | |
| ├── requirements.txt | |
| └── README.md | |
| ``` | |
| ## Installation | |
| ```bash | |
| git clone <repo-url> | |
| cd Task62 | |
| pip install -r requirements.txt | |
| make setup | |
| ``` | |
| ## Quick Start | |
| 1. Download the Flickr8k dataset from Kaggle. | |
| 2. Run data preparation: `python setup_data.py` | |
| 3. Extract features: `python scripts/extract_features.py` | |
| 4. Train the model: `python scripts/train.py` | |
| 5. Evaluate: `python scripts/evaluate.py` | |
| ## Training | |
| | Hyperparameter | Value | | |
| | -------------- | ----- | | |
| | Embed Dim | 256 | | |
| | Hidden Dim | 512 | | |
| | Batch Size | 64 | | |
| | Learning Rate | 1e-3 | | |
| Features include early stopping, LR scheduling, gradient clipping, and automated checkpointing. | |
| ## Evaluation | |
| | Metric | Expected Score | | |
| | ------- | -------------- | | |
| | BLEU-1 | ~60.0 | | |
| | BLEU-4 | ~20.0 | | |
| | ROUGE-L | ~45.0 | | |
| | METEOR | ~22.0 | | |
| _Qualitative examples showing generated captions will be displayed here._ | |
| ## Web Interfaces | |
| ### Streamlit App | |
| Start the Streamlit interface to upload and test your own images: | |
| ```bash | |
| streamlit run app/app.py | |
| # or | |
| make app | |
| ``` | |
| Available at: `http://localhost:8501` | |
| ### Gradio App | |
| Start the Gradio interface: | |
| ```bash | |
| python app/gradio_app.py | |
| # or | |
| make gradio | |
| ``` | |
| Available at: `http://localhost:7860` | |
| ## Docker | |
| Run both the Streamlit UI and Gradio UI inside Docker containers: | |
| ```bash | |
| # Build the image | |
| docker build -t image-captioning . | |
| # Run both services | |
| docker-compose up | |
| # Services: | |
| # Streamlit UI → http://localhost:8501 | |
| # Gradio UI → http://localhost:7860 | |
| ``` | |
| ## Model & Deployments | |
| ### 🤗 Hugging Face Spaces | |
| Try the model live on Hugging Face Spaces: [Image Captioning Demo](https://huggingface.co/spaces/Nada0salim/image-captioning) | |
| ### 🐳 Docker Hub | |
| Container image available at: [Docker Hub Repository](https://hub.docker.com/repository/docker/nada123456aa/image-caption/general) | |
| ## Testing | |
| Run the test suite using pytest: | |
| ```bash | |
| pytest tests/ -v --cov=src | |
| ``` | |
| ## Technologies | |
| - Python 3.10+ | |
| - PyTorch & torchvision (ResNet-50) | |
| - Streamlit & Gradio (Web UIs) | |
| - Docker & Docker Compose | |
| - NLTK, rouge-score | |
| - HuggingFace Hub | |
| ## License | |
| MIT |