replace readme
Browse files
README.md
CHANGED
|
@@ -1,158 +1,66 @@
|
|
| 1 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
-
|
| 4 |
|
| 5 |
-
|
|
|
|
|
|
|
| 6 |
|
| 7 |
-
|
| 8 |
-
2. Open-source model (CLIP: `openai/clip-vit-base-patch32`)
|
| 9 |
-
3. Closed-source model (OpenAI Vision API)
|
| 10 |
|
| 11 |
-
|
|
|
|
|
|
|
|
|
|
| 12 |
|
| 13 |
-
##
|
| 14 |
|
| 15 |
-
|
|
|
|
| 16 |
|
| 17 |
-
|
| 18 |
|
| 19 |
-
|
| 20 |
-
-
|
| 21 |
-
-
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
-
|
|
|
|
|
|
|
| 24 |
|
| 25 |
-
|
| 26 |
-
- Test images: 25
|
| 27 |
|
| 28 |
-
|
| 29 |
|
| 30 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
-
|
| 33 |
-
- Train transforms:
|
| 34 |
-
- `RandomResizedCrop(224)`
|
| 35 |
-
- `RandomHorizontalFlip()`
|
| 36 |
-
- `ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2)`
|
| 37 |
-
- ImageNet normalization
|
| 38 |
-
- Eval transforms:
|
| 39 |
-
- `Resize(256)`
|
| 40 |
-
- `CenterCrop(224)`
|
| 41 |
-
- ImageNet normalization
|
| 42 |
|
| 43 |
-
|
|
|
|
|
|
|
| 44 |
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
## 4. Model and Training
|
| 48 |
-
|
| 49 |
-
### Custom Transfer Learning Model
|
| 50 |
-
|
| 51 |
-
- Backbone: `ResNet18` with ImageNet pretrained weights
|
| 52 |
-
- Final layer replaced to output 6 classes
|
| 53 |
-
- Loss: CrossEntropyLoss
|
| 54 |
-
- Optimizer: Adam
|
| 55 |
-
- Learning rate: `1e-4`
|
| 56 |
-
- Weight decay: `1e-4`
|
| 57 |
-
- Epochs run: `4`
|
| 58 |
-
- Device used: CPU
|
| 59 |
-
|
| 60 |
-
### Training Output Artifacts
|
| 61 |
-
|
| 62 |
-
- Model checkpoint: `models/custom_resnet18.pth`
|
| 63 |
-
- Training metrics: `reports/custom_model_metrics.json`
|
| 64 |
-
|
| 65 |
-
## 5. Evaluation and Comparison Results
|
| 66 |
-
|
| 67 |
-
### Summary Accuracy
|
| 68 |
-
|
| 69 |
-
- Custom transfer learning model: **0.80**
|
| 70 |
-
- CLIP (open-source): **0.72**
|
| 71 |
-
- OpenAI Vision (closed-source): not executed locally in this run because `OPENAI_API_KEY` was not set in the local environment
|
| 72 |
-
|
| 73 |
-
### Interpretation
|
| 74 |
-
|
| 75 |
-
- The custom fine-tuned model performs best on this domain-specific dataset.
|
| 76 |
-
- CLIP performs reasonably well in zero-shot mode without task-specific training.
|
| 77 |
-
- OpenAI Vision integration is fully implemented in the app and evaluation pipeline and becomes active once `OPENAI_API_KEY` is configured.
|
| 78 |
-
|
| 79 |
-
## 6. Application (Hugging Face Space Ready)
|
| 80 |
-
|
| 81 |
-
The Gradio app provides:
|
| 82 |
-
|
| 83 |
-
- image upload
|
| 84 |
-
- predictions from custom model, CLIP, and OpenAI Vision
|
| 85 |
-
- built-in example images (`app/examples/*.png`)
|
| 86 |
-
|
| 87 |
-
### App files
|
| 88 |
-
|
| 89 |
-
- Main HF entrypoint: `app.py`
|
| 90 |
-
- App implementation: `app/app.py`
|
| 91 |
-
- Space dependencies: `app/requirements.txt`
|
| 92 |
-
|
| 93 |
-
## 7. Links (to fill after publishing)
|
| 94 |
|
| 95 |
- Hugging Face Space: `https://huggingface.co/spaces/<your-username>/<your-space-name>`
|
| 96 |
-
- Hugging Face model repo: `https://huggingface.co/<your-username>/<your-model-repo>`
|
| 97 |
-
|
| 98 |
-
## 8. OpenAI Key Setup (for Hugging Face Space)
|
| 99 |
-
|
| 100 |
-
In your Hugging Face Space:
|
| 101 |
-
|
| 102 |
-
1. Open `Settings`
|
| 103 |
-
2. Go to `Variables and secrets`
|
| 104 |
-
3. Add new secret:
|
| 105 |
-
- Name: `OPENAI_API_KEY`
|
| 106 |
-
- Value: use the key provided in the exercise sheet
|
| 107 |
-
|
| 108 |
-
Do not hardcode the API key in repository files.
|
| 109 |
-
|
| 110 |
-
## 9. Reproducibility: Local Run Commands
|
| 111 |
-
|
| 112 |
-
```bash
|
| 113 |
-
# from project root
|
| 114 |
-
python -m venv .venv
|
| 115 |
-
.venv\\Scripts\\activate
|
| 116 |
-
pip install -r requirements.txt
|
| 117 |
-
|
| 118 |
-
# train custom model
|
| 119 |
-
python -m src.train_custom_model --epochs 4 --batch-size 16
|
| 120 |
-
|
| 121 |
-
# evaluate and compare models
|
| 122 |
-
python -m src.evaluate_models --openai-max-samples 24
|
| 123 |
-
|
| 124 |
-
# launch app
|
| 125 |
-
python app.py
|
| 126 |
-
```
|
| 127 |
-
|
| 128 |
-
## 10. Hugging Face Model Upload
|
| 129 |
-
|
| 130 |
-
A helper script is included:
|
| 131 |
-
|
| 132 |
-
```bash
|
| 133 |
-
python -m src.upload_to_hf --repo-id <your-username>/<your-model-repo>
|
| 134 |
-
```
|
| 135 |
-
|
| 136 |
-
This uploads `models/custom_resnet18.pth` to your Hugging Face model repository.
|
| 137 |
-
|
| 138 |
-
## 11. Notes
|
| 139 |
-
|
| 140 |
-
- This implementation follows the mandatory exercise requirements end-to-end (training, comparison, app, example images, documentation).
|
| 141 |
-
- If you want to switch from Pokemon to a car dataset, you can keep the same pipeline and replace `data/pokemon/*` with your car class folders.
|
| 142 |
-
- On Windows with very long folder paths, package installation can fail with path-length errors. In that case, create a short-path venv (example: `C:\cv-modelcmp-venv`) and run the same commands with that Python executable.
|
| 143 |
-
# Computer Vision Classification and Model Comparison
|
| 144 |
-
|
| 145 |
-
Project implementation for the mandatory exercise.
|
| 146 |
-
|
| 147 |
-
The submission-specific documentation is in [readme.md](readme.md), following the requested structure.
|
| 148 |
-
|
| 149 |
-
## Quick start
|
| 150 |
-
|
| 151 |
-
```bash
|
| 152 |
-
python -m venv .venv
|
| 153 |
-
.venv\\Scripts\\activate
|
| 154 |
-
pip install -r requirements.txt
|
| 155 |
-
python -m src.train_custom_model --epochs 4
|
| 156 |
-
python -m src.evaluate_models
|
| 157 |
-
python app.py
|
| 158 |
-
```
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: Computer Vision Classification Comparison
|
| 3 |
+
emoji: 📷
|
| 4 |
+
colorFrom: yellow
|
| 5 |
+
colorTo: blue
|
| 6 |
+
sdk: gradio
|
| 7 |
+
sdk_version: "6.12.0"
|
| 8 |
+
app_file: app.py
|
| 9 |
+
pinned: false
|
| 10 |
+
---
|
| 11 |
+
# Gradio Pokemon Classification App
|
| 12 |
|
| 13 |
+
This app compares 3 image classification approaches on custom Pokemon images:
|
| 14 |
|
| 15 |
+
- Fine-tuned transfer learning model (ResNet18) from this project
|
| 16 |
+
- Zero-shot CLIP (`openai/clip-vit-base-patch32`)
|
| 17 |
+
- OpenAI vision model (LLM image classification)
|
| 18 |
|
| 19 |
+
## Dataset Used For Training
|
|
|
|
|
|
|
| 20 |
|
| 21 |
+
- Custom dataset from course materials (week 8 style): `data/pokemon`
|
| 22 |
+
- Train split: `data/pokemon/train`
|
| 23 |
+
- Test split: `data/pokemon/test`
|
| 24 |
+
- Number of classes: `6` (`charizard`, `charmander`, `charmeleon`, `ditto`, `eevee`, `ekans`)
|
| 25 |
|
| 26 |
+
## Trained Model
|
| 27 |
|
| 28 |
+
- Local model artifact: `models/custom_resnet18.pth`
|
| 29 |
+
- Hugging Face model link: `https://huggingface.co/<your-username>/<your-model-repo>`
|
| 30 |
|
| 31 |
+
## Training Performance
|
| 32 |
|
| 33 |
+
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|
| 34 |
+
|---:|---:|---:|---:|---:|
|
| 35 |
+
| 1.6525 | 1 | - | 0.4324 | 0.2848 |
|
| 36 |
+
| 0.9985 | 2 | - | 0.6216 | 0.7285 |
|
| 37 |
+
| 0.6668 | 3 | - | 0.7838 | 0.8344 |
|
| 38 |
+
| 0.4450 | 4 | - | 0.8378 | 0.9007 |
|
| 39 |
|
| 40 |
+
Notes:
|
| 41 |
+
- Validation loss was not logged separately in the training script, so the table reports validation accuracy in the validation column context and train accuracy in the accuracy column.
|
| 42 |
+
- Final test accuracy of the custom model: `0.80`.
|
| 43 |
|
| 44 |
+
## Example Image Results
|
|
|
|
| 45 |
|
| 46 |
+
The table below reports the true class and Top-3 predictions for Custom ResNet and CLIP.
|
| 47 |
|
| 48 |
+
| Image | True Class | Custom ResNet Top-3 (score) | CLIP Top-3 (score) | OpenAI LLM (label, confidence) |
|
| 49 |
+
|---|---|---|---|---|
|
| 50 |
+
| `charizard.png` | `charizard` | `eevee` (0.5464)<br>`charizard` (0.3056)<br>`charmander` (0.0711) | `charizard` (0.4536)<br>`charmander` (0.3524)<br>`charmeleon` (0.0896) | unavailable (missing `OPENAI_API_KEY`) |
|
| 51 |
+
| `charmander.png` | `charmander` | `charmander` (0.5801)<br>`charmeleon` (0.3410)<br>`ekans` (0.0315) | `charmeleon` (0.5400)<br>`charmander` (0.4202)<br>`charizard` (0.0268) | unavailable (missing `OPENAI_API_KEY`) |
|
| 52 |
+
| `charmeleon.png` | `charmeleon` | `charmeleon` (0.3503)<br>`eevee` (0.3164)<br>`ekans` (0.2516) | `charmeleon` (0.4247)<br>`eevee` (0.3977)<br>`charizard` (0.0802) | unavailable (missing `OPENAI_API_KEY`) |
|
| 53 |
+
| `ditto.png` | `ditto` | `ditto` (0.8271)<br>`ekans` (0.0959)<br>`eevee` (0.0370) | `ditto` (0.5273)<br>`ekans` (0.2055)<br>`eevee` (0.1169) | unavailable (missing `OPENAI_API_KEY`) |
|
| 54 |
+
| `eevee.png` | `eevee` | `eevee` (0.9962)<br>`ekans` (0.0022)<br>`charizard` (0.0006) | `eevee` (0.9984)<br>`ditto` (0.0008)<br>`charizard` (0.0004) | unavailable (missing `OPENAI_API_KEY`) |
|
| 55 |
+
| `ekans.png` | `ekans` | `ekans` (0.5366)<br>`eevee` (0.3374)<br>`charmander` (0.0682) | `ekans` (0.5120)<br>`charmeleon` (0.2303)<br>`charmander` (0.2177) | unavailable (missing `OPENAI_API_KEY`) |
|
| 56 |
|
| 57 |
+
## Overall Comparison (Test Set)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
|
| 59 |
+
- Custom transfer learning model accuracy: `0.80`
|
| 60 |
+
- CLIP zero-shot model accuracy: `0.72`
|
| 61 |
+
- OpenAI vision model: integrated in app, local evaluation skipped without API key
|
| 62 |
|
| 63 |
+
## Links
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
|
| 65 |
- Hugging Face Space: `https://huggingface.co/spaces/<your-username>/<your-space-name>`
|
| 66 |
+
- Hugging Face model repo: `https://huggingface.co/<your-username>/<your-model-repo>`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|